CAEP Standard 5 – Provider Quality, Continuous Improvement and Capacity


Standard 5.1

The EPP has had and continues to have a conceptual framework that includes the mission of the Unit and its Core Values. They guide much of what is done with content and pedagogy, professional practices and ethics, and understanding of the candidates served by the EPP. These Core Values include Scholarship and Reflective Practice, Diversity and Inclusion, Health and Wellness, Creativity and the Arts, and Social Justice and Leadership. There has been an effort to align the quality assurance system with them to demonstrate and evaluate candidate progress and completer achievement through multiple measures.

The Assessment system is equipped with a technology infrastructure provided by the University that supports data collection, management, analysis and reporting to ensure operations of a quality assurance system and continuous improvement. This infrastructure includes a storage system in both hard share drives and cloud share drives, software to manage data on admission (SLATE) that interface with academic data (eSAAS), a learning management system (Moodle), advisement software (CLASS, Degree Audit, EAB) and a host of other software that supports the academic operations of the University. The technology infrastructure supports archiving, aggregating and disaggregating, synthesizing, and reporting data metrics identified to evidence the effectiveness of the EPP and compliance with CAEP standards.

The EPP Assessment System is supported by the Office of Assessment and Accreditation staffed by a director, an assistant director and graduate assistants who work closely with the Dean’s office, the Professional Experiences and Community Engagement (PECE) Office, the Certification Office and the Teacher Education Programs in the Ruth S. Ammon School of Education (Teacher Ed programs, Department of Health and Sport Sciences (Health and Physical Education) and College of Arts and Sciences (Art and Music Education).

Data collection and school-wide surveys from EPP faculty, clinical educators and the field are electronic through Survey Monkey, Google Forms and Excel spreadsheets. Moodle is used as the electronic platform for all data tables, reports and information related to the assessment system. All EPP faculty and supporting institutional administrators have access to Moodle where they can download date tables in excel and conduct queries and analysis.

The EPP’s assessment system is one that is continuously updated, planned, and operationalized. It is applied fairly and equally for all candidates as well as, employs multiple strategies to confirm candidates’ preparation. The assessment system shares and utilizes findings to confirm the quality of educator preparation and inform change strategies while engages multiple entities. It is sustainable and manageable, and meets the standards determined by regulatory and accrediting agencies. There is a “cross-walk” of the quality assurance system with the evidence cited for Standards 1-4 (5.1.3 Crosswalk of Quality Assurance System with Evidence in Standard 1-4). The EPP places program-based use of evidence as the heart of the quality assurance system. Programs develop and revise assessments of candidate knowledge, skills and professionally-related values, ethics and dispositions and it is disseminated by the Office of Assessment and Accreditation. These elements are aligned with standards for educators with state and national professional associations, captured through assessments used for our SPA, InTASC, and CAEP reviews (5.1.4 Alignments of Assessments with Professional Standards, INTASC and CAEP).

There are four decision points at which a candidate is evaluated to progress in the program, based on meeting requirements through the multiple measures used to assess academic performance, content and pedagogical knowledge, and professional dispositions and ethics. The measures are as follows: standardized tests, content grades and course-based assessments, field-based assessments, evaluation of teacher mentors and supervisors, assessment of dispositions, and State-mandated tests for certification. The decision points, dating back from 2006, include “Induction” (Transition Point 1) when teacher candidates are admitted into the program, “Exploration/Synthesis” (Transition Point 2) before entry into clinical experiences, “Reflective Practice” (Transition Point 3) during clinical experiences, and “Professional Practice” (Transition Point 4) with completion requirements. This system allows for monitoring of candidate progress beginning at matriculation to the institution and are continuous throughout the educational experience into employment as a professional educator. All of the assessments—both proprietary and EPP-designed that are reviewed in Standards 1-4 for initial and advanced programs—are given in 5.1.5 Schedule of Data Collection, Processing, Reporting and Review of Assessments for Standards 1,2,3,4.

There is an effective data/record management system in place for recording, storing, and retrieving data on candidate knowledge, skills, and dispositions for assessments aligned with Standards 1 and 3 (for the EPP fall 2018 retreat) and on the candidates’ application of professional knowledge and skills in field settings for assessments aligned with Standards 2 and 4 (for the EPP spring 2019 retreat). Outcomes and trends are established for the retreats that involve program action plans that are data-driven. These are given in chart form in 5.3.1 Evidence Based Program Actions. With regard to the InTASC standards that inform Standard 1, the EPP analysis of the Learner and Learning multiple sources of data suggest that there is considerable variability in the candidates’ performance by program over the three years. It ranges from mostly high percentages (above 80% as target) on the Danielson Framework for intial and advanced programs, reaching target on the edTPA rubrics involving knowledge of students, to variable percentages of candidates reaching target on components of the EAS . It is important to note with the Special Education teacher-candidates the State acknowledges that the edTPA is more challenging for that certification. The conclusion to be drawn from this is that the assessments are quite varied in their structure, design and purpose, ranging from observation of teaching skills to multiple-choice items on case studies. Alumni responses to survey items aligned with the InTASC standards addressing learner and learning were low relative to the rest of the responses, particularly as related to assessment of diversity.

On InTASC standards-aligned domain elements related to content knowledge and application on the Danielson Framework, most candidates demonstrated high (90+%) and increasing proficiency over the three years. Similarly, most of the candidates (more than 90%) demonstrated target scores on the edTPA rubrics aligned with InTASC content planning across the three years with slightly lower scores on content learning and analyzing student language use. The content-based CST went from virtually 100% pass rates in 2015-16 to variable percentages of candidates passing the test (between 50 and 100%) with the revised version over the next two years. Content grades, across all programs and across all years, were above the 3.0 target. The multiple sources of data about content suggest that the EPP programs have effectively supported candidates in content mastery. Across the EPP, 77% of the program completers felt that their lessons could be aligned with the common core in 2015-2016, increasing to 90% in the following year, and down to 67% in 2017-18 (there has been confusion as to New York State’s position about the expectation to align curriculum and instruction to the common core standards from 2017 to the present day). About an equal number of program completers (between 73% and 81%) across the three years reported that they could include literacy/reading or mathematics/problem-solving in their lessons. Given the emphasis placed on literacy and mathematics content, the EPP faculty will meet by program during the fall 2019 retreat (for Standard 1) to discuss potential strategies for better supporting their candidates’ ability to integrate the content that they have mastered.

In terms of instructional practices, the teacher candidates in the EPP demonstrated the highest competence on these domain elements in the Danielson Framework (93% to 99%) across the three years. However, the breakdown of teacher candidates by individual program showed greater variation with some (Childhood Education, TESOL, Physical Education) demonstrating 90+% of candidates achieving target while others (Adolescent Education, Special Education and Art Education candidates) with considerable variability in reaching target over the same time frame (50% to 100%). Perhaps this is partially explained by the relatively lower enrollment in some programs, making cross-course comparisons difficult. Beyond this, there was a trend in the lower percentages for candidates in the programs involving higher-level, open-ended questions. The program faculty addressed this in the fall retreat as reflected in the 5.3.1 Evidence Based Program Actions.

Evidence from the edTPA rubrics aligned with instructional practices (13 of 15 rubrics) across the EPP and three years of data collection suggested strength in planning positive learning environments (97% to 100% of all candidates in all programs) and areas of challenge in assessment work—particularly in student use of feedback (between 65% and 73% reaching target). Several of the programs’ faculty in the EPP—in partnership with professionals in the field–have acted to strengthen assessment work in the courses and field experiences. Sixteen of the items on the Exit Survey aligned with instructional practices involving curriculum, assessment and instruction had percentages well above the criteria, ranging from 90% in 2016-17 for engaging students in critical thinking and in 2015-16 for understanding the social and emotional needs of all students to 80% in 2017-18 for using assessment results to guide instructional decisions. Items resulting in lower than 80% agreement among program completers included working with English Language Learners (58% to 65% over the three years) and developing literacy/reading lessons (76% to 81%).

The Danielson Framework for Teaching was used to look at our candidates’ competency in professional responsibility. The percentages of teacher candidates reaching target were the highest in comparison to the other CAEP standards with between 96% and 100% of the teacher candidates scoring at target across the three years. SCALE (2014), responsible for the design of the edTPA, viewed Professional Learning and Ethical Practice as aligned with analyzing teaching effectiveness and the use of assessment to inform instruction. Relative to other rubrics on the edTPA, these had lower percentages of teacher candidates across programs achieving target (while still above the criteria of 80%). The EPP is looking at the nature of these two rubrics which involve assessment and self-reflection. Other measures of these dimensions partially corroborate these findings on the edTPA rubrics aligned with professional responsibilities. The EAS, for example, involved competencies thought to align and in two of the three years reported, the Teacher Responsibilities mean for the EPP’s teacher candidates across programs was one of the two lowest on the EAS (2.2 in 2016-17, N=137; 2.4 in 2017-18, N=125). The Exit Survey results tend to support the view that professional responsibilities as an area to strengthen in the EPP, with mean scores relatively lower (3.81 to 4.08) in each of three survey years. Program actions related to these data are addressed in the paragraphs that follow.

There is a mechanism in development to support collaborations between the EPP and school/clinical sites to determine the terms, structure and content of field experiences, reflected in 2.1.4 Phase in Plans for Standard 2.1. During the annual spring retreat, minutes of meetings with programs and their advisory groups are taken, regularly discussing these elements of experiences hosted at the partner site. Minutes of meetings between EPP supervisor and teacher mentors reflect feedback to the EPP on candidate strengths/needs, and input is provided from the Teacher Mentor Survey on ways to strengthen the programs (2.2.1 Teacher Mentor Survey Data). There is an ongoing system in place to ensure that clinical placements occur with breadth, depth, duration and coherence with a predominance in diverse settings using the definition established by New York State. 2.3.1 Clinical Experiences by Programs in Diverse Settings offers a record of candidate placements, noting these elements. Explicit policies with regard to field experience time requirements, guidelines for what the field experiences should include and mandates for a placement in a diverse setting are addressed in the 2.3.4 Participant Observer Handbook and 2.3.5 Student Teaching Handbook.

There is a system being developed to manage recruitment initiatives attracting applicants from diverse and highly-qualified groups and in identified high-need areas and professional positions, described in 3.1.5 Phase in Plans for Standard 3.1. Currently, the programs use a monitoring system with current candidates on a semester basis with their grades, using a “flagging” system for those identified by faculty as in need of additional supports. There is a system being developed to collect and update contact information for alumni for a three-year period, as well, beginning with those who were contacted for the alumni survey in Spring 2018 and subsequently, for requesting them for their performance appraisals (APPR) for analysis of completers’ instructional practices and impact on P-12 student learning. The Office of Assessment and Accreditation (OAA) has piloted and collected alumni survey data for candidates in both initial and advanced programs, results offered in 4.4.2 Pilot Alumni Survey Data. In addition, an employer survey is sent out to key partners with results offered in 4.3.1 Pilot Employer Survey Data in both cases, the survey returns reflected the acceptable level for valid results, above 20% of those sent out. Program Actions will be generated from these data in the spring retreat by faculty.

Research and assessment of P-12 student learning is a component of this multiple measurement system in the initial programs that has been analyzed by the EPP. Overall, teacher candidates in the EPP reached the target score in these two areas on the edTPA rubrics above criteria levels (80% to 82%). The two programs with the greatest percentage of teacher candidates achieving target were Art Education (86% to 100%) and Childhood Education (88% to 89%). Several of the programs had percentages ranging in the 70-80% over the three years, including English Education, Math Education, and Physical Education. The Danielson Framework involves several domain elements that are aligned with research and evidence of the impact of their teaching on P-12 student progress, as well. While 96% and 100% of the teacher candidates scored at target on these domain elements related to research and assessment, several of the programs had relatively lower percentages in the domains and elements aligned with research and assessment, suggesting that these would be important aspects of the coursework and fieldwork in the curriculum for further review and action. This is corroborated by alumni and employer survey results, addressed below, where assessment and use of data to guide instruction were areas to strengthen in the EPP.

The support of college and career readiness skills are viewed as central to the Core Values of the EPP, addressing effective teaching of P-12 students’ critical thinking and problem-solving, transfer of these and other skills to other areas of learning, and collaboration/communication. The EPP views the skills and competencies as involving effective teaching of diverse student populations. Several of the programs had over 90% of teacher candidates reaching target scores on the domain elements of the Danielson Framework over the three years yet there were teacher candidates in programs that had lower percentages in higher-level questions, designing coherent instruction and managing classroom procedures. The percentages of teacher candidates in the initial programs achieving target over the three years on aligned rubrics of the edTPA were over 90% in establishing a positive learning environment, engaging students in learning, deepening student learning, planning for varied student learning needs, using student knowledge, and assessment to support learning across programs over the three years. However, the subject-specific pedagogy and use of assessment to inform instruction were more variable by program and suggested that more detailed review would be important to consider how to strengthen a range of competencies for teacher candidates that involved teaching of diverse populations. Combined with Exit Survey results that pointed to trends where teacher candidates felt equipped to respond to unique strengths/needs of students, adapting instruction, and understanding the social and emotional needs of all students while less prepared to address multicultural perspectives in the developed curriculum, bias of different forms, work with ELL students, and use technology to differentiate instruction has given the EPP areas to emphasize in actions across programs. Teacher candidates completing the program were also asked on the Exit Survey what methods they had the opportunity to apply in their field experiences or student teaching for diverse groups of students—ranging from different learning styles to exceptionalities to linguistic and cultural diversity—including alternative methods for literacy, grouping, individualized supports and assessments. Only the question about methods based on learning styles reached the criterion the EPP had established. Taken together, the EPP faculty and partner discussions are emphasizing how to support differentiated instruction in diverse settings with the use of assessment findings, grouping practices and individualized instruction.

Other measures of the system that can support the candidates in the EPP are to be developed and strengthened with school and professional partners described in the phase-in plans, including the teacher mentor survey and the Mid-Point Assessment for the initial programs, and the assessment of data literacy, research, and application of these in clinical settings across the EPP in the advanced programs. The integration of technology in the programs has become an important priority in the assessment system, and a unit wide Effective Technology Use rubric is being validated and piloted for use during content and methods courses and finally during student teaching.

Standard 5.2

The EPP’s faculty has sought to ensure the relevance, verifiability, representativeness, and cumulative and actionable measures in the quality assurance system, offered in 5.2.1 List of Assessments and Validity/Reliability Outcomes. These assessment results and actions taken to establish their measurement qualities confirm the legitimacy of the candidates’ preparation to meet CAEP standards while addressing both the institution’s strengths and areas to be strengthened. Noted in the 5.1.4 Alignments of Assessments with Professional Standards, INTASC and CAEP is whether the assessment is proprietary or not, the CAEP standards it has been used for evidence. EPP created assessments are tested for reliability and validity to ensure that each instrument used in confirming candidate effectiveness meets the sufficiency level described in the CAEP Assessment Evaluation Rubric. Individual evidence documents contain the processes used to establish reliability and validity. EPP faculty, PK-12 partners and candidates participated in establishing the reliability and validity of EPP-created assessments. Work to accomplish this task included members of the Assessment Committee and the CAEP Core Team.

Throughout the self-studies, evidence is tagged to the appropriate CAEP component, InTASC standard and/or cross-cutting theme. Data are representative of candidates in the initial or advanced programs separately, and in all cases, data are disaggregated by undergraduate/ graduate programs and location (Garden City vs. Manhattan) and compared to state and national benchmarks. Three cycles of data have been presented where possible, and in the case of phase-in plans for initial programs with new assessments, data from a previous assessment cycle has been included to meet the CAEP requirement. Interpretations of the data that have been analyzed are included in Standard 5.1, along with any data-driven actions taken by the EPP in response to the data analysis. These interpretations include any trends or patterns identified in the data. Evidence charts for both initial and advanced programs are given in 5.1.1 CAEP Matrix of Evidence for Initial Programs and A.5.1.2 CAEP Matrix of Evidence for Advanced Programs.

While EPP-wide assessments have been the norm, the requirements for specialty professional associations have led to innovations by individual programs in their assessments that must be accounted for in the quality assurance system. This has been particularly true in the assessments used for the induction and exploration/synthesis points of the programs. For example, all programs have consistent criteria for admission into the program (see 3.4.2 for Admissions Criteria and Rubrics by Program) but criteria may be aligned to professional standards without also being aligned to an EPP-wide standard. Reviewing and revising these criteria are part of the phase-in plan for both initial and advanced programs (see 3.3.3 Phase-In Plans for Standard 3.3). This is also the case with a Mid-Point assessment that was the subject of study with EPP faculty in summer 2018. 5.2.1 List of Assessments and Validity/Reliability Outcomes – Attachment 3 describes this which was to validate the accuracy of content of a mid-point assessment for the initial programs, drawn from the key assessments of the programs’ specialty professional association reports, and to ensure that the candidate samples drawn from them were accurately assessed with a corresponding rubric. Other goals were to collect data on inter-rater reliability on the assessments and to provide data for the mid-point assessment of InTASC standards as a second assessment besides the Teacher Work Sample (TWS). EPP faculty submitted 46 samples of candidates’ work, part of the previously-submitted SPA reports. Two questions were posed, with accompanying scores from “1” (inadequate or inaccurate) to “4” (exceptional or complete accuracy): 1) does this rubric measure the topic that it is supposed to measure and 2) is the sample score accurate based on the rubric. The argument was that the first question addressed content validity and the second, reliability. The construct validity was found to be high (79%) and the reliability was higher (86%).

The TWS was developed by a consortium of EPPs called the Renaissance Group that sought to design a curriculum planning format, aligned with professional standards. The EPP has used this assessment for a dozen years as a formative measure that allows for goal-setting, use of multiple assessments for P-12 students, instructional designs based on goals, needs and learning contexts, systematic evaluation of student learning and use of data to communicate student progress and achievement, with candidate reflection on the teaching and learning process. The eight components of the TWS have established rubrics that are part of the data analysis in Standard 1. The measure is aligned with the Danielson Framework and the edTPA as summative assessments used by the EPP. The EPP faculty and clinical educators used the Lawshe’s Content Validation tool to validate the instrument. They read three TWS standards and tasks: 1) the standard is that the candidate designs instruction for specific learning goals, student characteristics and needs, and learning contexts with a task of providing two examples of instructional decision-making based on students’ learning or responses; 2) the standard is that the candidate uses assessment data to profile student learning and communicate information about student progress and achievement with the task of analyzing assessment data, including pre/post assessments and formative assessments to determine students’ progress related to the unit learning goals and using visual representations and narrative to communicate the performance of the whole class, subgroups, and two individual students; and 3) the standard is that the candidate analyzes the relationship between his or her instruction and student learning in order to improve teaching practice with the task of reflecting on performance as a candidate and linking it to student learning results, evaluating performance and identify future actions for improved practice and professional growth, then deciding if each is essential, useful and/or necessary for teacher candidates. Based on the Lawshe tool, the results included content validity ratio scores of 1.0 for the first question, .56 for the second, and .78 for the third. The mean content validity ratio was .78. An inter-rater reliability study will be conducted with the TWS during the Spring 2019 semester, prior to the site visit.

A particularly important assessment in the EPP that has undergone several changes since 2006 is the dispositions assessment. The faculty have sought to align it with the Core Values, professional standards, and the input of professional partners who oversee the candidates’ experiences in the field. After some early attempts, the EPP faculty undertook construct validation work in summer 2014 with a set of pilot data, moving from four factors on the dispositions to three based on psychometric results: 1) fairness/growth mindset/grit; 2) respect/responsibility; and 3) positivity/reflectiveness. Another set of pilot data was collected about candidates during student teaching in Spring 2015 with the Assessment Committee and Fieldwork and Educator Preparation Committee holding meetings about the process of validating a field-based instrument with the goal of collecting data on the candidates’ pre-service ability to teach; to use a criterion or reference point against which to correlate dispositions. Correlation analyses were run among identified items on the Danielson Framework and the dispositions assessment with the result of some significant correlations but most were less than .30, suggesting some similarities (particularly between the dispositions and Danielson’s domain of professional responsibility). The faculty then sought to examine the assessment while using the Danielson Framework as an outcome variable in a regression analysis. It was found that the regression was significant, with 10% of the variance on the Danielson Framework predicted by GPA and the dispositions assessment (a review of the beta weights revealed that only the dispositions assessment was significant in the regression). In a fixed effect regression study, the faculty looked at candidate effectiveness from supervisor ratings in relation to candidate knowledge, pedagogical skill and dispositions and found that pedagogical skill was the most weighted in overall ratings.

In fall 2015, a study was conducted assessing internal validity of factors in the dispositions assessment and results confirmed three factors among 13 items. With 165 candidates, the overall reliability alpha values was .924. In spring 2017, 161 candidates being assessed on the dispositions tool had comparable alpha values. However, a Dispositions Subcommittee of the Assessment Committee noted that the items’ face validity did not warrant administration in both class and field settings as a formative measure. The decision was made to have three assessments of dispositions during the program: during select classes and related field experiences and again during student teaching. In spring 2017, 175 teacher candidates were assessed in the classroom and 98 in the field. The means for the three factors were above 4.0 on a 5-point scale and the means of field-based dispositions were slightly higher, yet all were positive. As a unit, the average student teacher was rated a 4.75 (out of 5) on Fairness, a 4.70 on Respect, and 4.78 on Positivity. By program, none of three areas scored lower than 4.20.

Since then, the EPP faculty (through SPA report feedback and from the field) decided that the construct validity of the three factors on the current dispositions tool should be conducted again with the possibility of a revision of the measure (see Phase-in Plans for 2.1.4 and 3.3.3).

In a factor analysis, the classroom dispositions loaded three factors that were different in item weightings from the last analysis. The field-based and student teacher factor analyses were unreliable as the disposition items were all correlated with one another. In addition, the Field-Based dispositions have a great number of Not Applicable items, which was treated as missing value, further confounding the analysis. Student teaching and field-based dispositions data were combined for factors to load. However, the results are not reliable, since correlations revealed once again all of the items are related to one another. The field-based instrument is not allowing correlations across disposition type or to do an analysis of variance to compare the means of the students across time. The instruments do not have a common basis to proceed. The EPP faculty will meet in two spring retreats to contribute to a Lawshe analysis of the dispositions and to review the results. The EPP with its partners will seek to redesign the dispositions to reflect the identified professional responsibilities, ethics and attitudes for effective teachers.

The Danielson Framework has been a part of the EPP assessment system for a decade and has its own validity and reliability measures that has helped to make it a preferred measure for assessing teachers’ planning, instruction, classroom management and professionalism. The EPP has had several years of adding items to the Framework to look at areas more directly aligned with the Core Values. In addition, there have been modifications of the rubric used to assess teachers in New York State and the EPP, out of concern for relevance of its measures, decided to undertake a study of the validity and reliability of the Framework of its own.

At the first spring 2019 retreat, EPP faculty gathered by program to conduct an initial validation of the Danielson Framework for Teaching indicators. Program faculty discussed each Danielson indicator aligned with INTASC standards and commented individually on each item. Using Lawshe’s process on content validity, Danielson indicators were determined to be either essential to teacher candidate’s knowledge and skills, useful, but not essential, or not necessary. Outcomes of the faculty input were analyzed using Lawshe’s Content Validity Ratio revealing a Mean Content Validity Ratio (CVR) of .656. Since the total number of faculty exceeded 15 (N=24), the acceptable ratio for determining valid items was .49. The EPP mean CVR of 0.656 showed that the Danielson Framework was a valid tool to measure teacher candidate competencies in the field. Since this was the first administration of the Lawshe validation process, faculty found that several Danielson indicators showed outcomes that would suggest faculty revisit the individual items and either re-word or eliminate specific indicators. The next step in the process was to have the Assessment Committee review the Danielson Framework to re-tool the items and bring forward to the next EPP faculty retreat for further validation.

Reliability of the Danielson Framework was conducted by inter-rater agreement. Ratings between the teacher candidates’ Teacher Mentors in the field and their University Supervisors on the Summative Danielson Framework rubric yielded agreement across the 16 Danielson indicators aligned with INTASC standards as fair to moderate agreement (correlations of between .20 and .50) with the exception of three items which showed slight agreement, “plans with knowledge of student’s background, skills, and interests,” “assesses student learning,” and “reflects on teaching.” The items with the strongest agreement were, “selects appropriate instruction goals, “designs coherent instruction,” and “engages students in learning.” Within all four levels of Danielson’s domains, there were no significant variations in agreement between the Teacher Mentor and University Supervisor on the Summative Danielson Rubric that exceeded more than one performance level rating. For example, there were none too few cases of items where a Teacher Mentor (or University Supervisor) would rate a teacher candidate’s performance as “Distinguished” while the same candidate’s performance on the same measure would be rated as “Basic” by the University Supervisor (or Teacher Mentor). The same were true for “Proficient” and “Unsatisfactory.” The EPP faculty will meet in the second spring 2019 retreat to review the results from the field.

Several of the assessments given as measures during the reflective practice point of the programs in the EPP are proprietary and had validity and reliability studies undertaken at state and national levels that are addressed in Standard 1: the edTPA, the EAS and the CST are all high-stakes tests that must be completed successfully for program completers to become certified in New York State.

The assessment system for the advanced programs has been a challenge as the distinct professional disciplines involved (teaching English to speakers of other language, special education, literacy support, and health education) must involve relevant and representative measures of candidate knowledge and skills while allowing for generalized or EPP-wide outcomes about the effectiveness of the curriculum, faculty and supervised field experiences. The EPP has begun to develop such an assessment system with the assessments that reflect the focus on data literacy, research, technology and collaboration.

Standard 5.3

In the Spring 2019 retreat, the EPP faculty identified the specific evidence that each program had that led to program actions and improvement (see 3.1.6 Program Actions for Recruitment and 5.3.1 Evidence Based Program Actions), overall, as well as the findings from the Danielson Framework, the edTPA, the EAS and the CST. They were asked by the Core Team to identify what other measures were important for program improvement, and how the data was used to inform their decisions. The EPP faculty were also asked to identify the specific program actions at the exploration/synthesis and reflective practice points of their programs. Questions related to the measures and evidence used for admission to determine teaching potential and for assessing candidate growth during student teaching were also posed by the Core Team at the retreat. All of these were an effort to gain additional data points for the quality assurance system. The EPP faculty responses are given in a comprehensive chart at 5.3.1., summarized here.

While the SPA reports for the programs have yielded data analyses and program actions that are addressed elsewhere in this submission, this process allowed the EPP to review the evidence and program actions overall to address the quality assurance system. The SPA data (specifically, key assessments), New York State exams for certification (including the edTPA, EAS and CST), the Danielson rubric scores, and the dispositions were identified by EPP faculty as central to evidence that is used for program improvement. Changes based on research, new standards, and student feedback were also identified as important sources of evidence for program improvement. However, only one program noted that Danielson measure was central to their program actions in instructional practice, in particular. Four of the programs noted the edTPA, on the other hand, as being an important source of evidence. Three of the programs identified the EAS as important and four noted the CST as important in program actions. Other measures that identified as important by the programs included more systematic and thorough evaluative feedback from school partners, mentor teachers and supervisors, data on technology use and needs, dispositions, and candidate feedback via their work, informal assessments and their feedback. Both initial and advanced programs in the EPP are working towards regular and systematic assessment of candidate performance with their respective standards in mind, tracking the results across transition points and using results to improve the programs, as indicated in the respective SPA reports.

When EPP faculty were asked how evidence informs their decisions, several program responses included articulating gaps in professional expectations, sharing of data among faculty to identify areas of strength and weakness while addressing the meaning and significance, monitoring change and progress along with justifying changes to curriculum and program requirements. One program noted that there is a “‘continuous data feedback loop’ for improving courses, building partnerships…” and gaining feedback from the partners about program actions. Another program noted the ongoing revision of key assignments in coursework and adaptation of proprietary assessments to meet the standards of the profession.

The question about admissions criteria and the potential for teaching led most of the EPP faculty to respond that the essay question, recommendations, interviews and grades all offer some insight into potential. It was noted that these do not show clear alignments with the InTASC or CAEP standards and several programs suggested that a more detailed or augmented process (with interviews and written responses) might be helpful. One program offered the analyses of this as related to admissions into residency programs, for example, which are more aligned with the standards. It was conveyed that candidates are given prompts from the EAS involving scenarios of classroom practice with diverse populations. It is a point of future discussion in our phase-in plans for the EPP to consider to have the admissions criteria more aligned with this process.

EPP faculty are engaged in program actions related to specific points of their programs. For example, they commented on the exploration/synthesis point as involving a “reimagining” of the curriculum with the expansion of field experiences and initiatives as well as State examinations. One program noted the importance of more thorough evaluation of dispositions, grades, written expression, and mathematical content background.

It was also commented on that the disposition scores, technological skills and health education skills would also seem to assist in these formative phases. There were repeated comments about greater integration of field experience with course work (whether through mentoring or assignments), and this represents a topic that is addressed in the phase in plans for Standard 2. In the reflective practice point of the programs, faculty in the EPP noted that the evidence suggested more systematic partnerships with mentor teachers and administrators through, for example, professional development opportunities, advocacy and interaction with parents/communities, as is greater feedback from them about the curriculum.

Standard 5.4

The EPP faculty has always valued contact with its alumni and, over the years, the relationships with them has been a key strategy for the growth of the programs. Until recently, the direct contact with larger numbers of alumni has been routed only through the Alumni Affairs Office at the University, thereby making coordination more challenging. That has also been the case with regard to collection of evidence of employment and surveys of program impact, which have been undertaken by the University’s Office of Research, Assessment and Planning. The EPP has had a flourishing association through Phi Delta Kappa, dating back 10 years, that has annually increased its alumni membership. In 2017-18, the alumni were formally invited to be a part of an organization known as Panthers on the Prowl, the first EPP-wide association of its kind that now boasts contact and participation of alumni. Surveys of alumni and employers have been conducted on a regular basis but the ability to contact either group has only recently been more comprehensive and effective. The outcome data on P-12 student growth has been part of the New York State Annual Professional Performance Review (APPR), described in detail in the 4.1.4 Phase-In for Standard 4.1 and A4.1, is a New York State-wide system of evaluating teachers, leaders and other school professionals. The APPR data includes student achievement standardized test data and Student Learning Objective scores (40% of multiple measures, including teacher observations which are often the Danielson Framework) (Source). Within these general guidelines, each district submits a specific plan for their district. See specific district plans. In a document issued by the New York State Regents for the State Education Department in May 2018, they note that, “…these assessments are used for high stakes purposes (and) meet standards for validity and reliability under the Standards for Educational and Psychological Testing…” (Source).

In 2017-2018, requests were sent out to the New York State Education Department and to the New York City Department of Education for APPR scores on alumni from the EPP, along with number of high needs licenses and retention rates after 1-3 years of hire. The New York City data on 242 EPP alumni from 2015-2018 were received and included in 4.1.3 NYC-DOE 2014-2017 Data. The State scores have not yet been received as the data is being cleaned (but should be available by the time of the site visit in November 2019) and the EPP Dean’s Office and CAEP Core Team reached out through Panthers on the Prowl and alumni from each of the programs identified by faculty for information to obtain APPR scores. Another 14 alumni provided these APPR scores voluntarily. The programs of program completion have been identified but the breakdown of the APPR scores by student achievement and growth as well as teacher observations is not available.

The alumni and employer surveys issued by the EPP in 2018 have yielded over 20% return rates established by CAEP as the acceptable level for evidence, including alumni for the past three years. The survey instruments are scheduled for review by the established partners in 4.2.4 Phase in Plan for Standard 4.2 and 4.3.3 Phase in Plan for Standard 4.3.

Standard 5.5

An area that the EPP has identified for upgrading is involving stakeholders in the retreat work undertaken by the faculty for reporting and reviewing EPP assessment results and recommending changes. While data discussions were already occurring with these stakeholder groups, the EPP recognized data analysis needed to be much more in-depth to increase the richness of these discussions, and a much more focused meeting time was needed. Another related change is to strengthen the work between the Assessment Committee of the EPP with the Office of Assessment and Accreditation within the EPP, providing a bridge between programs’ data collection and the analyses and actions to be taken within the unit. The EPP and the programs collect and analyze a considerable amount of data. To have a Committee to help lead the EPP’s assessment efforts is an important part of the EPP’s quality assurance.

In addition to the direct and indirect measures cited to evidence the candidate and completer, the University Office of Research, Assessment and Planning annually provides department chairs and dean with indicators that include year-to-year comparison and trend data for metrics relevant to outcome and output variables. These include student headcount, student credit hours generated, full-time equivalencies, faculty load indicators, student headcount to faculty ratios, student credit hour to faculty ratios, first-to-second year retention, completion outcomes, cost per credit hour and cost per student to deliver instruction at the program level. These indicators serve to inform the department and programs on an annual basis and serve to support the work on SPA reviews as well as annual accreditation and Title II reviews. This helps to support stakeholder involvement in EPP evaluation, improvement and identification of excellence. 5.5.1 SOE Advisory Meetings and Partner Meetings that Demonstrate the Quality Assurance System shows dates, topics, co-construction of instruments/evaluations (see 2.1.3 Partner Activities that Demonstrate Co-Construction), criteria of selection of teacher mentors, involvement in ongoing decision-making, input in curriculum development, and EPP and P-12 educators providing feedback to candidates from 2015-2018. This is followed by back-up evidence of minutes. Across the EPP, the predominant category has been in ongoing decision-making. It will be important for the quality assurance system to enhance the partner relationships’ participation in these areas.

Phase-in plans address how practitioners serving as mentor/cooperating teachers will belong to advisory groups for each program within the EPP and, currently, there is an ongoing EPP Advisory Board that helps to oversee the programmatic and systemic support of candidates. Retired practitioners are hired to be clinical supervisors. Advanced programs have professional partners helping to support candidate placement, field experiences and curriculum. The evidence documents previously identified provide a number of examples of stakeholder involvement in decision-making, program evaluation and selection and implementation of changes for improvement. Without question, the residency programs that have begun to flourish within the EPP (with multi-year partnerships), are supporting stakeholder participation in program evaluation and improvement. The EPP seeks to support an increasing reliance on the input of its stakeholders for continuous improvement.

 
Follow Adelphi Education at: