CAEP Standard 4 – Program Impact


Standard 4.1

In 2013, it was reported by the Wall Street Journal that Adelphi was one of two EPPs whose graduates had the highest student achievement scores on New York State high-stakes tests in English and mathematics, 4th through 8th grade, in the New York City system (4.1.1 NYC DOE 2008-2012 Data). Adelphi faculty have significant concerns about tying teacher effectiveness to student test scores, as do many educational researchers including Pivovarova, Amrein-Beardsley & Broatch (2016), who challenge such practices as inequitable and unjust. However, we realize that the outcomes that can be used to determine the impact of teaching on learning are limited. In order to document student-learning growth, we sought the cooperation of the New York State Education Department and public school district partners, including the New York City Department of Education to supply data on the Annual Professional Performance Review (APPR) from alumni who were teacher candidates in our EPP from 2014-2017. The APPR is a composite of P-12 growth and performance on State high-stakes tests and observations by supervisors—largely using Danielson’s Framework for Teaching. Ratings are given in a rubric format that designate teachers as “highly effective” (“4”) , “effective” (“3”), “developing” (“2”), or “ineffective” (“1”). Target is established by New York State and the EPP for this report as “effective” or “highly effective.” (4.1.2 APPR Instrument).

The data that we have collected thus far is from the New York City Department of Education, which provides a rich data set of APPR scores for 228 Adelphi completer teachers, compared with the average of over 14,000 NYC teachers (4.1.3 NYC-DOE 2014-2017). This data indicates that Adelphi trained teachers are well-prepared to enter the classroom as effective teachers. The percentage of alumni who achieved an effective or highly effective rating in the City system has increased from 84 to 94% in the three years from 2014-2017, virtually the same percentages and increases as the City teachers overall. However, more (10% vs. 7%) Adelphi alumni achieved a highly effective rating. The retention rates of alumni as teachers over a 1-3 year period (based on entry into the system) are comparable to the City teachers. Overall, the remaining 90% for one year and then lowering to 79% after three years. Data was shared with faculty in Fall 2018 for analysis and interpretation.

In addition, the EPP has developed a phase-in plan (4.1.4 Phase in Plan for Standard 4.1), which will include an effort to collect the supervisor observation scores, an essential component of the APPR scores, directly from recent alumni. We plan to conduct a study of the relationship of the Danielson Framework observations when they were near program completion while student teaching in relation to the supervisor scores (which also use Danielson-based observations) as teachers. From a research standpoint, we would then be able to empirically study how the student teaching experience was correlated or predictive of their observed effectiveness as teachers later on. Part of the plan (which as a pilot in Spring 2018 included 12 alumni who offered supervisor observations) is to conduct a “deeper dive” into their experiences in the EPP’s course and fieldwork, culminating in the observation by clinical supervisors when student teaching, and their subsequent evaluated effectiveness through their teaching supervisor.

Table 4.2.2 provides Pilot data, which demonstrates that alumni teachers receive consistently good scores on observations (Danielson element scores range from 3.3 to 3.8).

Standard 4.2

The first source of data regarding the effectiveness of alumni teachers is collected from the New York City Department of Education, which provides a rich data set of APPR scores for 228 Adelphi completer teachers, compared with the average of over 14,000 NYC teachers (4.1.3 NYC-DOE 2014-2017). Between 30 to 60% of the APPR score is based on supervisor observations. Because of the way this data was received from NYC DOE, the EPP was unable to disaggregate observation scores. However, the APPR scores of Adelphi teachers demonstrate that they are well-prepared to enter the classroom as effective teachers, as discussed above.

The clinical supervisors offer an assessment of the advanced candidates’ work through their field experiences, with a sample of the observation tool and report depicted in 4.2.1 Sample Supervisor Observation Tool and Report. The second source of evidence is 4.2.2 Pilot Administrator Observation of Program Completers Data, as discussed in 4.1, which demonstrates that alumni teachers receive consistently good scores on observations (Danielson element scores range from 3.3 to 3.8). This table offers pilot data of administrator observations from 12 alumni, who are graduates of 5 initial programs. Our (4.2.4 Phase in Plan for Standard 4.2) phase-in plan discusses EPP plans to provide data on program completers from all programs.

The EPP’s faculty made a decision in 2006 to utilize the Danielson Framework for Teaching as its main source of observational data with teacher candidates’ clinical performance of professional knowledge, skills and dispositions. The Framework was adopted by public school partners around New York State in large numbers. It seemed to be reasonable preparation path for our teacher candidates and to ensure that our clinical experiences supported those expected in the school settings after program completion. For the 4.2.4 Phase in plan in Standard 4.2, the Danielson Framework that is the basis of supervisor observations were collected from alumni APPRs and will be assessed in relation to the observations made during their studies in our programs. A key research question for us, raised for several years in our Fieldwork and Educator Preparation Committee, is whether the Danielson Framework is appropriate as an observation instrument for preservice teacher candidates in the field.

Finally, Pilot Employer Survey data (4.3.1 Pilot Employer Survey Data) demonstrates that employers judge Adelphi graduates to be effective teachers. When asked to rate the overall effectiveness of teachers who are Adelphi grads, the mean was 4.31 on a 5 point scale, indicating that the majority of employers rated them between very and extremely effective. The employer survey will be discussed further in 4.3.

Standard 4.3

While we have emphasized employer feedback about our alumni over the past 12 years, we have tended to ask about the program’s support of those teachers they have hired in more general terms. In 2017-18, it was determined by the Fieldwork and Educator Preparation Committee and Associate Dean, based on survey results, that a revised survey be developed that was more specific about the behavioral competencies of our alumni in school settings. The revised survey has 14 items, ranging from content and pedagogical knowledge, treating students all equally, demonstrating a commitment to professional growth to the communication with caregivers. Employers are asked to rate each of the items on a four-point scale. However, the survey has not been shared with our partners for their feedback. In addition, our survey’s yield (13 responses, which is 24%) is acceptable for accreditation standards but we sought more input and will develop a system that involves an expanded list of employers (now 55), part of our 4.3.3 Phase-in plan for Standard 4.3.

The results of the survey, which was administered in Summer 2018, are provided in 4.3.1 Pilot Employer Survey Data. We established target as a rating of “proficient” (“3”) and “distinguished” (“4”) on the items. The mean and standard deviation for items is given on the table, along with some qualitative comments. All 14 items on the survey were rated above an average of 3.0 (proficient). At least 90% of the employers rated the alumni as proficient or distinguished on almost all items. Three comments from school administrators detail our alumni’s strengths, drawn from the survey results: “Generally graduates are well prepared to immerse themselves into a school community, and do well in the classroom.” “They have a good understanding of the need to differentiate instruction and adjust instruction to meet student needs. Very student centered.” “Teachers are better prepared take over a classroom with authority. Teachers are more aware of pedagogical practices and curriculum.” The one item with lower ratings involved understanding how to interpret data to identify gaps in student learning, where 23% of the alumni were rated at the “basic” level (“2”). A comment from a school or district administrator on the survey corroborated this and is an area for us to address in our coursework. We recognize that teaching skills takes time and experience to develop and address a plan to focus more on these results in our pedagogy courses.

Another source of data on employer satisfaction comes from the Employment Follow up Survey (4.3.2 Employment Follow-up Survey). This survey sent to completers six months post-graduation, asks alumni to self-report whether they are employed in education, whether they are full or part-time, and their current position. Alumni also provide information about the school in which they are teaching. In 2016, the survey was sent to 175 graduates, 48 replied (27% response rate), 88% of graduates were employed in the field of education. In 2017, the survey was sent to 146 graduates, 31 replied (21% response rate), 88% of graduates were employed in the field of education. In 2018, the survey was sent to 126 graduates, 50 replied (40% response rate), 100% of graduates were employed in the field of education. While this data provides a somewhat indirect indicator of employer satisfaction, the EPP judges it as important, given the tight teaching market in the Long Island and greater NYC area. This data also provides specific information on where graduates are employed, enabling the EPP to maintain contact and develop connections with alumni teaches in surrounding schools.

Standard 4.4

Over the past 12 years, an annual survey to alumni has been distributed in coordination with the University’s Office of Research, Assessment and Planning (ORAP, which has its own survey) and the Alumni Relations Office, which has helped to provide updated contact information. In 2017-18, a new Alumni Association for the EPP began, called Panthers on the Prowl, that has provided a more regular and consistent contact with alumni in regular meetings and communications. The Association was begun by the Chair of the Fieldwork and Educator Preparation Committee and Associate Dean who also initiated a revision of the current alumni survey in Spring 2018, with an eye towards aligning it more with InTASC and CAEP Standards. The survey provides feedback of recent alumni, between one and four year post-graduation about their perception of the teacher education program and whether that preparation was effective to prepare them for the responsibilities they face as P-12 teachers. The survey has not yet been not reviewed by our partner schools and districts, which is part of the rationale for this 4.4.3 Phase-in plan for Standard 4.4.

The survey was sent to a final list of 380 alumni who had provided a permanent (non-Adelphi) email. In 4.4.1 Alumni Contact Information, there is a spreadsheet of Alumni Contact Information (scrubbed of all contact and identifying information). Survey Monkey was used to collect survey data through emails to alumni. Eighty-three of the 380 alumni completed the full survey, for a 22% response rate. The 83 alumni who responded to the survey represented all teacher education programs. The majority of responses were from Adolescent Education (21%), Childhood Education (20%), Childhood Special Education (18%), and Physical Education MA (6%). However, most of the EPP teacher education programs were represented in the results. A third of the alumni who responded are employed in Long Island districts; another third are in New York City schools. Over half were the regular classroom teacher and the same number not yet tenured.

Exhibit 4.4.2 Pilot Alumni Survey Data provides a summary of the results at EPP and disaggregated program level. The data address employment status, the types of settings and functions in their professional roles, and evaluation of the programs and field experiences they had while teacher candidates in the EPP. Forty-five percent of the respondents were satisfied or very satisfied with their educational experience while 80% (the established target percentage) were satisfied or very satisfied with their student teaching experience preparing them for their teaching roles after program completion. Over 80% of the respondents indicated that they were teaching in their content areas, work with diverse populations, use varied instructional approaches, work with special needs populations, use technology for learning purposes, collaborate with colleagues, differentiating instruction and planning lessons. The strongest area of agreement (with over 90% of the respondents) was in being a reflective practitioner—one of the core values of the EPP.

While many of the comments from those completing the survey addressed how pleased they were with their educational experience in the EPP. Many commented on the need for better preparation for the careers in diverse settings including New York City. The comments were corroborated by a focus group held in summer 2018 where several alumni addressed the need for better preparation to work with diverse populations. Results of the alumni survey have been shared with faculty members, but have not yet been shared with school partners. 4.4.3 Phase-in Plan for Standard 4.4 provides details about next steps.

Standard A.4. Program Impact (Advanced Programs)

A.4.1.3 Phase In Plan for Standard A.4.1

A.4.2.3 Phase In Plan for Standard A.4.2

 
Follow Adelphi Education at: