Leveraging Completer Data for Program Improvement: Analyzing Multiple Measures Using a Plan-Do-Study-Act Model

• In Event: 21.035 - What Is the Impact of Teacher Education? Research on the Effectiveness of Completers During Induction

Abstract

Purpose
In recent years, the expansion of teacher evaluation systems, including value added measures, drives Educator Preparation Programs (EPPs) to utilize completer data as an evidence base for continuous program improvement. New national standards explicitly require EPPs to demonstrate the impact of their graduates on P-12 student learning. These requirements further the need for valid and reliable data for program improvement efforts (CAEP, 2013).
In North Carolina, EPPs receive data for graduates working in the state's public schools to evaluate alongside EPP data. In this paper, we employ improvement science principles for analyzing completer data collected during the induction period (the first 3-5 years of teaching) to inform EPP program improvement efforts.

Theoretical Framework and Literature
Our EPP recently implemented new program elements and we are eager to assess the impact of those changes using an improvement science model, plan-do-study-act (PDSA) (Langley et al., 2009; Lewis, 2015). For this analysis, we employ short PDSA cycles seeking to make iterative refinements to the new program as it is implemented (Lewis, 2015).

Methods
We identified completer measures which, when analyzed together, provide valuable data from graduation and the induction period for evidence-based program improvement efforts. They are:
1. EPP Graduate/Exit Survey: collected by EPP at program completion
2. Recent Graduate Survey (RGS): collected by state at end of first year of teaching
3. Teacher Effectiveness Data: collected by state through the first three years of teaching
Across all three measures, there is strong alignment with InTASC and the five NC Professional Teaching Standards (NCPTS; NCDPI, 2013) used in the NC Educator Evaluation System. Through iterative textual analysis of survey data and comparison of mean scores across similar domains, we identified themes across the three datasets.

Findings and Action Steps
Content knowledge is an EPP strength. Content knowledge preparation was highly rated on the EPP Exit Survey and the RGS which aligned with the high principal ratings evidence in the Teaching Effectiveness data (NCPTS3). As an improvement step, the program is seeking to leverage content level expertise across the middle-secondary range. With these actions, candidates become licensed in multiple grade ranges within their content area and become more marketable when seeking first teaching positions.

Relationship building is valued. Teacher candidates report favorable preparation to develop positive relationships with students, families, and school communities. These feedback data align with favorable principal ratings on NCPTS2. These findings support the EPP’s equity and social justice mission and provide avenues for additional inquiry and improvement. As an improvement action, the EPP decided to seek additional school and community agency partnerships to offer a broader and more diverse set of field experiences.

Meeting the needs of diverse learners. Across the EPP Exit Survey and RGS, teacher candidates reported less than adequate preparation to meet the needs of diverse learners. These findings aligned with lower principal ratings in Teaching Effectiveness data for NCPTS4. Armed with multiple PDSA cycle data, the EPP developed new coursework specifically focused on supporting ESL/ELL students and special needs students leading to add-on licensure pathways in the EPP.
 

Authors

• Diana Lys, University of North Carolina - Chapel Hill

• Alison LaGarry, University of North Carolina - Chapel Hill

• Wenyang Sun, University of North Carolina - Chapel Hill