Formative Evaluation of the Process of Scaling Up Reading Apprenticeship

• In Event: 61.042 - The Evaluation of Reading Apprenticeship for Improving Secondary Education

Abstract

We will present the logic model, evaluation design, and initial findings from the formative evaluation of the process of scaling up Reading Apprenticeship (RA). This five year study spans four states; the overall goal is to understand how school systems build capacity to implement and disseminate RA, and sustain these efforts.

In developing the logic model guiding this evaluation, the authors built upon literature emphasizing the “shift in reform ownership” from the program developers to the local level (Coburn, 2003). The research questions address the spread of the initiative, the scale-up process through the stages of the logic model, and contextual factors that affect scale-up. Researchers use a mixed methods approach. In Years 1-3 of the study, researchers observed and documented key project activities; tracked the numbers of schools, teachers, and students served; surveyed participating teachers and administrators; and conducted case studies with four RAISE schools.
From our analysis of teacher survey data from the first cohort of RAISE schools, we found that participation in key RAISE activities and “scale-up” outcomes, such as buy-in and commitment to the initiative, decreased between the first and second year. These results, however, should be viewed in relation to the already high levels reported in the first year. One of the most obvious decreases in the second year was teachers’ attendance at the monthly team meetings. Furthermore, we did find a statistically significant positive relationship between the change in commitment to RA and attendance at the monthly meetings, as well as between change in buy-in and commitment and average use of RA practices. These findings suggest that SLI should consider how to encourage continued participation in the out years or offer alternative ways to support social networks or professional communities of RA teachers. Given the number of schools, districts, and states involved in the scale-up, building an online community or offering recurrent “refresher courses” may be a more efficient and cost-effective way of supporting new learning, multi-site collaboration, and a deepening of practice.

During the five year grant, RA will be scaled-up with multiple cohorts, across years, states, and subject areas, in complex educational contexts. To date, SLI has trained over 1,300 teachers from 179 schools, across the four scale-up states. Our current findings of a general decrease in key indicators of participants’ uptake of RAISE activities and scale-up outcomes between the first and second year of the first cohort, further lead us to investigate what contextual conditions may affect the scale-up process, and we will continue to explore if the process or the outcomes differ based on these factors. Examining these trends for teachers in the third year of implementation will be important. If the changes “level off”, it may suggest that the process has reached a natural level of enthusiasm. If we continue to find a decrease, we will further explore what factors are associated with the decrease. We will also have the opportunity to measure these trends across two years of implementation for a second cohort of teachers.

Authors

• Jenna Lynn Zacamy, Empirical Education Inc.

• Denis Newman, Empirical Education Inc.

• Li Lin, Empirical Education Inc

• Andrew P. Jaciw, Empirical Education Inc.