Formative Evaluation on the Process of Scaling Up Reading Apprenticeship

• In Event: Evaluation of Reading Apprenticeship Improving Secondary Education

Abstract

We will present the logic model, evaluation design, and initial findings from the formative evaluation of the process of scaling up Reading Apprenticeship (RA). This five year study spans four states; the overall goal is to understand how school systems build capacity to implement and disseminate RA, and sustain these efforts.

A review of the literature shows that a unified theory of scaling-up education reforms is still nascent, with few empirical studies. In developing the logic model guiding this evaluation, the authors built upon literature emphasizing the “shift in reform ownership” from the program developers to the local level (Coburn, 2003). The research questions address the spread of the initiative, the scale-up process through the stages of the logic model, and contextual factors that affect scale-up. Researchers use a mixed methods approach. In the first and second year of the study, researchers observed and documented key project activities; tracked the numbers of schools, teachers, and students served; surveyed participating teachers and administrators; and conducted case studies with four RAISE schools.

By the second year of the initiative, the developers trained approximately 1,000 teachers from 170 schools in the scale-up sites. Results from the teacher and administrator surveys suggest considerably high levels of buy-in and commitment from this initial cohort of RAISE participants. Across the four states,
· 95% of the administrators responded that they were fully or fairly committed to making RA work at their schools.
· 82% of the teachers responded that they were fully or fairly committed to making RA work in their classrooms.
· 97% of the administrators said that they believe student learning at their school will improve if more teachers join RAISE.
· 92% of teachers said that RA is an appropriate framework for literacy instruction in their classroom.

Additionally, teachers reported high ratings of effectiveness of the RAISE professional development, with 90% of these teachers strongly agreeing or agreeing that the initial summer training led to changes in their teaching practices. The results also suggest high levels of implementation, with 67% of the teachers reporting using RA pedagogical practices at least a few times a week, and 27% using them in each lesson in the first year. While half of the teachers reported that implementing RA was moderately challenging, 91% said that they plan to use the RA framework during the following school year.

Positive ratings by participants, however, may not be enough to assure continued usage of the program. Administrators identified competing initiatives (56%) and budget constraints (33%) as the primary challenges to sustaining RAISE long-term. Additionally, when asked if RAISE would continue in their schools without the i3 federal funding, 26% said “No”, and 36% said “I don’t know”. Over half of the teachers identified competing priorities as a primary challenge that they faced in implementing RA. These results indicate that developers need to continue to work toward the assurance that the commitment to RA is supported through district and state policy and fulfills mandates to which continued funding is attached.

Authors

• Jenna Lynn Zacamy, Empirical Education Inc.

• Denis Newman, Empirical Education Inc.