Lessons From Innovators: Exploring Variation in the Implementation and Effectiveness of Reading Recovery

• In Event: Linking Implementation to Impacts: Year 2 Results From the Mixed-Methods Evaluation of the Reading Recovery Investing in Innovation (i3) Scale-Up

Abstract

The average effect of a program or policy is a concise way to understand its impact and evaluate performance. In looking beyond the average effect, we might also investigate how a program’s effect varies across individuals and sites to explore under what circumstances the treatment has an effect and whether the impacts are generalizable across populations.
The previous symposium papers focus on understanding and measuring implementation of Reading Recovery, and they demonstrate substantial variation along important dimensions. Prior research has shown heterogeneity in the effects of Reading Recovery which is thought to be attributed to variation in program implementation. This paper combines measures of program implementation with data from the multi-site randomized experiment in order to investigate school level predictors of variation in program impacts when implemented at scale. In this way, this paper draw explicit connections between the two broad questions of the evaluation of i3 scale-up of Reading Recovery which are: (1) How much does Reading Recovery impact the reading achievement of low-performing 1st grade readers?, and (2) Is Reading Recovery being implemented across school sites according to the specifications delineated in the Standards and Guidelines of Reading Recovery in the United States?
Data from years 1 and 2 of the randomized evaluation provide evidence of large significant average treatment effect as well as considerable heterogeneity of school-specific program effects. Pooling across multiple years of this multi-site study allows for HLM analyses of school-level factors associated with variability in program effects. The use of random treatment effects for schools in a multilevel modeling framework supports estimation of cross-level interactions to explain variability in treatment effects. These predictors fall under three headings: School-level enactment of Reading Recovery (e.g. Level and Years of Implementation), Implementation fidelity, and Critical Implementation Factors (e.g. Teacher Strength, Principal Investment, and Site Coordinator Engagement). Indicators of implementation fidelity and critical implementation factors are informed by the Reading Recovery Standards and Guidelines, which document requirements and recommendations for successful implementation, and the findings of qualitative inquiry.
Exploring the innovation of RR professionals is essential to understanding implementation of the program as well as its potential for impacts at scale. Results will inform future implementations of Reading Recovery, including future waves under the i3 Scale-Up, in order to maximize effects on student outcomes.

Authors

• Philip M. Sirinides, University of Pennsylvania

• Michael Armijo, University of Pennsylvania

• Jessica Gillespie, University of Pennsylvania

• Henry May, University of Delaware