Year 2 Results From the Randomized Evaluation of the Reading Recovery Scale-Up

• In Event: Linking Implementation to Impacts: Year 2 Results From the Mixed-Methods Evaluation of the Reading Recovery Investing in Innovation (i3) Scale-Up

Abstract

Research has consistently shown that most of the initially low performing students who participate in RR achieve at levels similar to the average of their classmates in reading and writing by the end of the intervention (Ashdown & Simic, 2000; Center et al, 1995; D’Agostino & Murphy, 2004; Iversen & Tunmer, 1993; Neal & Kelly, 1999; Pinnell et al, 1994; Schwartz, 2005; Pinnell, 1989; Quay et al, 2001; Rodgers et al, 2004 and 2005). In addition, Reading Recovery produced the largest impacts on student reading skills of any intervention reviewed by the What Works Clearinghouse (WWC, 2013), making it one of the most promising reading interventions for scale up.
The evaluation design for the i3-funded scale-up of Reading Recovery includes a rigorous experimental research design (i.e., a multi-site randomized controlled trial) that supports strong causal inferences about program impacts, coupled with mixed methods descriptions of program implementation and contextual factors. Short-term impacts on students’ reading performance are estimated by comparing mid-year reading achievement of students randomly assigned to participate in reading recovery at the beginning of first grade to students randomly assigned to the control condition. Differences in performance of the treatment and control students is estimated using a three-level hierarchical linear model (HLM) (Raudenbush & Bryk, 2002), with students nested within blocks (i.e., matched pairs) and blocks nested within participating schools. This HLM includes pretest scores as a covariate, along with random effects for blocks, a random effect for overall school performance (i.e., random school intercepts), and a random effect for the impact of Reading Recovery (i.e., random treatment effects across schools). The primary impact analyses utilizes the reading words and reading comprehension subscales from the Iowa Tests of Basic Skills (ITBS) as the posttest outcome measure and scores from the Observation Survey of Early Literacy Achievement (OS) as the pretest covariate.
Results from the second year of the randomized experiment are very promising. Of the 349 schools selected for the study, 266 conducted random assignment, for a 76% compliance rate. Within these schools, complete data were collected for 1,040 students in 520 matched pairs. Baseline equivalence of treatment and control students was confirmed for gender, race, English Language Learner status, and prior text reading level. HLM analyses revealed overall effects of Reading Recovery were .98 standard deviations for the ITBS Reading Words subscale and .87 standard deviations for the ITBS Reading Comprehension subscale. Both effects were significant at the 99.9% level. Additional significant effects (at the 99% level) were found for the variance of treatment effects across schools, translating to 90% plausible value intervals that were plus/minus approximately one-half of a standard deviation. This suggests that while the vast majority of schools have positive effects, many schools have remarkably large effects (i.e., >1sd). The goal of the next three papers is to identify and test critical implementation and contextual factors that may explain why some schools have such large impacts and others do not.

Authors

• Henry May, University of Delaware

• Philip M. Sirinides, University of Pennsylvania

• Michael Armijo, University of Pennsylvania

• Jessica Gillespie, University of Pennsylvania

• Abigail Gray, University of Pennsylvania

• Heather D Goldsworthy, University of Pennsylvania

• Cecile Huynh Sam, The University of Pennsylvania

• Toscha J Blalock, University of Pennsylvania