Results From the Reading Recovery Investing in Innovation Fund (i3) Scale-Up Multisite Randomized Experiment

• In Event: Final-Year Results From the External Evaluation of the Reading Recovery Investing in Innovation Fund (i3) Scale-Up

Abstract

Reading Recovery has been the focus of a number of impact studies over the course of its 25‐year history and has produced the largest impacts on student reading skills of any intervention reviewed by the What Works Clearinghouse (WWC, 2008), making it one of the most promising reading interventions for scale up. Strong effects were detected in Years One and Two of the i3-funded scale-up of Reading Recovery (May et al. 2014; May et al. 2015); this paper presents the final results of the four year study.

The evaluation design for the i3-funded scale-up of Reading Recovery includes a rigorous multi-year experimental research design (i.e., a multi-site randomized controlled trial) that supports strong causal inferences about program impacts, coupled with mixed methods descriptions of program implementation and contextual factors. Short-term impacts on students’ reading performance are estimated by comparing mid-year reading achievement of students randomly assigned to participate in reading recovery at the beginning of first grade to students randomly assigned to the control condition. Differences in performance of the treatment and control students is estimated using a three-level hierarchical linear model (HLM) (Raudenbush & Bryk, 2002), with students nested within blocks (i.e., matched pairs) and blocks nested within participating schools. The primary impact analyses utilizes the reading words and reading comprehension subscales from the Iowa Tests of Basic Skills (ITBS) and the Observation Survey of Early Literacy Achievement (OS) as posttest outcome measures. This HLM includes pretest scores and school year as covariates, an interaction between treatment and school year, as well as random effects for blocks, a random effect for overall school performance (i.e., random school intercepts), and a random effect for the impact of Reading Recovery (i.e., random treatment effects across schools.

Of the 1,441 eligible i3 schools selected for the study, 1,255 conducted random assignment, for a 87% compliance rate. Within these schools, complete ITBS data for both students in a matched pair were collected for 70% of matched pairs. Baseline equivalence of treatment and control students was confirmed for gender, English Language Learner status, and prior text reading level. HLM analyses revealed overall effects of Reading Recovery were .48 standard deviations for the ITBS Total score, .43 standard deviations for the ITBS Reading Words subscale, .43 standard deviations also for the ITBS Reading Comprehension subscale, and .88 standard deviations for the Observation Survey of Early Literacy Achievement. All effects were significant at the 99.9% level. Additional significant effects were found for the variance of treatment effects across schools. Results are discussed with respect to the data collected on the experience of students in the control group.

Authors

• Philip M. Sirinides, University of Pennsylvania

• Henry May, University of Delaware

• Abigail Gray, University of Pennsylvania

• Horatio W Blackman, University of Pennsylvania