Results From the Reading Recovery Investing in Innovation Fund (i3) Scale-Up Regression Discontinuity Design

• In Event: Final-Year Results From the External Evaluation of the Reading Recovery Investing in Innovation Fund (i3) Scale-Up

Abstract

Although the multi-site randomized controlled trial (RCT) conducted under the external evaluation of the i3 Scale-Up of Reading Recovery produced rigorous evidence of large impacts, the RCT provides no information about whether these impacts are sustained beyond first grade. This is because most students assigned to the control condition in the RCT ended up participating in the Reading Recovery intervention during the second half of first grade, after the posttest outcomes data were collected for the RCT.

To estimate long-term effects, we implemented a regression discontinuity design (RDD) in a separate, randomly selected sample of schools during each year of the i3 Scale-Up external evaluation. This RDD used cutoff-based assignment based on pre-intervention test scores—students with the lowest scores on the Observation Survey of Early Literacy (OS), relative to other students in their school, were assigned to Reading Recovery, while those who scored above the cutscore did not receive Reading Recovery.

The OS was used to assign students to Reading Recovery under the RDD design, and also to provide short-term outcomes data from mid-year and year-end administrations in first grade. Long-term outcomes were measured by collecting scale scores on third-grade state achievement tests in reading (in Georgia, Iowa, Illinois, Kentucky, Ohio, and Texas) or English language arts (in Massachusetts and South Carolina). Because students who participated in this study during first grade did not reach third grade until two years later, long term outcome data were available for only the 2011-12 cohort of participants. The sample for these analyses includes 6,371 students from 331 schools.

Using multilevel statistical models, the performance of students above and below the cutoff score was compared, with students nested within each participating school. Models included the centered pretest assignment variable as a covariate at the student level, a parameter for the discontinuity associated with assignment to Reading Recovery, a random effect for overall school performance (i.e., a random school intercept), a random effect for the pretest slope (i.e., a random school slope), and a random effect for the impact of Reading Recovery (i.e., a random treatment effect across schools). Model fit and potential misspecification was assessed (1) graphically via scatterplots and spline curves, (2) by testing for an interaction between pretest scores and the treatment assignment variable, and (3) by imposing various restrictions on the bandwidth around the cutscore.

Results from analyses of short-term impacts were similar to results from the RCT—the standardized effect size for mid-year OS scores was .76 (p < .0001), while the standardized effect size for year-end OS scores was smaller .37 (p < .0001). Impact estimates for long-term effects on 3rd grade scores were not significant, likely due to a dramatically reduced sample size (N=665 students) given limited availability of 3rd grade test scores (82% missing data). Efforts are currently underway to collect additional 3rd grade test scores for other students in the 2011-12 cohort as well as several thousand students from the 2012-13, 2013-14, and 2014-15 cohorts.

Authors

• Henry May, University of Delaware

• Philip M. Sirinides, University of Pennsylvania

• Abigail Gray, University of Pennsylvania

• Horatio W Blackman, University of Pennsylvania

• Yanmiao Xie, Unniversity of Delaware

• Pragya Shrestha, University of Delaware