Reading Recovery's i3 Scale-Up: Long-Term Impacts Through Third and Fourth Grade

• In Event: Reading Recovery: Long-Term Effects and Cost-Effectiveness Under the i3 Scale-Up

Abstract

To estimate the long-term effects of Reading Recovery, we implemented a regression discontinuity design (RDD) in a randomly selected sample of Reading Recovery i3 schools during each year of the i3 Scale-Up external evaluation (2011-2015) and also in one additional cohort during the 2016-17 school year. The RDD study used cutoff-based assignment based on pre-intervention test scores—students with the lowest scores on the Observation Survey of Early Literacy (OS), relative to other students in their school, were assigned to Reading Recovery, while those who scored above the cutscore did not receive Reading Recovery. One key benefit of this design is that this design aligns with the selection process advocated in the Reading Recovery Standards and Guidelines (2012), which states that students selected for Reading Recovery should be those with the lowest OS scores, and the number of students served each year is determined by the number of Reading Recovery teachers at the school (i.e., 4 students per teacher, per cohort, with 2 cohorts each year). The result is that no student who would have otherwise received Reading Recovery services was denied those services under this research design, yet we can still obtain rigorous evidence of causal effects by tracking the outcomes for Reading Recovery and RD comparison students into later grades.

Long-term outcomes were measured by collecting scale scores on state achievement tests in reading or English language arts administered to students in third and fourth grades. We were able to record and link 3rd grade state test score data in reading or ELA for 8,730 students, and 4th grade state test data for 5,452 students. Scores on each state test were standardized to a common z-score scale by subtracting the statewide mean and dividing by the statewide standard deviation separately for each state (or by the national mean and standard deviation for nationally-normed tests). In addition to collecting state test scores, we also administered an online survey through which teachers, teacher leaders, or school administrators were able to document the experience of individual Reading Recovery and RDD control group students in terms of (a) continued performance monitoring, (b) participation in supplemental reading programs and interventions, and (c) instructional programs and curricula used in first, second, and third grade. These data serve as potential moderators in analyses of variation in long-term impacts. Furthermore, these data help to (a) define the long-term treatment experience (i.e., what did RR students experience as follow-up?), (b) define the counterfactual experience (i.e., what did comparison students experience during first, second, and third grades?), and (c) provide information for estimating the costs associated with specific services and interventions provided to RR and comparison group students (see paper 3). Results from the long-term impact analyses are statistically significant and noteworthy; however, given the high-profile nature of this study, details of results are not included in this proposal in order to maintain confidentiality of results until the final report completes a peer review process over the next few months.

Authors

• Henry May, University of Delaware

• Aly Blakeney, University of Delaware

• Pragya Shrestha, University of Delaware

• Mia Mazal, University of Delaware

• Tara Tracy, University of Delaware