The Impact of Summer Learning Loss on Measures of School Performance

• In Event: 73.047 - The Not-So-Lazy Days of Summer: Interventions to Improve Achievement and Attainment

Abstract

A primary goal of the federal ESEA waiver program is improving the measures of school effectiveness, including the use of student-level growth models. Past school accountability systems, including the No Child Left Behind act, have been criticized for focusing on achievement level rather than growth (Ho, 2008; Kelly & Monczunski, 2007; Neal & Schanzenbach, 2010), targeting students in a narrow band of the achievement distribution (Ladd & Lauen, 2010; Neal & Schanzenbach, 2010; Reback, 2008), and identifying too many schools as failing (Linn, 2003). In recent years, researchers have developed “value-added” models to measure school effectiveness, which use district administrative data in an attempt to isolate the school’s impact on test score performance from the many other factors that influence student test scores but are outside of the school’s control. Value-added measures purport to eliminate many of the concerns associated with older measures, and thus many have argued that the ESEA reauthorization should require some form of value-added measure.
Much has been written about statistical challenges to isolating a school’s causal impact on student test scores, however one obvious problem with typical value-added models has received less attention—the impact of student “summer learning loss.” Virtually all value-added models estimate the effects of schools using spring-to-spring outcome data, simply given that statewide annual testing occurs on this timeline. As a result, the school is attributed both with learning gains made by students during the school year as well as the prior summer. In essence, schools are in part evaluated based on what happens to students prior to their first day in their class each year, which runs counter to the fundamental goals of an accountability system.
If all students learn at the same rate during the summer or the variation in summer learning rates could be accounted for using observable student characteristics, then spring-to-spring testing would not systematically bias estimated school effects. However, research on summer impacts to student learning have shown that middle-class children exhibit gains in reading achievement over summer, while disadvantaged children showed losses (Alexander, Entwisle, & Olson, 2001). To date, little is known about precisely what causes students of to exhibit such different learning rates during summers (Cooper, Nye, Charlton, Lindsay, & Greathouse, 1996; Downey, Von Hippel, & Broh, 2004; Heyns, 1978; Gershenson, 2013).
Using a unique dataset that contains both fall and spring standardized test scores for all students in Southern we estimate value-added scores based on traditional spring-to-spring data, as well as competing models that predict fall-to-spring test score gains. We examine whether schools are ranked differently using these two testing timelines and whether certain kinds of schools are especially affected by the test timing. The impact of summer learning loss is not unique to value-added models. The impact can also affect the more traditional means of measuring school performance—proficiency counts, growth-to-proficiency models, and student growth percentiles—used in States’ ESEA waiver applications. We also explore how summer learning loss impacts these more traditional measures of school performance.

Authors

• Andrew McEachin, RAND Corporation

• Allison C. Atteberry, University of Colorado - Boulder