Re-Examining the Head Start Impact Study: Impact Persistence and the Counterfactual Condition

• In Event: Heterogeneity in Educational Intervention Fadeout and Long-run Mechanisms: Three Analyses of Block-Level Impact Patterns

Abstract

Objectives and Theoretical Framework
A growing body of evidence suggests that the immediate effects of early interventions fade shortly after the program’s conclusion (Authors, 2017). In response, researchers have begun to investigate the mechanisms behind this fadeout and whether specific features of interventions or outcomes moderate their persistence (Authors, 2023). Building on this work, we explore center-level variation in the implementation of Head Start. While studies of the federally funded initiative have found mixed long-term results (e.g., Deming, 2009; Garces et al., 2002; Pages et al., 2020), growing evidence suggests that outcomes may depend on contextual differences across centers. Prior work (Bloom & Weiland, 2015; Kline & Walters, 2016) has documented substantial variation in Head Start impacts by center, demographic subgroup (e.g., gender), and participant background (e.g., children likely to stay home without an offer, or those with low pretest scores). We extend these explorations by assessing center-level variation in Head Start’s longer-term effects, and the center-level conditions that predict impact persistence or fadeout.
Methods
We use data from the Head Start Impact Study (HSIS), a large-scale, nationally representative, randomized evaluation of over 4,000 children across more than 300 Head Start centers (Puma et al., 2010). We capitalize on children’s random assignment to centers to explore variation in post-test and follow-up effects. Importantly, we leverage the nesting of children within centers to treat each Head Start center as a unique randomized controlled trial, allowing us to use meta-analytic models to understand how center-level factors relate to variation in follow-up impacts across centers.
We conduct a series of random-effects meta-regressions, weighting estimates by the inverse variance-covariance matrix using R’s metafor package. Our primary independent variables are the grantee-level post-test impacts on cognitive and social-emotional skills, while our dependent variables are the grantee-level impacts on the same skills, measured at 1- and 2-years follow-up. First, we examine whether post-test impacts predict longer-term effects on cognitive and social-emotional outcomes. Then, we explore whether differences in counterfactual conditions (i.e., the proportion of control-group children in home care, with limited English proficiency, low pretest scores, or who were Black), or post-intervention learning environments (measured for treatment, control, and all children) help explain differences in persistence.
Results
As depicted in Figures 1-4, post-test impacts on cognitive skills were generally more predictive of impacts at 1-year follow up (b = 0.44, SE = 0.11; p <0.001), compared to impacts on social-emotional skills (b = 0.25, SE = 0.12; p < 0.1). At 2-year-follow-up, post-test impacts were slightly less predictive of impacts on cognitive (b = 0.31, SE = 0.15; p < 0.05) and social-emotional skills (b = 0.19, SE = 0.11; p < 0.1), indicating fadeout. Beyond the influence of post-test impacts, the characteristics of the counterfactual condition did not appear to drive variation in persistence. These findings are preliminary.
Future analyses will further assess how grantee-level factors, and the post-intervention environment may differentially influence the persistence of Head Start program effects.

Authors

• Caroline Botvin, Columbia University

• Emma Hart, Boston College

• Drew Bailey, University of California - Irvine

• Tyler Watts, Teachers College, Columbia University