Tracking the Impact of School AYP (Adequate Yearly Progress) Interventions Under No Child Left Behind

• In Event: Evaluation of Performance-Driven School Accountability Interventions in Korea Versus the United States

Abstract

Evidence for the effects of pre-NCLB high-stakes testing and test-driven accountability on student achievement has been mixed (Lee, 2008). Similarly, recent studies and syntheses yield mixed findings on post-NCLB academic progress, generating controversy over the policy’s usefulness (see Dee & Jacob, 2009; Lee, 2006; National Research Council, 2011)
While previous studies examined specific interventions at particular time points, it is necessary to examine the effects of time-varying NCLB interventions over time. This study examines interventions under NCLB targeted at schools in need of improvement that failed to meet the AYP target for at least two consecutive years and were subjected to sequential stages of NCLB-prescribed interventions, including: (1) school transfer, (2) supplementary education service (SES) (3) corrective action, (4) planning for restructuring, (5) restructuring .
Methods
This study employs a comparative interrupted time-series design to explore the impact of NCLB policy on student achievement outcomes based on school-level trend analyses of New York State public school fourth and eighth graders’ reading and math achievement results during pre-NCLB (1999-2005) and post-NCLB (2006-2009) periods. It compares post-NCLB trends in reading and math achievement with pre-NCLB trends among fourth and eighth graders from across the state. Hierarchical linear models are used to examine between-school variations in reading and math achievement trends over the 1999-2009 period. Recognizing that selection bias poses a threat to the validity of causal inferences about the effects of NCLB, we controlled for student demographics variables (e.g. percentage minority students) as time-varying covariates, and school characteristics (e.g. location,) as time-invariant covariates. Latent variable regression method was used to control for the effect of initial status and pre-NCLB growth rate on post-NCLB changes
The study also employs time-varying causal effect model estimation procedures to examine the effects of school-level AYP interventions under NCLB. It employs propensity score matching with inverse probability treatment weighting (IPTW) methods to account for selection bias . Data is from New York State public schools with 8 years of achievement outcome measures and time-varying AYP status and intervention history (2002-09).
Preliminary Findings and Implications
Findings from the first part of this study demonstrate that schools in need of improvement that went through NCLB interventions demonstrate faster growth during post-NCLB period despite having had relatively lower initial status and slower growth during pre-NCLB period., While the positive effects of interventions proved significant after controlling for time-varying and time-invariant covariates that influence selection into interventions, the positive effects disappear or become negative once controlling for initial status and pre-NCLB growth through latent variable regression. This suggests that the observed faster achievement growth of schools under intervention after NCLB may be due to statistical artifacts instead of a true effect; results from sensitivity analyses confirm this. The second part of this study, still underway, will identify which specific stages of interventions were effective or ineffective for schools in need of improvement. Findings from the study may have important implications for the debate on NCLB’s reauthorization.

Authors

• Jaekyung Lee, University at Buffalo - SUNY

• Hyejin Shin, University at Buffalo - SUNY

• Laura Casey Amo, University at Buffalo - SUNY