Looking Across Evidence Sources for Patterns in the U.S. K-12 School System Improvement Literature

• In Event: Politics, History, and Systems Improvement in 21st Century School Reforms

Abstract

Objectives or Purposes: Wide variation in the effectiveness of U.S. K-12 school system improvement initiatives in the post-No Child Left Behind era motivates the need to examine heterogeneity of impacts across cases to learn what works, where, and why.

Perspective(s) or Theoretical Framework: Prior work suggests that the policy impacts of school reform on student outcomes are interrelated but distinct from the public perceptions or political dynamics of reforms. A significant literature suggests that positive public reception should increase the likelihood of reform effectiveness, and this may be influenced by historical factors, demographic contexts, and power dynamics. This presentation draws on a range of sources to examine variation across school improvement initiatives, including their effectiveness, public perceptions of reforms, and the relationship between perceptions and policy effectiveness.

Methods, Techniques, or Modes of Inquiry: This presentation draws on two national studies and a cross-district, within-state, analysis to draw lessons about school system improvement. The first is a meta-analysis of post-NCLB impact evaluations of efforts to improve schools or districts labelled low-performing. The second is a public opinion study using a nationally representative sample to assess views on school and district improvement. The third involves cross-case studies of state-led district improvement efforts in Massachusetts in the era from 2013 to 2019.

Data Sources, Evidence, Objects, or Materials: We use both quantitative and qualitative methods. The meta-analysis relies on 141 estimates from 67 studies, and draws on qualitative coding of the interventions and contexts under study in each program evaluation. The public opinion work draws on a nationally representative 2017 survey of 4,214 respondents and uses, in some cases, experimental methods to assess the causal effect of various framings of reforms on public support for these reforms. Finally, the Massachusetts work relies on a longitudinal analysis of student outcomes tied to four state-led district improvement initiatives on a range of student outcomes, using statewide, student-level administrative data. The work also draws on careful qualitative deep dive work on one particular case, using analysis of news coverage, survey data, and interviews with state and district leaders.

Results and/or Substantiated Conclusions or Warrants for Arguments/Point of View: Meta-analytic results show that the racial/ethnic composition of communities is predictive of the effectiveness of school improvement initiatives. National public opinion data shows majority support for interventions but greater levels of opposition among those most likely to lose power as a result of school improvement reforms. Cross-case studies of district improvement efforts in Massachusetts suggest that balancing local and state authority in the context of state-led school system improvement efforts helps to improve both the public perceptions of and the effectiveness of educational reforms.

Scientific or Scholarly Significance of the Study or Work: This work has implications for state and district leaders seeking to select schools or systems most likely to benefit from interventions, and provides policymakers with guidance about framing reforms and designing interventions to improve the public reception of policies and the likelihood they will positively impact school system performance.

Author

• Beth Schueler, University of Virginia