Scaling Failure: A Review of Success Rates and the Scale-Up Effect in Education Research

• In Event: Roundtable Session 61
  In Roundtable Session: Research on Evaluation Tools and Trends

Abstract

Understanding why interventions succeed or fail is critical to advancing education evaluation research. This study presents an empirical framework for studying the “failure” of education innovations, particularly when brought to scale in large samples. We developed an analytical approach for defining failure and two measures of effect size degradation from small to large-scale studies: the scale-up effect index and the superrealization bias index. Using data from the What Works Clearinghouse and Best Evidence Encyclopedia, we found empirical evidence of the scale-up effect and estimated its magnitude for math and reading achievement interventions. This work provides an initial step toward the development of a comprehensive approach to failure analysis which we argue would greatly strengthen education evaluation research.

Authors

• Katie Barghaus, University of Pennsylvania

• Jennifer McMaken, University of Pennsylvania