What contributions can randomized experiments (RCTs) make to our understanding of education systems reforms?

Group Submission Type: Formal Panel Session

Proposal

Systems research is an emerging field in education, and ambitious researchers are tackling questions of how education reforms can achieve dramatic student learning gains at scale. Randomized experiments are a prevalent tool in social science research in large part because of the precision with which they can estimate the effects of specific programs. However, developing useful policy recommendations for systems reform requires understanding not only the effect of a single intervention, but also system dynamics and relationships between relevant actors. Studies that rigorously examine implementation at scale, reveal features of markets, or properties of institutions and their structures, can make valuable contributions. At the same time, while randomization can be a powerful tool in some circumstances, there are many contexts in which systems reform can be more effectively studied using a different methodology. The three papers and discussant who will present on our panel will elucidate how, and under what conditions, randomized methodologies can be used to generate evidence for education systems reform.

Karthik Muralidharan will discuss his paper (with Abhijeet Singh) examining an experiment in India that randomized the large-scale implementation of a school governance intervention designed in accordance with global best practices. They found that some aspects of the reform were well executed, but others were not. This ultimately resulted in precisely zero effect on student learning outcomes. The paper reveals how low state capability constrains the ability of some types of reforms to improve service delivery in developing country contexts. The paper also shows how RCTs can be used to test major system reforms at scale.

Leonard Wantchekon’s paper with co-authors Yves Atchade and Pierre Nguimpeu proposes a methodology for using randomized experiments to isolate the intrinsic effect of governance institutions on the overall effects of government policies. Their paper develops a theoretical model showing how institutional effects can be isolated and estimated experimentally and implements this model on school governance data. Their work is at the frontier of using randomized experiments to explain systems, and in this case institutional properties, that can in turn inform reform decisions.

Tassew Woldehanna, and his co-authors John Hoddinott and Ricardo Sabates, will tackle the question of what conditions are needed for randomized methodologies. Their paper takes large scale education reform in Ethiopia as a case, and examines what methodologies are appropriate for rigorously exploring the dynamics of this particular reform in its particular context. They conclude that a pure RCT is not appropriate in this case and discuss the alternative they recommend.

Susannah Hares will serve as our panel discussant, contributing her perspective on the three papers based on her experience as a partner to governments both as a researcher and policy advisor. She will discuss, among other things, the ways that experimentation can be incorporated into the policy process and how this type of work can influence reform design and practice.

Taken together, these three papers and the contributions of our discussant will tackle the question of the role that randomized control trials can play in generating the systems evidence needed to inform reforms at scale.

Sub Unit

• General Pool

Chair

• Laura Savage, DFID

Individual Presentations

• Improving public sector management at scale? Experimental evidence on school governance in India - Karthik Muralidharan, University of California, San Diego; Abhijeet Singh, Stockholm School of Economics

• Evaluating the intrinsic effect of institutions: theory with applications to school of governance - Leonard Wantchekon, Princeton University; Yves Atchade, Boston University; Pierre Nguimpeu, Georgia State University

• Is a RCT appropriate for evaluating Ethiopia’s large-scale quality education reform program? - Tassew Woldehanna, Addis Ababa University; John Hoddinott, Cornell University; Ricardo Sabates, University of Cambridge

Discussant

• Susannah Hares, Center for Global Development