Randomized Controlled Trials (RCTs) in education: revisiting the promises of the RCT method

Group Submission Type: Formal Panel Session

Proposal

Abstract for the Panel [max. 1,000 words]

Randomized Controlled Trials (RCTs) are revered by some as the “gold standard” for assessing the impact of interventions or reforms and condemned by others for reducing the complexity of cause-effect relations to a few controllable and measurable conditions. The method has experienced a popularity to the extent that some policy makers nowadays do not dare to allocate funds and make political decisions without the scientific stamp of approval provided by a randomized experiment. Spearheaded in its infancy by renowned development economists, such as Esther Duflo, Co-Founder and Director of the Abdul Latif Jameel Poverty Action Lab at the Massachusetts Institute of Technology, the method has for a while been considered a dictum for externally funded programs and reform projects of developing countries. RCTs are nowadays produced on grand scale, in particular, on topics that are controversial, such as studies that measure the impact of contract teachers or the impact of low-fee private schools on student achievement. The choice of method is not coincidental. RCTs have achieved such an authoritative status to afford policy makers with the necessary justification to make difficult decisions on controversial issues.

In fact, within a short period of time, many funding agencies and donors have come to consider RCTs as the only reliable method for assessing whether their financial investment was worth the actual outcomes. A decade after RCTs emerged as the preferred method of evaluation of influential international donors (notably, World Bank, DFID, USAID), the question is whether RCTs deliver what they claim to provide: a succinct understanding of what “works” and why it works? This panel revisits the promises of RCTs based on four RCT studies carried out in different country contexts.

Two of the studies—the ones by Prashant Loyalka and Moses Oketch—are published in the 2019 World Yearbook of Education, entitled “Comparative Methodology in the Era of Big Data and Global Networks”, edited by Radhika Gorur, Sam Sellar and Gita Steiner-Khamsi. In order to engage in a lively intellectual debate on the topic, a third presentation by Dana Burde and Elisabeth King has been secured. We invited two discussants to comment on the presentations: Pauline Rose, a scholar of international stature with extensive experience in RCTs in education and Radhika Gorur, internationally renowned for her work on the sociology of measurement.

The three presentations examine the opportunities and shortcomings of randomized controlled trials (RCTs), randomized experiments, or impact evaluations. They reflect on why policy makers are enamored with impact evaluations and how researchers design an RCTs to draw policy-relevant conclusions. Depending on their own experiences and viewpoints, the presenters either highlight or address some of the criticism of RCTs, such as: (i) randomized experiments are atheoretical, (ii) decontextualized, and (iii) only lend themselves for “stop” and “go” decisions rather than for a systematic improvement of an educational program or a reform.

Sub Unit

• General Pool

Chair

• Gita Steiner-Khamsi, Teachers College, Columbia University

Individual Presentations

• The strengths and shortcomings of RCTs: REAP's experiences in China - Prashant Loyalka, Stanford University

• Randomized Controlled Trials: Limitations for explaining and improving learning - Moses Oketch, University College London

• Don't Throw the Baby Out with the Bath Water: How to extract the best outcomes from your RCT - Dana Burde, New York University; Elisabeth King, New York University

Discussants

• Pauline Rose, University of Cambridge

• Radhika Gorur, Deakin University