Highlighted Session: Beyond Effect Size: Leveraging RCTs for Policy and Programming

Group Submission Type: Highlighted Paper Session

Proposal

In 2019, Nobel Prize in Economics was awarded to Abhijit Banerjee, Esther Duflo, and Michael Kremer for incorporating randomized controlled trials (RCTs) to answer questions in social-sciences; particularly to address issues of poverty in developing countries. Their work has been highly influential, with RCTs becoming a standard practice for impact evaluations of programs funded by donors such as the DfID and USAID.
In 2002, Robert Coe presented a paper entitled “It’s the Effect Size, Stupid” (Coe, 2002), which has been cited 1114 times according to Google Scholar as of October 13, 2020. The title intention is a take on James Carville’s Presidential election campaign slogan for Bill Clinton in 1992, “It’s the Economy, Stupid” (Jasinowski et al., p.500), basically intonating an idea so obvious it needs to take precedent over all others. Proposed initially by Jacob Cohen in 1988, the “Effect Size” is the “gold standard” parameter for evaluating the magnitude of impact evaluations of intervention, which use a control group as a comparison. An effect size is a standardized measure of the impact that allows the investigator to know the standardized magnitude of difference between a control and treatment group (Sullivan and Feinn, 2012). Early Grade Reading (EGR) program personnel wait with bated breath before the results of an RCT impact evaluation, hoping for an effect size of reasonable magnitude that shows their hard work has paid off.
Combining the RCT developed by Nobel Prize recipients and using the “gold standard” Effect Size creates considerable pedigree, and so when it comes to impact evaluations, it is hard to look beyond them. However, the empirical work by these Nobel prize winners has not gone without criticism; Stevano (2019) commented that the “. . . pitfalls of measuring the average treatment effect” such as not studying case studies or clusters to provide a thorough understanding of context, can create misleading or incomplete conclusions. This panel will present different perspectives on why looking beyond generalized impact and effect size is critical if programs are to receive worthwhile analysis and interpretation that can lead to meaningful programming adaption.
The first authors present evidence that exploring the variability of learning gains across and within implementing schools in practical ways can present more evidence about program impact and suggest that generalized impact can be misleading, especially if the generalized impact is unrepresentative of the average impact by school. The author will why learning outcomes alone cannot explain the “why” of impact, but more focused investigations of the data can suggest where programs need to look to learn more about barriers to implementation fidelity they need to mitigate to improve learning outcomes.
Our second author applies an equity lens to the problem by conducting a cross-country review of EGR program evaluations in the past decade in the Global South. The by analyzing learning outcomes using common dimensions of equity across the countries, complemented with available qualitative information, the panelist highlights the importance of moving from a “one size fits all” approach for evaluating EGR interventions. The panel finishes by giving practical recommendations on how to conduct equity analysis to improve evidence-based decision making.
The third author combines monitoring and evaluation data in a case-study from Sub-Saharan Africa. Making the case that Monitoring and Evaluation resources constitute up to 13% of program resources, implementers and evaluators alike are interested in getting more out of these investments and data. Using modeling techniques, the author shows how project implementation mediates impact and helps identify schools that need different support types.
Our final author leverages longitudinal data from a USAID-funded project reaching underserved students of the Western Highlands of Guatemala. Their study had three objectives: to evaluate program efficiency, to learn about the trajectory for learning to read in a bilingual context, and to determine those skills within the continuum to learn to read in which the project had the most impact. The data was leveraged using varied techniques, including decomposing reading competency into different skills and conducting regression discontinuity to determine the thresholds students in the second grade needed to reach in words per minute to comprehend grade-level text by the end of third grade.

Sub Unit

• Monitoring and Evaluation SIG

Individual Presentations

• The Repentant Randomista: Looking beyond generalized impact to inform adaptation of Early Grade Reading programming - Simon King, RTI International; Maitri Punjabi, RTI International

• Bringing monitoring and evaluation together: A case study from Sub-Saharan Africa - Fernanda Gandara, School-to-School International

• Measuring impact in the multiple dimensions of the reading ability - Fernando Rubio, Juarez & Associates, inc; Leslie Vanessa Rosales de Veliz, Juarez and Associates

Discussant

• Hetal Thukral, School-to-School International, Nathan Associates