The Repentant Randomista: Looking beyond generalized impact to inform adaptation of Early Grade Reading programming

• In Event: Highlighted Session: Beyond Effect Size: Leveraging RCTs for Policy and Programming

Proposal

In 2019, Nobel Prize in Economics was awarded to Abhijit Banerjee, Esther Duflo, and Michael Kremer for incorporating randomized controlled trials (RCTs) to answer questions in social-sciences; particularly to address issues of poverty in developing countries. Their work has been highly influential, with RCTs becoming a standard practice for impact evaluations of programs funded by donors such as the DfID and USAID. However, the empirical work by these Nobel prize winners has not gone without criticism; Stevano (2019) commented that the “. . . pitfalls of measuring the average treatment effect” such as not studying case studies or clusters to provide a thorough understanding of context, can create misleading or incomplete conclusions.
Since their inception, Early Grade Reading (EGR) Programs have used RCTs as the tool of choice to report generalized program impact on gains in learning outcomes. The author will present evidence that while the generalized impact is important, it can often miss an essential understanding of the variability of student learning outcomes that would otherwise be a critical consideration for programming.
Because EGR programs are implemented through local ministry systems, the author will present evidence that schools' impact on learning outcomes varies greatly depending on the clustered implementation. Only a low percentage of schools on most EGR programs account for generalized impact. The author will then present an argument that variation of learning outcomes within schools also becomes another critical layer of variability to explore for programming adaption.
Finally, the author will present evidence that some of our assumptions about education data should ‘say’ should not always be taken for granted across countries and projects. This evidence includes looking at learning gains by L1/L2 learners and socio-economic background.
In summary, the author will position that while generalized impact evaluation has an important place in understanding if an EGR program is effective or not, it is critical to describe further variability in learning outcomes to more comprehensively inform EGR programming.

Authors

• Simon King, RTI International

• Maitri Punjabi, RTI International