Go big or go home: implementation driven A/B testing to drive learning at scale in Botswana

• In Event: Improving education via evidence generation to inform implementation at scale: lessons from South Africa, Nepal and Botswana

Proposal

Many students in low- and middle-income countries go to school but cannot read a sentence or perform basic numerical operations. One of the educational reforms that has successfully addressed this “learning crisis” is targeting instruction to a child’s learning level rather than age or grade. A recent review by Angrist and Meager (2022) finds that targeted instruction has the potential to deliver 0.50 standard deviation gains in learning when fully implemented. Given large effects in education are considered to be 0.10 standard deviations, research and policy focused on increasing take-up of productive education interventions – such as targeted instruction – represents a 5x higher return to learning over identifying new effective interventions. We demonstrate concrete approaches to increase implementation fidelity in a rapid A/B test in Botswana where Teaching at the Right Level is being scaled and Youth Impact, one of the largest NGOs in the country, conducts rapid A/B tests to optimize the program for scale and to maximize government adoption every school term. Many of the questions to A/B test have emerged directly from instructors/implementors hunches on what they are currently unsure about, but think can work, and want to learn about in a structured manner. This approach to question identification ensures that new results will be taken up in practice since they will be most likely to be useful and informative, and the end users will have been bought in from the beginning. We present results from a A/B test generated in this manner which tested highly targeted instruction within and across classrooms and along multiple proficiencies. Results show 0.22 standard deviation gains in learning. These effects are similar in size to the gap identified in a recent synthesis between average effects (0.23) and effects accounting for implementation (0.50). These results reinforce that implementation research is tractable and high-return, reveals methods to achieve the highest effects in the literature, and approaches to enhance interaction between practitioners and researchers.

Authors

• Tendekai Mukoyi, Youth Impact

• Thato Letsomo, Youth Impact

• Noam Angrist, Youth Impact