Evidence and reflections on a five-country randomized trial of phone-based targeted tutoring

• In Event: Evidence as a tool for adaptation, adjustment, and scale through systems

Proposal

Youth Impact is a youth-led, Botswana-based organization focused on adapting rigorous, evidence-based health and education programs to local contexts, testing program effectiveness, and scaling proven programs to benefit youth across the world. We employ a school-based delivery model in partnership with governments and have reached over 100,000 youth. We specialize in rapid trials, including A/B tests, running over 25 rapid trials in just 36 months – some of the fastest, most rigorous evidence generated in the social sector.
This paper will review detailed monitoring data across a five-country randomized trial of phone-based targeted tutoring. The approach, called ConnectEd, was shown to work in Botswana and further replicated in Kenya, Nepal, India, Uganda and the Philippines. On average, the replication effort found a 65 percent increase in the percent of students who learn division across all trials, effectively mastering all basic numeracy concepts that we deliver in the course of programming. Results were strongest in the final two replication trials in Uganda and the Philippines - two countries experiencing some of the longest school closures in the world.
Although replications often find diminishing effects when moving from proof-of-concept studies to later replications, data from this replication study finds the opposite. Analysis shows that implementation fidelity, or the degree to which program implementers were able to accurately deliver instructional content, improved across replications and over time. This demonstrates a potential value in replication efforts that can improve delivery consistently throughout program development stages. Equipped with mechanisms to learn from experience, organizational "learning curves" can enable effective replication and scale-up.
The paper will conclude with practical suggestions on how to work across contexts and with partners to generate key program insights that improve delivery. The paper will also discuss continued methods for rapid evidence generation and program improvement beyond the context of full-scale RCT replications.

Authors

• Amanda Beatty, Youth Impact

• Noam Angrist, Youth Impact; What Works Hub for Global Education, University of Oxford

• Colin Crossley