Testing what works best: A/B testing for rapid learning and improved decision-making at scale

• In Event: Beyond RCTs: Developing and implementing right fit rigorous evaluation approaches toward improving children's foundational learning at scale

Proposal

Making data-driven decisions has long been a core objective for social organizations aiming to maximize impact at minimal cost. Randomized Controlled Trials (RCTs) have traditionally been the gold standard for evaluating program effectiveness, providing rigorous evidence to guide decision-making. However, a key limitation of RCTs is the long timeline required to obtain results, which can misalign with the pace of operational decision-making and reduce the relevance of findings.

Inspired by the technology sector, some social organizations — such as Youth Impact — have adopted A/B testing as a rapid, rigorous, and scalable approach to evaluating social programs. A/B testing is frequently used to assess the cost-effectiveness and scalability of interventions by comparing a standard model (Option A) with an optimized alternative (Option B). A/B testing operates on three core principles (3Rs): Rigorous, Rapid, and Regular. Like RCTs, A/B testing is rigorous in design, using randomization to establish causal evidence. It is rapid in execution, often lasting only weeks or months, enabling real-time program improvement. A/B testing is also regular, relying on routine Monitoring and Evaluation systems within organizations rather than external data collection, making it cost-efficient and sustainable. Whereas RCTs typically ask “Does the program work?” by comparing a treatment group to a no-program control, A/B tests focus on “how does the program work most effectively, cheaply, and scalably?”

At Youth Impact, A/B testing has been central to program development for over eight years. The organization has conducted more than 60 A/B tests across its education and health programs. These tests have led to consistent improvements in program design, including up to 40% gains in efficiency and cost reductions of up to 34%. A key factor in this success has been prioritizing implementer experience to identify operational innovations that reduce costs and improve outcomes.

For instance, in Youth Impact’s mobile phone tutoring program—which delivers weekly numeracy tutorials to primary school children via calls to caregivers—scheduling inefficiencies emerged as a major cost component. In fact, substantial time and cost was wasted on scheduling tutoring sessions. The bi-weekly, longer session model improved scheduling efficiency, reducing costs by up to 20% without compromising learning outcomes.

Another set of A/B tests evaluated the impact of caregiver engagement. Encouraging caregiver involvement more than doubled the program’s impact at a low incremental cost. Cost-effectiveness analysis revealed that caregiver engagement generated learning gains of up to 65 standard deviations per $100 spent—among the highest returns reported in the education literature.

By embedding A/B testing into program delivery, Youth Impact has demonstrated how the social sector can use rapid experimentation to improve outcomes, scale more effectively, and make better use of limited resources. As a complement—not a replacement—to traditional RCTs, A/B testing offers a practical and scalable tool for evidence-based decision-making.

Authors

• Doux Baraka

• Noam Angrist, Youth Impact; What Works Hub for Global Education, University of Oxford

• Amanda Beatty, Pratham International