Using A/B testing to optimize cost-effectiveness

• In Event: Educational technology and implementation science: Using A/B testing to iteratively improve and scale educational interventions

Proposal

A/B testing for education interventions, especially in low- and middle-income countries, remains rare. Far too few social programs are evaluated rigorously or scaled successfully (List 2022; Mobarak 2022). A/B testing, which optimizes program cost-effectiveness and scalability and allows for iterative testing between versions A and B of a program, could help address this pressing need.

RCTs and A/B testing both fill an important gap in evaluating social impact and maximizing social returns on investment. RCTs randomly assign individuals or groups of individuals to a program (treatment) or no program (control). In A/B testing, there is also random assignment between groups, except rather than include a pure control group, we compare multiple versions of a program: version A vs. B. Similar to RCTs, random assignment ensures equal groups, so any program modification reflects the causal impact of the program optimization. 

A central tenet of A/B tests is that programmatic decision-making is primary, with results influencing program decisions in real-time. While RCTs typically aim to answer the question “does the program work” with an external evaluator and a long-run lens, A/B tests are typically focused on internal program decision-making, are frequent, and aim to answer the question “how does the program work most effectively, cheaply, and scalably.”

RCTs and A/B tests can be used in sequence to generate complementary evidence. A/B testing can be used before and/or after an RCT. A use case for before is when a program is in the pilot stages and an organization wants to test out what version of a program it would subject to an RCT. For example, an A/B test could be used to determine how to best promote take-up or enrollment in a tutoring program before subjecting the program to an RCT to determine its impact. A/B testing can also be used after an RCT. For example, once there is proof of concept that the program has impact, A/B testing is useful for testing whether the program will work in a new context, with different implementers (say government teachers vs. volunteers), can be more impactful, or can generate the same impact at lower cost.

In this paper, we outline the benefits of A/B testing. The paper draws on experience implementing A/B testing for the last five years in health and education programs. We discuss the differences between RCTs and A/B tests and point out their complementarities. We also discuss how program implementers can use A/B testing to deepen their impact, reduce costs, and scale programming.

Author

• Amanda Beatty, Youth Impact