Using A/B testing to optimize your education programming: A “jump-start” workshop to help you get started

Group Submission Type: Workshop

Description of Session

A/B testing has become a common approach for program optimization in the technology sector (Kohavi et al., 2020; Siroker and Koomen 2015), yet A/B testing for education interventions, especially in low- and middle-income countries, remains rare. Far too few social programs are evaluated rigorously or scaled successfully (List 2022; Mobarak 2022). A/B testing, which optimizes program cost-effectiveness and scalability and allows for iterative testing between versions A and B of a program, could help address this pressing need.

RCTs and A/B testing both fill an important gap in evaluating social impact and maximizing social returns on investment. RCTs randomly assign individuals or groups of individuals to a program (treatment) or no program (control). In A/B testing, there is also random assignment between groups, except rather than include a pure control group, we compare multiple versions of a program: version A vs. B. Similar to RCTs, random assignment ensures equal groups, so any program modification reflects the causal impact of the program optimization.

A central tenet of A/B tests is that programmatic decision-making is primary, with results influencing program decisions in real-time. While RCTs typically aim to answer the question “does the program work” with an external evaluator and a long-run lens, A/B tests are typically focused on internal program decision-making, are frequent, and aim to answer the question “how does the program work most effectively, cheaply, and scalably.”

In this workshop we will explain A/B testing and provide examples of how we at Youth Impact have used it in our education programs. We suggest how other program implementers can use A/B testing to deepen their impact, reduce costs, and scale programming.

Learning objectives
The key learning objectives for this workshop are
Explain what A/B testing is and how it is different from RCTs
Demonstrate how A/B testing can be used to improve implementation, such as informing decisions on scaling and cost-effectiveness
Brainstorm A/B testing options with workshop participants
Provide workshop participants with ideas about how they can operationalize A/B testing in their organizations

Delivery plan
The workshop will consist of a presentation with examples from practitioners. Attendees will fill out a short survey about their objectives. Youth Impact and Building Tomorrow team members will support attendees in small groups in brainstorming A/B testing ideas. Attendees will leave the workshop with ideas about A/B testing plans for their organizations.

Sub Unit

• Book launches and Workshops / Workshop

Chair

• Amanda Beatty, Youth Impact

Workshop Organizers

• Noam Angrist, Youth Impact; What Works Hub for Global Education, University of Oxford

• Samuel Kembou, Jacobs Foundation

• Tendekai Nkwane, Youth Impact

Presenter

• Whitney Warren, Building Tomorrow