Optimizing educational technology impact: A strategic approach to A/B testing

• In Event: Educational technology and implementation science: Using A/B testing to iteratively improve and scale educational interventions

Proposal

In an era where educational technology is rapidly evolving, the ability to quickly and cost-effectively iterate and improve interventions is crucial. This paper explores how IPA's Right-Fit Evidence Unit helps educational technology providers leverage A/B testing to enhance their impact, offering a glimpse into a methodology that has the power to transform the landscape of educational interventions.

We delve into why A/B testing can be a game-changer for educational technology, enabling rapid, causal identification of effective product variations. By comparing this approach to traditional M&E methods, we highlight its potential beyond boosting user engagement to also enhancing learning outcomes and improving program efficiency.

Our paper introduces a structured, stage-based learning framework as the foundation for effective A/B testing. This framework helps organizations to identify and prioritize key learning questions. We then explore how we assess these questions alongside an organization's capabilities and resources to determine whether and when A/B testing is a suitable tool. Through this lens, we examine scenarios where A/B testing shines and others where alternative approaches may be more suitable.

Building on this foundation, we discuss how our stage-based learning approach elevates A/B testing from isolated experiments to a comprehensive optimization framework. This method not only identifies what to test but also sequences these tests to systematically answer crucial operational and impact questions. By doing so, we help organizations assess their current status, define clear objectives, and implement a series of interconnected tests that build upon each other.

Drawing from our advisory services experience under the Partnerships for Technology in Education (P4T-Ed), this paper offers insights into real-world applications of A/B testing in educational technology. We will share lessons from how our partner organizations have worked towards introducing A/B testing in their learning toolkit and how they navigate the associated challenges. These examples illustrate the key steps in the journey of adopting A/B testing strategies, from defining product objectives to strengthening data capabilities and prioritizing test variations.

This paper offers a deeper understanding of how to strategically integrate A/B testing into educational technology projects, including how to identify when this method is most valuable, how to structure a comprehensive learning strategy, and insights from real-world implementation. Alongside other in this panel, this paper promises to equip educational technology professionals with the knowledge to harness the power of A/B testing, driving continuous improvement to increase the impact of their educational interventions.

Authors

• Andres Parrado, Right-Fit Evidence Advisory Unit @Innovations for Poverty Action

• Thomas Chupein, Innovations For Poverty Action

• Lucy Rimmington, Innovations for Poverty Action

• Eduardo Vargas Sanchez, Innovations For Poverty Action