Optimizing an educational technology intervention with A/B testing

• In Event: Educational technology and implementation science: Using A/B testing to iteratively improve and scale educational interventions

Proposal

This paper explores how the nonprofit Rocket Learning has leveraged A/B testing to refine its in-person and digital interventions with care workers and parents in India, highlighting key insights from ongoing experiments aimed at enhancing engagement and learning outcomes among young children. The discussion will include challenges faced, strategies adopted, and the broader implications for policy and structural reform in the ECD sector.

Rocket Learning is a nonprofit organization dedicated to enhancing the quality of Early Childhood Care and Education (ECCE) in India, focusing on children under the age of 6. By engaging caregivers and frontline workers through digital interventions, particularly via WhatsApp, Rocket Learning builds communities of practice that support continuous learning in language, mathematics, cognitive, and socio-emotional skills. Operating across 10 states, Rocket Learning impacts 2.5 million children and 200,000 Anganwadi workers. Through innovative A/B testing, Rocket Learning continuously refines its interventions, driving measurable improvements in engagement and learning outcomes. With a vision to reach 50 million children by 2030, Rocket Learning is expanding its digital platforms and geographical reach.

This paper will explore the systematic approach Rocket Learning employs to refine its interventions through A/B testing. Over the past year, a series of structured trials (many in partnership with external researchers) have been conducted to identify strategies that effectively increase user engagement and improve learning outcomes. These trials have ranged from transitioning generic to personalized messaging, to experimenting with real-time feedback and various behavior change techniques.

The paper will cover the rationale behind these trials, the challenges encountered, and the iterative processes that have informed ongoing interventions. The paper will also consider the broader implications of these findings for A/B testing and rapid trials as an MME tool, and the place for such efforts within the broader MME toolkit (such as RCTs which Rocket Learning has also participated in). This paper provides a comprehensive view of how Rocket Learning optimizes its interventions, sharing valuable insights, lessons learned, and strategies for scaling successful practices in pursuit of their vision.

Author

• Azeez Gupta, Rocket Learning