Pratham's journey with A/B testing: from quick experiments to large-scale program improvement in a northern state in india

• In Event: A/B testing: Structured learning and iteration for program improvement in education

Proposal

Pratham Education Foundation will highlight its iterative approach to experimental design, demonstrating how A/B testing has been systematically integrated to create a continuous cycle of program improvement within a large-scale government partnership program for grades 4-5 in Uttar Pradesh, a northern state in India. The panel session will examine Pratham's novel approach to conducting A/B testing in a non-tech educational environment, focusing on how learnings from rapid experimentation enhanced foundational literacy & numeracy (FLN) program for grades 4-5 students, and how insights from each testing round strategically informed the design and refinement of subsequent A/B testing experiments.

The Annual Status of Education Report (ASER) 2024 reported that only 49 percent and 56 percent of children in grades 4 and 5 in Uttar Pradesh respectively could read grade 2 level text. With such a large percentage of children unable to read, Pratham - as part of its research under What Works Hub for Global Education (WWHGE) - focused on addressing this issue through A/B testing. From December 2024 to March 2025, we conducted our first A/B testing experiment in the state to understand which remedial method would 'work best' for improving FLN skills of grades 4-5 children in government schools. In this round, we tested four different approaches building on the state's existing curriculum.

During the panel, a key focus will be examining learnings from the first round of A/B testing, our approach to collecting 'high frequency data' for 'Pratham instructor-delivered tweaks' in a non-tech based intervention setting, and how data insights shaped our next round of experimentation. One critical learning which emerged: mentoring visits that guided instructors to group classrooms by 'learning levels' and teach accordingly proved promising.

Drawing from first-round learnings, we designed a second large-scale A/B testing experiment with government school teachers in April-May 2025. Our approach tested whether grouping and teaching as per learning levels could be effectively communicated through instructional materials (a low-cost model) compared to continuous mentoring visits by Pratham staff. Despite several implementation challenges, the results showed that the low-cost model worked better, slightly higher than its counterpart tweaks.

Real-time impact: Based on quick evidence generated of children's learning outcomes in grade 4-5 from A/B testing, Pratham successfully advocated for additional remedial classes in August-September 2025, reaching 12,000+ children across 430+ schools. In August-September 2025, Pratham continued building on previous experiments, evaluating improved variations of previously tested tweaks. In further explorations, the official mentoring cadre of the state (Academic Resource Persons - ARPs) will be involved to test the efficacy of mentoring variations through the cadre itself, in addition to testing the impact of sharing of digital resources to improve FLN skills.

Pratham’s case study offers learnings for the social sector on how structured A/B testing can transform both program effectiveness and organizational learning, turning research insights into actionable program improvements at scale.

Author

• Banishwar Singh, Pratham Education Foundation