BackHome
Refresh: OffView Personal Schedule
Session Submission Summary

Beyond RCTs: Developing and implementing right fit rigorous evaluation approaches toward improving children's foundational learning at scale

Sat, March 28, 1:15 to 2:30pm, Hilton, Floor: Fourth Floor - Tower 3, Union Square 21

Group Submission Type: Formal Panel Session

Proposal

While randomized controlled trials are often considered the ‘gold standard’ in rigorous evaluation, they are also expensive, slow, and somewhat inflexible – all of which can be particularly challenging in resource-constrained Global South education systems. In contrast, right-fit rigorous evaluation approaches – including A/B testing, innovative quasi-experimental methods and adaptive evaluation frameworks – offer promising approaches to ensure that the implementation and evaluation of foundational learning programmes at scale is not only rigorous, but also efficient and conducive to real-time adaptation in complex and ever-changing implementation contexts.

This panel brings together implementers, practitioners and researchers to showcase innovative approaches to understanding – and iteratively improving – the implementation of education programmes operating within government systems at scale. Together, they illustrate both the range of possible approaches and their contributions to evaluation, organisational learning, and programme improvement that can rigorously yet efficiently answer the necessary intellectual and practical questions to adaptively improve the implementation of large-scale programmes that cultivate children’s foundational learning.

Paper 1 addresses a critical challenge in scaling educational interventions: how to rigorously evaluate programs operating at scale where experimental design is not possible. Using a unique dataset from India, the paper demonstrates that quasi-experimental methods such as difference-in-differences estimation can produce reliable causal estimates comparable to randomized trials while being substantially more cost-effective and operationally feasible. This methodology is then applied and validated to conduct one of the largest education evaluations to date across over 100,000 government schools in Uttar Pradesh.

Paper 2 examines the World Bank Implementation Science in Education’s approach to using Implementation Science to improve the implementation and scale-up of a Differentiated Learning programme in Ghana. The paper discusses the overall approach taken, including insights and right-fit implementation enhancements that range from an implementation assessment to surface political, managerial and technical barriers to the programme’s success, as well as the collection of student learning outcomes data. Results from the approach are being used to adjust the programme where needed, reflecting an iterative approach to progressing towards learning at scale.

Paper 3 shifts the focus to using A/B testing as a right-fit approach to improving program design, implementation, and ultimately supporting learning at scale. It analyses Youth Impact’s experience implementing over 60 A/B tests across the organisation's education and health programmes, from its origins in Botswana and beyond, demonstrating how organizations can embed rigorous experimentation into routine program delivery. Such A/B tests embody a right-fit approach by shifting from the traditional question in randomized controlled trials of "does the program work?" to the more operationally relevant "how does the program work most effectively, cheaply, and scalably?" Through concrete examples like optimizing mobile phone tutoring schedules and caregiver engagement strategies, this paper will demonstrate how rapid, regular, and rigorous testing can drive continuous improvement and maximize impact per dollar spent, making it an essential tool for achieving learning at scale.

Paper 4 presents a mixed-methods case study that examines how Teaching at the Right Level (TaRL) has been implemented and scaled across four Nigerian states. TaRL Nigeria has partnered with the What Works Hub for Global Education to form a unique, right-sized approach to analysing and interpreting routinely collected government administrative and process monitoring data, enabling a closer look at various implementation nuances across the four states. Paired with in-depth qualitative interviews, this research project includes comparative analysis of different scaling pathways across states and enables a deeper understanding of how state-specific political, socio-economic and security contexts shape implementation processes and learning outcomes.

To round out the panel, two discussants from the Gates Foundation and Pratham will contribute both funder and implementer perspectives on how the right-fit rigorous approaches presented can be leveraged to achieve learning at scale. They will synthesize the findings, draw connections between the papers, and highlight the significance of the approaches discussed in working towards achieving learning at scale.

These presentations collectively aim to advance the use of right-fit rigorous evaluation approaches at scale. The panel aligns with the field of comparative and international education in its underlying premise that there are many ‘ways of knowing’ in education – even in the domain of programme implementation, evaluation, and improvement for foundational literacy and numeracy, which is sometimes (erroneously) regarded as a narrowly technocratic and/or universalizing endeavour. Rather, the papers in this panel shift the focus to fit-for-purpose evaluation approaches that are rooted in contextual needs and resources, driven by the recognition that ‘education for all’ looks different everywhere.

Sub Unit

Organizer

Chair

Individual Presentations

Discussants