Evidence as a tool for adaptation, adjustment, and scale through systems

Group Submission Type: Formal Panel Session

Proposal

A greater body of evidence exists about what works in education when implemented by NGOs than when implemented at scale, by governments, in low-income countries. Government programs are plentiful but may be less likely to be rigorously evaluated. Additionally, NGO programs that have been rigorously evaluated and shown positive results are not always able to transition ownership to government systems in a sustainable way. Programs run by NGOs may not be designed to consider the cost and delivery realities of government implementation at scale, and may not invest resources at the design stage of testing the drivers of success as well as how adaptations will work within government systems. This panel will explore the question: How can organizations use evidence at various stages of program design and adaptation to determine the necessary conditions and components for scaling within government systems? We will hear from two evidence-driven organizations on their iterative processes of trial and error, adjustment, and adaptation through randomized evaluations as well as other forms of evidence.

Scale can be defined in a variety of ways. One approach is to first test an innovation at a smaller scale, rigorously evaluate it, and scale the pilot up using the same delivery pathway through which it was shown to be successful. However, there may be limits to how far a small-scale program can be scaled, and various ways to think about how an effective model can be adapted to be integrated into a government system. It is necessary to consider both the system’s capacity to absorb the model, adaptations that will improve its functionality, as well as how impacts might change at scale. Using this guiding framework, Educate! will present on their scale-up experience supporting the Rwanda Education Board in 2015 to integrate active learning pedagogies and student business clubs into the Entrepreneurship subject for all upper secondary students, as well as their recent work to support exam reform as a system-level structure to incentivize long-term adoption.

Another approach is to apply evidence-based designs to entirely new contexts. Programs that are effective at improving outcomes in one context may help in other places where the key problems and underlying reasons for the problems are similar. But since no two contexts are identical, scaling programs in a different place is not as simple as transferring an “out of the box” solution. To scale appropriately and effectively, policymakers and researchers must account for local conditions and implementation capacity, and then adapt programs to that new context accordingly. In this panel, Youth Impact will share their experience rapid testing, refining, and scaling the ConnectEd program across six countries (Botswana, Kenya, India, Nepal, the Philippines, and Uganda) to reach over 35,0000 students.

J-PAL will present overarching frameworks and guiding questions for testing, learning, and adapting programming for scale through governments. First, J-PAL will share a framework for assessing the generalizability of interventions (Bates & Glennerster, 2017), providing guidance on how to think through (1) the disaggregated theory behind a program, (2) the local conditions that must hold for that theory to apply, (3) the strength of the evidence for the required general behavior change, and (4) the evidence that the implementation process can be carried out well. J-PAL will also frame questions and considerations around assessing scalability of interventions, designing projects with scale in mind, and managing the scaling up process in partnership with governments (MSI, 2016).

Panelists will engage audience members in a discussion centered on leveraging evidence to bring interventions to scale in partnership with governments. Presenters will discuss key questions such as: how can organizations test throughout the scaling process to ensure uptake in government systems? What preconditions are necessary for sustainable handover of responsibility to governments? What are different ways that organizations think about generalizability within country contexts and beyond into different countries and regions?

Overall, the panel will convene innovative, evidence-driven organizations in discussion with participants to think through key concepts in the science of scaling through education systems.

Sub Unit

• General Pool

Organizer

• Madeline Brancel, Abdul Latif Jameel Poverty Action Lab J-PAL at MIT

Chair

• Madeline Brancel, Abdul Latif Jameel Poverty Action Lab J-PAL at MIT

Individual Presentations

• Sustainability at Scale: How Educate! is Leveraging Evidence to Deepen Youth Impact through National Education Systems - Sarah Ojuando Ndinya, Educate!

• Evidence and reflections on a five-country randomized trial of phone-based targeted tutoring - Amanda Beatty, Youth Impact; Noam Angrist, Youth Impact; What Works Hub for Global Education, University of Oxford; Colin Crossley

• Practical frameworks for using evidence to scale education programs - Madeline Brancel, Abdul Latif Jameel Poverty Action Lab J-PAL at MIT