Requiring Evidence: A Model for Integrating Evaluation into Program Design

• In Event: Improving Effectiveness and Sustainability by Localizing Evidence: Models for Testing, Scaling, and Institutionalizing What Works

Proposal

Educate! prepares youth in Africa with the skills to succeed in today’s economy. Our core experience delivers the most essential skills young people need to transition to work, combining skills training, mentorship, and practical experience. Randomized evaluations have found that this core experience significantly improves skills, educational attainment, and gender equity outcomes for participants, particularly for young women. Using this evidence as proof-of-concept for our model, we have now shifted to scaling and adapting this experience to reach more youth through three pathways: directly in schools, through government integration, and to youth unable to access formal education.

As a learning organization, Educate! depends on data to improve model design. To do this, we value generating rigorous qualitative and quantitative evidence of our impact as we scale. However, we know that without a concerted effort to ensure that decisions are driven by evidence, teams can suffer from the three common pitfalls that often prevent research from informing design: confirmation bias, inertia, and evaluation decoupled from scale.

To overcome these challenges, Educate! created an internal process for program design that clearly outlines the criteria a model must meet in order to advance to the next stage of scale and to unlock the funding required to achieve that scale. One of the most important criteria is evidence — models must produce positive evaluation results before advancing to the next stage.

This process has four distinct stages: discovery, validation, efficacy, and scale. In each one we ask different questions, which influence methodology choice and conclusions based on how mature the model is. The first two stages focus primarily on creating a model that works. Through qualitative research and small-scale descriptive evaluations, we consider how to best reach and impact youth. Once teams have validated a model, models may advance to the final two stages. These stages focus on developing and refining the delivery systems needed to produce measurable impact at scale. The evaluations in these stages are larger and employ more rigorous designs and analyses, culminating in a controlled evaluation. By the time we submit a model to the most rigorous test, we have generated enough evidence to feel confident that the model, and the delivery systems, will create the desired impact at scale.

In this session, we will share Educate!’s framework for program design and how we determine what evidence is required. To illustrate how this framework is applied, the presentation will also share results from a series of evaluations that have contributed to the development and scale of bootcamps for youth outside of the school system in East Africa, providing a case study for how evidence has directly informed design.

Author

• Meghan C Mahoney, Educate!