BackHome
Refresh: OffView Personal Schedule
Session Submission Summary

Designing, implementing, and learning from mixed-methods randomized controlled trials in humanitarian contexts: lessons from remedial education in Dadaab and Kakuma.

Wed, April 17, 10:00 to 11:30am, Hyatt Regency, Floor: Pacific Concourse (Level -1), Pacific I

Group Submission Type: Formal Panel Session

Proposal

This panel will include presentations about the impact of a remedial education program implemented by World University Service of Canada (WUSC) in Dadaab and Kakuma, the program’s journey to scale, and WUSC’s experiences with conducting impact evaluations in protracted humanitarian crisis settings. The first presentation will focus on a randomized controlled trial that assesses the impact of WUSC’s remedial education program on educational and psychosocial outcomes based on four rounds of quantitative data collection. The second presentation will emphasize the barriers and facilitators toward scaling WUSC’s remedial education program. The third presentation will examine methodological lessons on how to conduct rigorous yet flexible impact evaluations of education programs in complex emergencies.

WUSC’s EERCK remedial program aims to improve learning outcomes, increase school retention, and improve transition to secondary education for 1,280 refugee girls. The remedial program provides weekend and holiday remedial education to 1,280 seventh- and eighth-grade girls in Dadaab and Kakuma refugee camps in Kenya. The program targets medium- to high-achieving students (relative to other students in the camps) who might be at risk of dropping out of school because of no achievement-related reasons.

American Institutes for Research (AIR) and WUSC designed a two-part study to evaluate the impact of the EERCK program on cognitive and noncognitive skills for seventh- and eighth-grade girls in the 2017 and the 2018 cohorts. In Kakuma, AIR is using a randomization-by-oversubscription methodology in which we randomly assigned half of 1,300 program-eligible girls to receive the program and half to a control group that will not receive the program. In Dadaab, AIR is using a regression discontinuity design to determine the impact of the program. We compare the outcomes from 496 girls who were above the cutoff for admittance into the program to 179 comparison girls who were just below the cutoff for admittance.

Midterm impact evaluation results show high levels of remedial education attendance among program participants. The impact evaluation findings reveal a small substitution away from primary education to remedial education in Kakuma and evidence for small increases in attendance in primary education in Dadaab. Midterm impact evaluation results do not show statistically significant impacts of WUSC’s remedial education program on learning outcomes, but the point estimates are all positive and in line with meta-analyses synthesizing previous evaluations of remedial education programs in low- and middle-income countries. Exploratory evidence suggests that WUSC’s remedial education program could have positive and statistically significant effects on learning outcomes for girls in food-secure households.

The process evaluation of WUSC’s EERCK program focuses on three interrelated key areas: the program’s journey to scale, its model for teaching and learning, and its model for mobilization and community engagement. Program participants have positive perceptions about the EERCK program and its potential to improve girls’ education; the EERCK program is so popular that community demand for the program is larger than WUSC’s capacity to deliver the program. Key implementation challenges include a shortage of remedial teachers (particularly female teachers), insufficient training for remedial teachers, and the combination of the remedial teacher and community mobilizer roles (e.g., the same people are serving as both teachers and mobilizers.)

WUSC’s experiences with the impact evaluation show that balancing cultural context, Do No Harm principles, ethical concerns and gender issues is critical for designing a high-quality impact evaluation in a protracted crisis setting. Since the EERCK program focuses exclusively on girls in a refugee setting with limited access to quality education, WUSC has developed strong relationships with stakeholders to navigate cultural and gender sensitivities. In addition, WUSC has actively communicated the rationale for the evaluation to key stakeholders, including beneficiaries. WUSC has successfully partnered with AIR in training enumerators, collecting data with transient populations, and implementing research protocols. The lessons from this evaluation are likely relevant to other humanitarian settings.

The mixed-methods evaluation of a remedial education program also provides valuable insights into the challenges of delivering quality education and implementing rigorous evaluation methods in refugee settings. Preliminary results hint at mechanisms through which the program can be improved and scaled. This project also sheds light on the mechanics of collaboration between researchers and practitioners in settings where both program implementation and data collection are difficult.

In addition, the research contributes to addressing evidence-gaps on what works to improve educational outcomes in complex emergencies. Major evidence gaps exist on how to scale education innovations effectively in protracted humanitarian crisis settings. In particular, evidence from rigorous impact evaluations of education innovations in these contexts is very limited. (Burde, Guven, Kelcey, Lahmann, & Al-Abbadi, 2015; Puri, Aladysheva, Iversen, Ghorpade, & Brück, 2015). Furthermore, a previous literature review showed that there is only limited strong evidence on how to scale education innovations in protracted humanitarian crisis settings (de Hoop et al., 2018).

The evaluation is funded through the Humanitarian Education Accelerator, which was created by Department for International Development (DFID), the United Nations Children’s Fund (UNICEF), and the UNHCR. The HEA serves to generate rigorous evidence on how to facilitate the effective scaling of innovations in education.

Sub Unit

Individual Presentations