Presentation #2: Randomized controlled trial impact evaluation in conflict-effected region: The case of ALSE in Afghanistan (2014 – 2018)

• In Event: Letting communities take charge: a randomized controlled trial on sustaining schools in remote villages in Afghanistan

Proposal

This presentation will illustrate the randomized controlled trial design, treatment variations, sampling and fieldwork of ALSE research. ALSE consisted of two Phases: In Phase I, the treatment condition was based on the community-based education program implemented by CARE Afghanistan and CRS Afghanistan, versus a control condition of no CBE classes during 2014 – 2016 evaluation period. The Phase II, which this panel focuses on primarily, assessed the efficacy of a CBE sustainability model that involves village level community institutions in continuing CBE classes after NGOs end programming, from 2016 – 2018.

Phase II employed a “non-inferiority” equivalence trial approach (Lakens et al., 2018). The ALSE team gave a special consideration to the sensitivities in working with a context affected by conflict, low state capacity, and generally low incomes. As a non-inferiority equivalence trial and in keeping with goals of international organizations that hope to leave sustainable programs behind once their service is complete, we evaluated whether this sustainability model is at best equivalent to NGO service provision or at worst only modestly inferior. To do so, we implemented an experimental design that randomly assigned half of communities to have the sustainability model and the other half to have NGOs continue administering CBE classes longer than they would have otherwise. In no cases did NGOs withdraw with no additional programming to follow. Thus, in Phase II, the treatment condition was the “sustainability model” that involved handing over the CBE classes to community institutions at the start of Phase II and the control condition was a continued NGO management through the duration of Phase II.

Moreover, ALSE tested two enhancement variations regarding i) teacher recruitment enhancement, and ii) community engagement enhancement. Teacher recruitment enhancement had two conditions: hiring teachers who meet the Ministry of Education’s (MOE) requirements of qualification (12 years of education) which entails hiring from within and outside the village; and hiring teachers from inside the village without imposing the MOE qualification requirement. The conditions of the community engagement enhancement included implementation of an adult reading program, distributing Qur’anic messages about the importance of education for girls, and the default NGO implementation of CBE. The community engagement enhancement aimed to mobilize parents and help them support their children’s education. The goal of the teacher recruitment enhancement was to examine if there was a trade-off between hiring teachers from within the villages or outside the villages on education access and learning achievement.

ALSE’s sample includes 197 villages in six provinces of Afghanistan: Herat, Ghor, Bamiyan, Daykundi, Parwan, and Kapisa. Sample households are predominantly low income and the poverty rate in ALSE sample is higher than the national rate. Social demographics differ among the six provinces, including Hazara, Pashtun, and Tajik. The ALSE team administered four research instruments: Household Survey, Children’s Learning Assessment, Community Leader Survey, and Teacher Survey. In total, 4,620 household surveys, 5,544 learning assessments, 176 community leader surveys, and 125 teacher surveys were completed. Response rates, calculated using the American Association of Public Opinion Researchers’ Response Rate’s 3 criteria, showed 95.1% for household surveys, 78.4% for children’s learning assessments, 98.3% for community leaders, and 98.6% for teachers’ surveys.

Author

• Cyrus Samii, New York University