Beyond outcome evaluations – learning from program reviews in the era of implementation at scale

• In Event: Primary education, digital tools in schools and the impact on literacy and numeracy

Proposal

Since 2000, an NGO has been successfully implementing a comprehensive literacy and library intervention at government-run primary schools in more than 10 countries. The intervention aims to develop literacy skills in the early grades and promote strong reading habits among primary school children. The design of the intervention differs based on the local context; but it includes a set of common elements across locations, such as: 1) the development of early grade language curriculum following scientific and proven pedagogical approaches; 2) teacher professional development and the provision of teaching and learning materials (TLMs) in line with the pedagogy; 2) on-site coaching and mentoring for teachers; 3) school or classroom libraries providing children with regular access to a variety of high-quality, contextually relevant books; and 4) regular activity and output monitoring alongside periodic classroom-based formative assessments for children to support data-driven implementation. The NGO has conducted several outcome evaluations to assess the effectiveness of its intervention across locations. For example, in 2019 alone, nine such evaluations were completed across several countries. These evaluations mainly focused on assessing children’s early grade literacy skills and measuring the changes in them during the implementation period. Results from these evaluations produced robust evidence on the effectiveness of the intervention.

This intervention was implemented for nearly two decades, with the NGO primarily responsible for the design and delivery of the common elements. Studies have shown that many of the intervention’s common elements are also included in other effective, sustainable, and scalable literacy interventions (Stern et al., 2023). However, most of these elements (e.g., frequency, duration, and quality of teacher training and in-person coaching and mentoring support) require significant financial and human capital resources. Experts have opined that effectively scaling such interventions by NGOs operating on limited resources or by resource-strapped governments could be difficult (Jordan, n.d.).

At the same time, learning outcomes, especially literacy skills, in many low- and middle-income countries remain low, despite recent progress in other areas (Stern et al., 2022). The COVID-19 pandemic not only caused widespread learning losses between 2020 and 2023, but it also exacerbated existing inequities within and across countries, reversing some of the recent progress in learning outcomes among primary grade children (Betthäuser et al., 2023). To address this problem, donors, governments, and the development sector at large have been pushing for scalable interventions to reach these children. In parallel, NGOs are increasingly working to integrate their programmatic activities within existing government delivery systems, hoping to achieve greater scale, improve local ownership, and ensure sustainability.

In this context, the NGO adapted its implementation model over the last 3-5 years to rapidly scale through government systems in different geographies while trying to maintain its historical achievements in improving literacy skills outcomes. The adaptations to the intervention design vary by country but can include: 1) primary use of government curriculum, teacher’s guides, and other mandated TLMs, with the NGO’s materials becoming supplementary; 2) training modalities now include a combination of in-person and distance learning conducted via digital technologies; 3) government master trainers are now used to cascade training on a large scale down to the local level; 4) system-based actors, such as local government officials or headteachers, are now used to coach and mentor teachers; and 5) coordinated use of digital tools for collecting, analyzing, and disseminating implementation data by both NGO and government stakeholders. In summary, the NGO increasingly relies on systems-based actors at various levels of government for key elements of the intervention.

With these adaptations and the rapid expansion of the intervention, the NGO needed an alternative approach to program evaluation. Specifically, it required evaluation strategies that quickly assess how activities are implemented and how they relate to outputs and outcomes. Consequently, the NGO has integrated a series of Program Reviews into its monitoring and evaluation strategy across several geographies. These reviews are conducted in the initial year of implementing the adapted design and aim to gather critical primary data (both qualitative and quantitative) from various in-school and out-of-school stakeholders involved in implementation. They collect quantitative data from schools, early grade teachers, and early grade children, and qualitative data from local government officials and headteachers, among others. Additionally, they leverage robust implementation monitoring data to triangulate findings and identify bottlenecks.

To date, the NGO has completed program reviews in Laos (April 2024), Nepal (May 2024), and India (July 2024), and will complete one in Bangladesh by the end of 2024. The main objectives of these program reviews are to: 1) examine the perceptions of various system actors on program implementation, including the delivery, frequency, and quality of various activities; 2) assess teachers’ knowledge, perceptions, and practices in developing children’s literacy skills and reading habits in the early grades; 3) assess children’s literacy skills; 4) identify programmatic challenges and strengths that in-school and out-of-school actors face in implementing the intervention; and 5) identify the contextual and programmatic factors within each geography that influence children’s literacy skills and reading habits.

This presentation will synthesize the results of these program reviews, focusing on the lessons learned, strengths, and areas for improvement identified within and across geographies. It will discuss the assessment of new design elements, scale-up efforts through government systems, and the effects of these shifts on children’s learning outcomes. Additionally, it will present the NGO’s critical reflections on the alternative evaluation strategy, including its advantages and limitations. Lastly, it will provide insights for external audiences on policymaking, designing, implementing, and advocating for similar large-scale programs worldwide.

Authors

• Rebecca Alice Westbrook, Room to Read

• Pinaki Joddar, Room to Read

• Japheth Ogol, Room to Read