Long Term Impact Evaluation of Amazonia Lee Reading Program

• In Event: Education across the globe: Facing conflicts and challenges

Proposal

Abstract. Low- and middle-income countries (LMIC) often lack sufficient resources to support schools, train teachers, and conduct data collection. Over the last decade, stakeholders in LMICs, donors, and nongovernmental organizations increasingly have paid attention to reading achievement and instruction in primary grades, but there is little evidence on the impact of those investments, the sustainability of reading programs, or the way such programs contribute to local or national shifts in education policy. This presentation focuses on an impact evaluation of a program focused on early grade reading in Ucayali and San Martín, Peru and the sustainability of the results one year after the program ended.
Program. The Amazonia Lee (AL) program was funded by USAID with the goal of improving early-grade reading performance and addressing the learning gaps of children in primary school (grades 1 to 3) in two Amazonian regions in Peru: San Martín and Ucayali. The AL program was implemented by the Regional Education Directorates (Direcciones Regionales de Educación [DREs]) from 2015 to 2017 with technical assistance from Universidad Peruana Cayetano Heredia (UPCH). Most of the program activities were capacity building such as teacher training, teacher coaching, teacher study groups, and training for principals and regional staff.
Evaluation design and data. USAID commissioned an impact evaluation of the program. The evaluation was based on a randomized control trial where eligible schools in each region were randomly assigned to a group that was offered Amazonia Lee and another group that was not offered the program. In Ucayali, the comparison group received the business-as-usual services in 2015 and 2016 but in San Martin, about 75% of the schools in the comparison group received support through a national capacity building program called Soporte Pedagogico. The short-term results of the evaluation focused on second grade students who were in the study schools in 2016. After the program ended, USAID commissioned a long-term impact evaluation to assess whether the immediate impacts of Amazonia Lee were sustained over time. The long-term evaluation was based on national assessment data collected by the Ministry of Education in Peru for fourth graders in 2018.
Findings. We found similar short-term impacts when we use the reading assessment data collected by the Ministry of Education than what we found with the EGRA collected for the impact evaluation.
• In Ucayali, Amazonía Lee second-graders had on average higher ECE reading scores than students in the control group in 2016, the second year of Amazonía Lee implementation with Amazonía Lee schools scoring 26 points higher on the ECE test than the control group, an impact equivalent to 0.39 standard deviations (SD).
• In San Martin, Amazonía Lee had no impact on ECE scores in 2016 in San Martín compared to the control schools that implemented the Soporte Pedagógico program.
The short-term impacts were not sustained as students progressed to higher grades.
• In Ucayali, the 2018 ECE reading scores of fourth-grade students in Amazonía Lee schools were higher than scores of students in the control schools, but the differences were not statistically significant. The adjusted mean for Amazonía Lee fourth-graders on the ECE reading test was 430 points, 12 points higher than the adjusted mean score for control group students.
• In San Martín, students in control schools had higher adjusted means on ECE fourth-grade reading scores in 2018 than Amazonía Lee students, and the difference was statistically significant. Fourth-graders had on average an adjusted ECE reading score of 477 in control schools, 11 points higher than the adjusted mean scores for Amazonía Lee schools. A plausible explanation for the difference is that Soporte Pedagógico continued to be implemented though 2018 in control schools while Amazonía Lee was not implemented at full strength in treatment schools after external funding ended.
Conclusions. In-service teacher training on early-grade reading instruction, whether supported by local technical assistance providers or by the national government, contributes to better student reading skills in the primary grades in the short and the long term. The legacy of Amazonía Lee in both regions is rooted in the cooperative alliances established between UPCH and the DREs, and the regional adoption of the reading instruction approach used in the Amazonía Lee program.

Author

• Larissa Campuzano, Mathematica