Using long-term follow-up to learn more about projects’ impacts: Evidence from Honduras, Haiti, Burkina Faso, and globally

Group Submission Type: Formal Panel Session

Proposal

Evaluations are typically restricted to the life cycle of a program or activity, due to constraints such as funding or timing. Yet, theories of change almost always hypothesize long-term impact that goes unmeasured. In this panel, we discuss the potential added value of long-term follow-up evaluations, challenges that arise with long-term follow-ups, and specific considerations related to data collection for long-term follow-ups. This panel will also include presentation of the results of several recent long-term follow-up evaluations of early elementary education interventions in Honduras, Haiti, and Burkina Faso. Our panel’s chair is a senior education researcher specialized in education in lower- and middle-income countries. Our discussant is a senior education advisor to a major funding organization, who will comment on the panelists’ presentations and offer thoughts on the funding organization’s support for long-term follow-up evaluations.

Relevance to CIES24. This panel on long-term follow-ups is relevant to CIES24 because it draws attention to the impacts educational interventions can have on the life course development of the interventions’ participants–evidence that already exists for longitudinal studies in higher-income country contexts, but is scant in lower-income country contexts. Providing rigorous evidence of the long-term effects of early elementary interventions, if positive, may provide justification for social movements demanding public investment in their future.

The need this panel addresses. Investing in elementary education is worthy in its own right because youth deserve to have high quality educational experiences. At the same time, policy makers are interested in using their scarce resources to invest in the interventions that are most likely to generate future benefits as well in terms of improved skills and employment outcomes for participants once they reach adulthood. Nonetheless, most evaluations are short term and unable to provide information on long-term impacts. There is a need for more evidence on long-term effects on early elementary interventions in low- and middle-income contexts.

Advice offered. In addition to providing examples of recent long-term follow-ups, this presentation will discuss opportunities and challenges associated with long-term follow-up evaluations, as well as some strategies to navigate these challenges.

What we would have done differently. Each presenter will offer their lessons learned based on their experience with long-term follow-up studies. These may include lessons related to: gathering varied types of contact information to allow for long-term tracking of participants, making changes to original interventions to promote larger long-term impacts, among others. Because one of the presentations will be based on a study that is still underway, we do not yet have the full set of lessons identified.

Impacts of the project on the problem targeted. The original Honduras study (described in the second presentation) found significant impacts on students’ reading test scores after a year and a half of the intervention–long term results are not yet available but will be in time for CIES 2024. The Burkina Faso study (second presentation) found significant positive impacts on primary school completion for young women and young men 10 years after the intervention, and the Haiti study (second presentation) found positive impacts on emergent reading skills and oral reading fluency after two years of exposure to the curriculum.

In our first session, we will present on several recent long-term follow-ups and offer considerations on the role of long-term follow-up studies. This presentation will include the results of a long-term follow-up to the Read to Learn program, which employed a scripted mother-tongue curriculum for 1st and 2nd grade students in two districts of Haiti. The follow-up evaluation re-tested students at 4th grade, two years after program completion. This presentation will also report on the BRIGHT program in Burkina Faso, which included the construction of new schools and provision of complementary services in 132 communities. The evaluation conducted a 10-year follow-up that assessed impacts on enrollment, educational attainment, child labor, and child marriage.

In our second session, we will present on a recently completed long-term follow-up to a randomized evaluation in Honduras. The EducAcción-PRI project provided materials and ongoing pedagogical support for school principals and teachers in Honduras to use formative and summative assessments. For both types of assessments, the original evaluation found significant impacts on students’ reading test scores. In the long-term follow-up, the study team conducted interviews with national and local government officials to understand in what ways, if any, they used the results of the original study, and how their policies on assessment have evolved since the original study. The study team also interviewed principals and teachers from the original study schools to learn in what ways the schools continued to use the strategies they adopted during EducAcción-PRI.

In our third session, we will go in-depth on strategies for collecting data for a long-term follow-up study. The lead data collector for the Honduras study that we discuss in the second presentation will present on the strategies used for tracking respondents in the follow-up study. These include working with local education authorities to solicit their support for the study and associated data collection, sharing clear and accessible information about the study with potential respondents, using phone interviews, and being flexible in timing and location for in-person data collection activities, among other strategies.

Sub Unit

• Monitoring and Evaluation SIG

Organizer

• Sarah Liuzzi, Mathematica

Chair

• Emilie Bagby, Mathematica

Individual Presentations

• Long-term follow-up studies: potential, challenges, and evidence from Haiti and Burkina Faso - Danice Guzman, University of Notre Dame-Pulte Institute

• Long-term follow-up on an intervention to use student assessment to improve learning in Honduras - Sarah Liuzzi, Mathematica

Discussant

• Melissa M Chiappetta, USAID