Equity Implications from Program Impact Evaluations

• In Event: Education Equity Research Initiative: Problematizing the use of program learning data to investigate equity

Proposal

In this analysis, we utilize baseline data from an FHI 360 implemented program for early grade reading to highlight equity dimensions found within existing data and to inform our programming on any pre-existing disparities between different student subpopulations (gender, home language, SES, and geographic). This analysis serves several purposes that advance and highlight the importance of taking into account not only baseline disparities between different student groups but also differences in post-implementation program impacts. This approach will enable researchers, implementers, and key stakeholders and policy makers to design more focused programming and policy to both increase base skill acquisition but also to close gaps between student groups. Additionally, this approach is useful in that it allows the researcher to reverse engineer the characteristics most associated with lagging academic achievement and identify key areas to strengthen and focus on. As such, this analysis aims to address the following research questions:

- Are the lowest performing students systematically different from higher performing students?
- How large is the variability in student performance between schools, and what are the key determinants of these gaps?
- Are resources allocated equitably between schools, or are high performing schools receiving more or less government support?
- How are program impacts distributed? Are all student groups affected similarly or are gaps exacerbated between advantaged and disadvantaged students?
The early grade reading program’s primary purpose is to support USAID’s Goal One and improve reading skills for 100 million children in primary grades. At this stage of the analysis we rely on the first year of implementation to set up a quasi-experimental research design to evaluate the initial impact of the program on program participants relative to non-participants in control schools. Throughout the analysis, we address several equity elements from the data. First, we display the overall distribution of the outcomes of interest, in this case EGRA subtask scores. From the overall distributions, we are able to group students by ability levels and construct statistical ‘demographic profiles’ based on their background demographic and socioeconomic characteristics (gender, SES, home language, geographic location, disability status). As a result, we are able to identify the student groups that are most likely to struggle with reading and provide a data-driven basis for investigating specific groups when estimating program impacts.

Finally, we incorporate the equity element into this analysis by stratifying the impact estimates by gender, home language, region, SES, and disability status. We are then able to test for heterogeneity in the program effect and examine whether existing gaps are narrowed or exacerbated by the program.

Authors

• Wael Moussa, FHI 360

• Carina Omoeva, FHI 360

• Charles Gale, FHI 360