Measuring impact in the multiple dimensions of the reading ability

• In Event: Highlighted Session: Beyond Effect Size: Leveraging RCTs for Policy and Programming

Proposal

To improve reading skills in underserved students of the Western Highlands of Guatemala, the USAID-funded project Lifelong Learning was implemented from 2014 to 2020. The project supported the MoE to implement a Literacy Model for Bilingual and Intercultural Education Contexts (EBI, Spanish acronym) from pre-primary to third grade. The Literacy Model implementation considered top-down and bottom-up approaches and a systemic perspective. It involved actors at the national, regional, departmental, municipal, and school levels.

To evaluate project success in improving reading outcomes the project conducted a longitudinal study from 2014 to 2019. This study had three research objectives: a) to evaluate the efficiency of the project by comparing reading outcomes in implementation regions with those in comparison regions in Spanish and the Mayan languages, b) to learn about the trajectory for learning to read in a bilingual context, and c) to determine those skills within the continuum to learn to read in which the project had the most impact.

To evaluate efficiency of the project, we compared results of reading performance and growth by intervention and comparison regions. To measure performance, we estimated student's ability at the end of each school year, per grade. We used student ability to classify them in performance levels as established by the MoE. Likewise, to measure growth of reading ability, this was compared with that of the previous year. Results showed significant differences between intervention and comparison groups in both performance and growth estimates. The effect size for reding was bigger for reading in the mother tongue.

To learn about students’ trajectory to become independent readers, we conducted an exploratory analysis of the different skills that students had to master in each phase towards learning to read. In this analysis we decomposed the reading competency into the different skills that needed to be learned sequentially until a student becomes an independent reader. Performance of each skill was analyzed separately using descriptive statistics. Such study allowed the project to understand those skills that students were stagnating, and which needed a timely intervention.

Finally, we carried out a regression discontinuity to determine the threshold students in second grade need to reach in words per minute to be able to comprehend grade level text by the end of third grade. Findings of the regression discontinuity allowed the project to better intervene in improving comprehension, both oral and reading, in the early grades of the bilingual setting.

We will present the findings found in each objective of the longitudinal study. The discussion of the different perspectives with which the data of this study were analyzed will allow the reflection of the relationship between the designs of an intervention and the indicators that it intends to modify over time. In this study, the disaggregation of reading ability allowed to better visualize what stage of the trajectory to learn to read a project can reach with an intervention.

Authors

• Fernando Rubio, Juarez & Associates, inc

• Leslie Vanessa Rosales de Veliz, Juarez and Associates