Criteria for reporting improvement of a reading intervention in bilingual settings

• In Event: Improving M&E for vocational ed, assessment systems and reading outcomes

Proposal

Juarez & Associates implemented the Lifelong Learning Project (LLP) in Guatemala from 2014 to 2021. The main objective of the project was to improve students' reading skills.
When the project finalized in 2021, LLP demonstrated that the students whose schools were located in the intervention municipalities had better reading comprehension skills in the mother tongue than those whose schools were in the comparison municipalities. Furthermore, the project demonstrated that children in target municipalities were no different than children in schools in comparison municipalities in their reading ability in the second language (Spanish). The project also demonstrated that as the intervention consolidated in the region, the difference between intervention and comparison students became larger.

LLP project was designed to support the MoE in implementing a Literacy Model for Bilingual and Intercultural Education Contexts (EBI, Spanish acronym) from pre-primary to third grade. This model responded to the National Bilingual Education Model (Mineduc, 2009) and underpinned the integration of EBI throughout the education system as a cross-cutting issue in the country's educational policy. The Literacy Model implementation considered top-down and bottom-up approaches and a systemic perspective. It involved actors at the national, regional, departmental, municipal, and school levels, such as MoE authorities and national directorates, WH EBI Advisory Committee, Mayan Language Academy of Guatemala, departmental education directorates, local universities, municipalities, local leaders, local educational coaches and technical supervisors, school principals, teachers, parents, and children. To evaluate project success in improving reading outcomes the project conducted a longitudinal study from 2014 to 2019.

As discussed earlier, the project was successful in meeting its objective of improving reading ability. In order to contribute to better planning future reading interventions with similar objectives, this proposal aims to discuss criteria of project success based on LLP recent experience. Some criteria that will be discussed in this presentation is the following: a) reading outcome selection, contrasting fluency and reading comprehension outcomes; 2) evaluation design selection, contrasting transactional and longitudinal designs; 3) development approach, contrasting a systemic and bottom-up approach and a top-down focalized approach, and finally, 4) lessons learned from the reading intervention design that allowed having larger and sustainable effects will be presented. Some of those lessons were to build capacity within local actors over implementing a prescribed intervention; to carefully select intervention regions to avoid contamination of similar interventions, among others project design decisions.

In discussing the success criteria mentioned above, the paper will discuss the experience of the LLP project in the selection of reading comprehension on fluency as a result of the project. You will also discuss the advantages and limitations of a longitudinal assessment over a transactional assessment design to demonstrate reading improvement. And finally, the advantages and disadvantages of systemic interventions and with a bottom-up focus on reporting the sustainability of the intervention will be discussed.

Author

• Leslie Rosales, Juarez