Challenges to Telling an Accurate Story of Change: Lessons Learned from Eight Years of Early Grade Reading Interventions in Rwanda

• In Event: The role of EGRA in reading education, policy and reform

Proposal

Understanding change in students’ learning over time, and in particular their literacy learning, is a critical tool curriculum, teacher professional development, and school reform policy makers. This is evident by the inclusion of indicator 4.1.1 in the Sustainable Development Goals which calls for countries to track the proportion of students at each stage of education who reach at least the minimum reading proficiency level.

Over the past decade students’ oral reading fluency (ORF) levels have become the most prevalent means of measuring children’s overall reading skills in low- and middle-income countries. It’s use has been based on the following arguments : 1) it can be measured quickly and with minimal cost (Wagner, 2011, 2012); 2) it produces a single quantitative, and by implication objective, measure of reading skills; 3) it is a skill that both the general public and education specialists understand and acknowledge as important and 4) it correlates with reading comprehension (LaBerge & Samuels, 1974; Beach & O’Connor, 2014; Pikulski & Chard, 2005; Armbruster et al., 2001; Bashir & Hook, 2009; Wise et al., 2010), a decidedly more complex skill to measure (Abadzi, 2011, Fuchs et al, 2001). ORF has become the most public measure of educational quality in low- and middle-resource countries, and an important lever to justify investments in early grade reading (EGR) (Bartlett et al, 2015).

This presentation challenges some of the above assumptions. It uses a longitudinal analysis of ORF results from five national assessment carried out in Rwanda between 2011 and 2018 to examine the extent to which they tell an accurate, compelling story of improvements over time and one that justifies the significant resources directed to EGR over that period. The latter is critical for justifying the governments’ continued focus on EGR, including its commitment to having teachers adopt instructional practices deemed necessary for improved reading outcomes. The issue of sustainability is the central theme of this year’s CIES conference.

The presentation focuses on why USAID Rwanda and the Rwanda Education Board are unable to tell a compelling story of improvements over time: the percentage of P1 students meeting minimal ORF benchmarks has fluctuated dramatically, from a high of 14% to a low of 3%. The findings support Wagner’s 2011 assertion that high-level skills are required to design valid and EGR tasks. They suggest that the perceived simplicity of the EGR tasks have led implementers and Ministry counterparts to underestimate their technical sensitivity and by extension their ability to produce reliable and comparable results over time.

The presentation identifies initiatives instituted to address the above challenges, including working with Ministry counterparts to establish rigorous, grade and task-specific empirical metrics for assessment tasks, implementing a strenuous evaluation process for determining whether tasks align with the metrics and establishing a singular, common purpose for collecting and reporting on ORF. The lessons learned have the potential to inform efforts in other jurisdictions to accurately measure systemic improvements over time.

Authors

• Norma Evans, Evans and Associates

• Sharon Haba, Chemonics International