Outcomes of a longitudinal study of an early grade reading program in Nepal

• In Event: Contextual inquiry & reading programs: using evaluation tools to understand impact

Proposal

Project description:
Through the Equity in Education in Disadvantaged Districts program (2016-2018) supported by UNICEF, World Education worked to improve early grade learning (reading and mathematics) in 400 schools in four western rural districts of Nepal. The project’s core activities included: (1) developing a wide range of supplementary reading materials, such as posters, letter and number cards, decodable readers, leveled readers, games, and read-aloud stories; (2) supporting teacher professional development through training, coaching and learning circles; (3) strengthening support systems including school leaders, school management committees, and local and national governments; and (4) engaging parents and communities to support learning both at home and at school.

Study goals and design:
To measure progress in learning and assess the project’s impact, World Education conducted annual Early Grade Reading Assessments (EGRA) with a sample of roughly 2,000 students per year from 32 project schools as well as 16 comparison schools which received EGR materials but no training or other support from the project. The objectives this three-year study were to: (1) determine the current state of reading ability among early grade students in the project area; (2) measure program impact; and (3) identify specific areas of strength and weakness to inform programming. The study involved three waves of data collection: baseline (2016), midline (2017), and endline (2018). Researchers tested all children present from relevant grades on the day of the visit to the schools each year. In addition, the study tracked 363 individual children (253 in project schools and 110 in comparison schools) from baseline to endline (as well as larger sub-samples tested in two consecutive years). Enumerators used the Nepali EGRA test, plus a demographic questionnaire. The only change to the EGRA subskills tasks was to allow children to read the entire fluency/comprehension paragraph with no time limit; the enumerators noted the number of words ready at the end of 60 seconds, but children were allowed to keep reading the passage until they either finished it or stopped reading.

Findings:
Overall, 38.1% of students in project schools are meeting grade-level targets for both reading fluency and comprehension. Grade 3 targets align with those set by the Government of Nepal (GoN)—45 correct words per minute (CWPM) and 80% comprehension— while World Education and UNICEF set targets for grades 1 and 2 to build toward the GoN’s grade 3 targets.

The project also exceeded its target of a 5 percentage point improvement in reading achievement at each grade level in both years of the project. At the grade 3 level, 43% of students in project schools met targets in 2018, up from 15% at baseline in 2016. Proportions of students meeting targets also doubled for grade 1 and grade 2 students.

The project had statistically significant and “educationally significant” (cf Lindsey et al, 2008) impacts compared to control schools for nearly every EGRA task at every grade level. Most importantly, effect sizes were large and statistically significant at each grade level for reading comprehension and oral reading fluency, the two tasks that most accurately represent overall reading ability. For grade 3 students, the project boosted reading fluency scores by 20 correct words per minute and comprehension scores by 1.6 correct answers compared to students in control schools.

In total, 63.6% of grade 3 completers demonstrated an ability to read with “full” comprehension (five or six correct answers on comprehension questions) after two years of project intervention. Of special interest was the finding that 20% of grade 3 students achieved full comprehension even though their fluency was less than the target 45 correct words per minute. This is an indication that full fluency (at least at the GoN’s 45 CWPM standard) was not necessary for children to comprehend the passage. The implications of this for EGRA testing in other countries are important for early grade reading researchers and practitioners.

The program’s interventions attacked learning challenges at both ends of the spectrum—both raising the reading levels of children already somewhat proficient, and building the foundational reading skills of students who, at the start of the project, could not read a single word. Longitudinal data show that of non-readers at baseline, those in project schools were more likely to show emergent literacy skills at later assessments than those in control schools. This finding provides an important complement to findings on the number of students meeting targets.

Another interesting finding of this study is that providing teachers and schools with packages of supplementary reading materials is not, on its own, sufficient to improve children’s reading scores. Students in the comparison group achieved significantly less reading scores, even though the school had received the full materials package. Thus, it appears that the teacher training and coaching, school and community support, and encouragement for parental involvement are important contributors to children’s reading development, above and beyond high-quality materials.

The presentation will provide additional context on these results by also briefly sharing results from similar World Education interventions implemented in other areas of Nepal which covered an additional 1,000 schools, as well as USAID-supported EGRA results from Nepal. We will also discuss coordination with local and national government stakeholders and achievements in sustainability as these actors have adopted certain elements of the program. Finally, the presentation will also share lessons learned and areas that future EGRA studies in Nepal might seek to examine in greater depth, such as mother tongue issues and the impact of school feeding programs, as well as challenges created by changing contextual factors in Nepal (such as rural-urban migration and decentralization) that future programs must address to continue delivering improvements in reading outcomes.

The full research report is already written and ready for dissemination before and at the conference.

Authors

• James Macneil, World Education, Inc.

• Cristine Smith, University of Massachusetts Amherst

• Catharine Morgan, World Education, Inc.