From experimental pilot to national scale-up: endline evaluation results for the Tusome early grade reading program in Kenya

• In Event: Educational reforms for innovative approaches to strategy and learning

Proposal

The Government of Kenya’s passage of a reform package in 2003 guaranteeing free primary education precipitated drastic improvements in education access and equity throughout the country. While encouraging, enrollment increases were not accompanied by a corresponding increase in resources, straining the education system and leading to sharp declines in the quality of instruction and pupil reading performance. In response, USAID’s Kenya and East Africa mission partnered with the Kenyan Ministry of Education to implement an $88.8 million basic education initiative to improve the reading skills of approximately 7.8 million children in public and non-formal primary schools across Kenya. The project, Tusome (“Let’s Read” in Kiswahili), built upon earlier research-based reading initiatives to create a sustainable and affordable national reading program. Specifically, Tusome was designed to scale up the most impactful and cost effective components of the experimentally evaluated Primary Math and Reading (PRIMR) pilot (2011-2014). These components include a scope and sequence for language instruction in English and Kiswahili; development and distribution of matched teaching and learning materials; in-service teacher training and mentoring; and monitoring and supervision of program implementation using ICT. Tusome was also designed to assist the Government of Kenya at the technical and policy levels to sustainably improve reading skills beyond the USAID-funded activity. To this end, Tusome worked to build the capacity of the Kenyan government, specifically the Ministry of Education and relevant Semi-Autonomous Government Agencies, to carry forward program activities following the closeout of the USAID activity.

In 2019, USAID contracted NORC at the University of Chicago to conduct an independent endline evaluation in order to address descriptive and normative questions about Tusome following the activity’s national rollout. Given the diverse goals of the evaluation, several methodologies were used including: (1) pre-post outcomes assessment, which focused on outcomes at the school-level using quantitative data collected at baseline, midline, and endline; (2) summative evaluation using a cross-section of quantitative and qualitative data to retrospectively assess program effectiveness, stakeholder beliefs and practices, and lessons learned from a variety of perspectives; and (3) formative evaluation to prospectively identify procedures, policies, and guidelines that can enhance the sustainability of Tusome-related activities over the long term.

Primary data collection included interviews with Head Teachers, Curriculum Support Officers, teachers, and pupils at a nationally representative sample of 204 schools that were selected randomly at baseline. Within each school, 24 pupils (approximately 5,000 total) in grades 1 and 2 were randomly selected to complete an Early Grade Reading Assessment (EGRA) which measures phonological awareness, understanding and application of the alphabetic principle, and reading fluency and comprehension. In addition, NORC conducted key informant interviews and focus group discussions with approximately 40 stakeholders including representatives from USAID and its implementing partner as well as Kenyan education officials at the national- and county-levels.

Overall, findings provide strong evidence for the efficacy of the Tusome model of instruction. From baseline to endline, pupils have shown statistically significant improvements on all EGRA subtasks in both English and Kiswahili. Of particular note, grade 2 English reading gains observed from baseline to endline are roughly equal to gains from an additional full year of schooling. Furthermore, over the life of the USAID activity exposure to Tusome has consistently corresponded with reading performance: increased exposure to Tusome between baseline and midline/endline corresponded with an increase in pupil reading performance. Similarly, reduced exposure to Tusome between midline and endline under the revised Competency-Based Curriculum (CBC) timetable corresponded with a decrease in reading performance.

Statistical models at endline also show that better implementation of Tusome in the classroom positively correlates with reading fluency. Tusome lesson plan progress is positively associated with English reading performance: each 10-unit advance in the Tusome teacher’s guide is associated with an increase of one correct word per minute in English reading fluency. Similarly, teacher-reported frequency of pupils sounding out unfamiliar words is positively associated with English reading performance. When teachers use both English and Kiswahili while teaching Kiswahili—i.e., use “code switching"—pupils score 3.19 words per minute higher in Kiswahili.

We also find that teachers generally demonstrate moderate to high levels of support for Tusome, yet faced a number of implementation challenges. While teachers broadly support Tusome, they have struggled to keep pace with the instructional approach from the midpoint of the program onward. Insufficient time was the main implementation challenge teachers reported at endline, with 39 percent of teachers who face challenges saying they lack sufficient time to cover the content and 23 percent claiming that Tusome lesson pacing is too fast, likely owing to the reduction in instructional time from five periods to three periods per week for English and five periods to three periods per week for Kiswahili under the 2019 CBC timetable.

Finally, we find that Tusome was largely successful in securing ownership and buy-in from the government at different levels of government and at different points in the implementation process, but more work is needed to ensure long-term sustainability. Most government respondents were personally supportive of the methods and materials developed under Tusome and hoped to see them continue in the future. However, long-term sustainability of Tusome will depend on upholding existing procedures, policies, and guidelines—particularly the centralized book procurement procedure, with its facilitation of the 1:1 textbook to student ratio, and school-based coaching and teacher mentoring.

Tusome ranks among USAID’s first experiences partnering with government to take a piloted literacy program to national scale. As such, this evaluation offers an important and timely case study for translating USAID-funded pilot programs into large-scale national educational reforms. More broadly, results of this evaluation can offer key insights on strategies for transitioning donor-supported educational activities to partner governments, with the ultimate goal of ensuring program sustainability and reducing dependence on foreign assistance.

Author

• Erika Keaveney, NORC