Navigating heterogeneity in learning outcomes in Rwandan classrooms: empowering education through tech-enabled customized instruction across the educational system

• In Event: Policy, Governance of Education Systems

Proposal

Motivation

Learning outcomes in low- and middle-income countries (LMICs) are consistently low (World Bank, 2018). However, curriculum expectations remain much higher than where most pupils stand, indicating a misalignment between student performance and policymaker expectations. This discrepancy between instruction and the low levels of learning is one of the contributing factors to the persistent learning crisis (Rodriguez-Segura and Mbiti, 2022).

Beyond low average learning levels across many LMICs, researchers have also documented considerable heterogeneity in learning outcomes within classrooms (Muralidharan et al., 2019). This heterogeneity makes it difficult to cater instruction within classrooms, as teachers can deliver more targeted and effective instruction when classes are more homogenous (Duflo et al., 2011). When teachers must target their instruction to a narrow segment of the performance distribution, a broader baseline distribution of performance will result in instruction that only reaches a smaller number of students. Thus, the literature is appropriately concerned about the issue of within-class heterogeneity. Yet, at a macro-level, policymakers face a similar challenge when they perform activities like setting curricula, developing mastery benchmarks, and writing textbooks. That is, policymakers are tasked with selecting a narrow level to cater to in each grade. Therefore, in the face of within-system heterogeneity, it is difficult to select a single level to be taught across a large territory and effectively meet the needs of all students.

Collectively, the existing literature has not focused as much on within-system heterogeneity of foundational learning outcomes, often focusing on either the overall low learning outcomes (World Bank, 2018) or discussing within-class heterogeneity (Muralidharan et al., 2019). While it is necessary to acknowledge these challenges, it is equally important to recognize that pedagogical needs vary, even among students performing at low levels. Our work protests the status quo of system-wide curriculum leveling, in support of a more equitable approach that appropriately caters to schools and grades within an education system. As such, we will present work that, first, quantifies the extent of heterogeneity within a representative sample of a large number of public schools in Rwanda, mapped against their respective pedagogical needs towards mastering foundational literacy and numeracy, and second, provides preliminary insight on how to mitigate this using technology-enabled school-grade differentiation.

Research Questions

How significant is within-system heterogeneity in foundational learning outcomes in Rwanda, and what implications might this have on pedagogical needs and curricular expectations?

How can technology-enabled instruction be leveraged to better address the needs of children, in the presence of within-system heterogeneity?

Methods and Data

Our descriptive analysis relies on two pillars. First, the collection of foundational literacy and numeracy data for a representative sample of 40 schools from across Rwanda, which represents more than 511 schools and 600,000 students across the country. This dataset includes measures of oral reading fluency and a numeracy assessment with a wide range of skills measured, demonstrating a high level of internal validity. Second, we leverage the pedagogical needs of different numeracy and literacy programs, which are mapped against the literacy and numeracy assessments used in this study. This helps us understand the pedagogical requirements of each student in the sample, as well as the broader needs for each school and grade.

Furthermore, we describe an ongoing randomized controlled trial (RCT) across 761 schools in Rwanda aimed at evaluating the effectiveness of a more carefully planned school-grade leveling approach in numeracy and literacy. This approach stands in contrast to the current status quo of a single pedagogical level for each grade across the system.

Preliminary Findings

Our study on system-level heterogeneity in Rwanda highlights several facts:

First, substantial heterogeneity exists in upper primary grade levels (Primary 4-6). There are very large potential gains to differentiating curriculum levels in literacy and numeracy across education systems.

Second, learning levels are fairly homogeneous in lower primary (Primary 1-3), suggesting that, at least in this context, this leveling method is not necessary in the lower grades and the status quo of setting one curricular expectation for the entire system might be enough.

Third, we demonstrate how a technological platform can be leveraged to determine the appropriate learning levels for schools and provide instructional materials that align with these levels. We will briefly discuss an evaluation of this approach through an ongoing RCT.

Fourth, we discuss challenges to successfully implementing technology to cater to children’s individual learning levels. For example, measurement error presents the possibility of incorrectly identifying a school’s curriculum levels, regardless of accuracy in utilizing the platform. Currently, we estimate that assessing 8-10 children per school grade allows us to make the correct decision 70% of the time. However, even with measurement error, our approach still captures more students than current practices.

Contribution

Existing studies show that customized interventions are more effective in increasing student achievement for both high- and low-performing students, compared to more standardized interventions (Duflo et al., 2011; Muralidharan et al., 2019; Rodriguez-Segura, 2021). Additionally, the literature highlights that technology-enabled interventions are highly cost-effective and can help to better tailor curriculum at a specific level (Rodgriguez-Segura, 2021). Despite these contributions, there is still a lack of research that has explored system-wide heterogeneity and the impact of implementing differentiated levels of instruction within a system. Our work fills this gap by exploring how school-grade leveling can inform systematic approaches to more equitable learning opportunities and student outcomes. Ultimately, three key themes emerge from our findings:

Policymakers typically establish learning expectations and benchmarks based on a single learning level for each grade, resulting in mismatched curricula that fail to effectively reach all students in a given grade or school, especially in the presence of within-system heterogeneity.

System-wide heterogeneity is very prevalent, with larger learning gaps emerging in upper primary grades. The significant variation in learning levels across schools and grades suggest that potentially large learning gains can result from differentiating schools’ curricula.

Students and teachers could benefit from a customized intervention that tailors curricula to accommodate various performance levels. Using technology-enabled instruction, the educational needs of more students can be met, resulting in more equitable learning opportunities.

Authors

• Timothy Sullivan, NewGlobe

• Daniel Rodriguez Segura, University of Virginia