Using implementation research for a root cause analysis of barriers to quality implementation during a nation-wide edtech scale-up in Malawi

• In Event: Overcoming Barriers and Navigating Processes: Expanding Technology Use in Education

Proposal

Education technology (EdTech) programs have become increasingly prevalent in Sub-Saharan Africa and have been shown to be one of the most effective education interventions in developing contexts (McEwan 2015; Damon et al., 2016; Conn 2017; Rodriguez-Segura 2020). Over the past decade, multiple rigorous studies of onebillion’s tablet-based software onecourse have consistently shown positive and significant learning impacts in literacy and numeracy across learning settings, languages, and countries (King et al., 2019, Levesque et al., 2020, 2022; Pitchford et al., 2017). onecourse is an award-winning tablet-based curriculum geared towards developing children’s foundational literacy and numeracy skills in both in-school and out-of-school settings (Pitchford et al., 2017; King et al., 2019; Levesque et al., 2020; Levesque et al., 2022). This “adaptive” software provides early grade instruction that is personalised to meet the proficiency level of each child.

In Malawi learning poverty is estimated at 87%. Moreover most students, whether or not they complete the first few years of primary school, are not becoming fluent readers. As a response to these challenges, the Malawi Ministry of Education launched the Building Education Foundations through Innovation & Technology (BEFIT) program, a national scale-up of onecourse to all lower primary school children in Malawi in 2022. The BEFIT program delivers supplemental instruction aligned with the national curriculum in Chichewa through onecourse during dedicated periods in the school timetable (targeting a minimum of 150 minutes per week). The tablets are rotated among classrooms so that all children in grades 1-4 use the tablets daily. The vision is to empower every primary school child in Malawi with the foundational literacy and numeracy skills they need to reach their full potential. BEFIT is being rolled out over 6 years starting in September 2023 to all 5,770 public primary schools, ultimately serving 3.5 million learners in Standards 1-4 annually. Scaling this program country-wide is anticipated to significantly increase foundational literacy and numeracy skills for students.

While onecourse interventions have produced positive effects in both literacy and numeracy, careful monitoring is needed to ensure sustained effects upon scaling. Without proper monitoring of implementation, the quality of delivery and uptake may be compromised at such a large scale. Many programs produce reduced effects at scale, in part because implementation factors are not carefully considered (Stern et al., 2021). In an effort to ensure the continued positive effects on children’s literacy and numeracy development as BEFIT scales, we are conducting Implementation Research (IR) during the first years of the program rollout. IR can be defined as the study of methods to improve the uptake, implementation, and translation of research findings into routine and common practices, obtaining information to guide scale-up and sustainability (MEASURE Evaluation Implementation Research Technical Working Group, 2015). IR seeks to understand what, why, and how interventions work in “real world” settings and to test approaches to improve them (MEASURE Evaluation 2012; Peters et al. 2013; USAID 2014).

This paper will cover Imagine’s IR efforts related to the Malawi BEFIT program which seek to ensure that the learning gains obtained previously are maintained at scale. IR findings are shared termly and annually with BEFIT program managers, implementers, and other stakeholders, to inform ongoing program monitoring as well as subsequent-year implementation plans, trainings, and sensitizations. During BEFIT Year 1, IR efforts focused on identifying barriers to and enablers of quality implementation through site visits, including capturing variations in implementation across sites. Site visits involving stakeholder interviews and tablet session observations were conducted in each of the 17 Year 1 districts, including 2 schools in different zones in each district, resulting in visits to a total of 34 BEFIT Year 1 schools and zones. Approximately 5-6 districts were visited in each of the three school terms. The site visits were designed to represent all districts that were part of Year 1.

Following completion of the BEFIT Year 1 IR data collection in July, the Imagine and Malawi IR teams have engaged in a root cause analysis of the barriers to quality implementation. The most critical barriers to quality implementation can be grouped into two main categories: (1)
barriers to learning gains and (2) barriers to program sustainability. The most frequently encountered barriers to learning gains include the following: (1) community fear in that some parents, community, and religious leaders initially discouraged learners from attending the program as they were fearful that the tablets were Satanic or “blood-sucking” devices and (2) local discretion in program delivery led to wide variations in time-on-task as some schools scheduled only 90 minutes per week - the minimum government guidance - and others as much as 300 minutes per week. The most frequently encountered barriers to sustainability included the following: (1) teacher engagement was affected in some cases by perceptions that the program was not a government program leading some teachers to expect to be compensated for their participation; an inadequate number of teachers in some schools also hindered session facilitation efforts and (2) engagement of district and zonal officials was also affected in some cases by a misunderstanding about their role in the program, combined with capacity strengthening of local government officials not being prioritized during BEFIT Year 1.

Further analysis will be conducted and presented at the 2025 CIES conference where we will discuss the ways in which many of the barriers mentioned above have already begun to be addressed by the BEFIT implementation team. Furthermore, a root cause analysis of the barriers is ongoing and will be shared with key BEFIT stakeholder groups during workshops in Malawi in October, where stakeholders will collaborate to identify the most critical factors and promising approaches for implementation during BEFIT Year 2. Coming out of these workshops, Imagine and University of Malawi colleagues will develop an IR agenda for Year 2, including possible design-based implementation research (DBIR) on alternative methods for addressing critical barriers to quality implementation.

Authors

• Karen Levesque, Imagine Worldwide

• Celeste Lopez, Imagine Worlwide