Examining the predictive validity of an assessment battery to identify key reading precursors for EdTech in developing contexts

• In Event: Highlighted Session: Across ages and stages: effects and effectiveness drawn from literacy assessments for children and youth

Proposal

In recent years, education technology (EdTech) programs have become increasingly prevalent in Sub-Saharan Africa and have been shown to be one of the most effective education interventions in developing contexts (Conn 2017; McEwan 2015; Damon et al., 2016; Rodriguez-Segura 2020). Rigorous research conducted in Africa over the past decade has shown the positive effects of onebillion’s onecourse, an award-winning tablet-based curriculum, on children’s literacy development in both in-school and out-of-school settings (King et al. 2019; Levesque et al., 2020; Levesque et al., 2022; Pitchford et al., 2017). A 15-month randomized controlled trial (RCT) conducted for the Global Learning XPrize with out-of-school children in Tanzania produced effect sizes of 0.46-0.59 in early literacy skills (King et al., 2019). Further, two RCTs with Malawian Standard 2 learners in government primary schools in 2018-19 (8 months) and in 2019-21 (13 disrupted months) produced effect sizes of 0.34 and 0.37 in overall literacy, respectively (Levesque et al., 2020; Levesque et al., 2022). Gains in literacy are also associated with a wide range of positive outcomes for children, including long-term school achievement, school retention, employment, and life expectancy (Duncan et al., 2007; French, 2012; Gilbert et al., 2018). However, while Edtech programs that support literacy development have the potential to shape the trajectory of children’s lives, it is critical that these programs can support diverse learning needs to ensure equitable access to educational content.

While onecourse interventions have produced positive average literacy effects, further analysis revealed that a substantial portion of children who used the tablet-based curriculum remained non-readers at the end of the intervention periods: 77% after 8 months, 42% after 13 months, and 41% after 15 months. Being a “non-reader” means not being able to read a single word of connected text (words presented in sentences and paragraphs), as measured by the Early Grade Reading Assessment (EGRA; Dubeck & Gove, 2015). A recent exploratory study conducted by Bardack and colleagues (2023) examining divergent patterns in children’s reading progressions at a peri-urban Malawi primary school showed that children’s working memory skills emerged as the strongest unique predictor of high versus low progress in reading while controlling for other factors. These findings align with prior research conducted on a sample of first-grade learners in the Netherlands who participated in a tablet-based word reading efficiency game intervention, which found that multiple reading precursors that were related to children’s in-game accuracy (working memory, phonological awareness and letter efficiency) and in-game efficiency (phonological awareness and rapid automatic naming [RAN]) predicted higher growth in reading efficiency (van Utter et al., 2022). Yet, more research is needed to better understand how these critical reading precursors may co-occur to support children’s reading progress in tablet-based curriculum interventions in a developing context.

The current proposal presents results from a study designed to assess the reliability and predictive validity of a battery of assessments that will be used to replicate and extend prior research identifying important predictors of reading progress (Bardack et al., 2023; Daneman & Carpenter 1980; Dubeck et al., 2017; Engel de Abreu et al., 2014; Follmer 2018; van Utter, 2023). The battery includes three assessments that test known predictors of early literacy and that have been previously administered in Chichewa in Malawi: a digitized spatial span measure of working memory (Pitchford & Outhwaite, 2016) as well as an initial syllable measure of phonological awareness and a rapid automatic naming (RAN) measure of processing speed (Dubeck et al., 2017).

We purposively selected two primary schools that are participating in the nationwide scale-up of the tablet-based curriculum through the Building Education Foundations through Innovations & Technology (BEFIT) program in Malawi. Although Standards 1-4 are participating in the BEFIT program, this study focuses specifically on Standard 2 learners to build on previous related work conducted on Standard 2 learners in Malawi government public schools (Bardack et al., 2023, Levesque et al., 2020; Levesque et al., 2022). In both schools, all Standard 2 learners (approximately 350 children) were invited to participate in the baseline assessment that was conducted in February 2024. All baseline participants (307 learners) completed the battery of three assessments (digitized spatial span task, initial syllables task, and the RAN task) as well as digital versions of the Early Grade Maths and Reading Assessments (EGRMA) to assess children’s literacy and numeracy skills. During the endline assessment, we retained 93% of the sample (284 children) and only administered the digital EGRMA assessment in order to test the predictive validity of the baseline battery. In addition to evaluating the reliability of these four assessments on the retained sample, we are running regression analyses to examine whether baseline measures of children’s working memory, phonological awareness and processing speed predict children’s literacy and numeracy gains.

With the endline data collection recently concluded in July 2024, we will present our full findings at the 2025 CIES conference. Our results will be used to: (1) assess whether these instruments are reliable and valid for continued use; (2) determine the feasibility and utility of a battery of assessments to identify children who are at-risk for non-progress in reading and; (3) contribute to continuous improvement of the onecourse software during the BEFIT program. As the BEFIT program scales to provide access to child-directed, tablet-based learning for all lower primary school children in Malawi, this research will contribute to ensuring that children who are at-risk for non-progress in reading can benefit from the program Thus, this work has the potential to contribute important insights on how the adoption of technology in classroom settings can be used to promote equity across different types of learners.

Authors

• Sarah Bardack, Imagine Worldwide

• Margaret Peggy M Dubeck, RTI International

• Celeste Lopez, Imagine Worlwide

• Karen Levesque, Imagine Worldwide