Understanding and measuring reading comprehension in multilingual, low- and middle-income countries: emerging research from the RELM and LITES studies

Group Submission Type: Formal Panel Session

Proposal

This panel presents the results of three works in progress from a unique cross-country dataset of 7,200 students' language and literacy skills in the 3rd and 4th grades across 6 countries and 16 languages. Each presentation offers a novel contribution to the field’s ongoing efforts to understand how to measure and foster reading comprehension in low- and middle-income countries.

With the advent of simple, cost effective assessments of students' foundational literacy, like the Early Grade Reading Assessment (EGRA) and ASER, the world has become aware of low rates of foundational literacy skills in low- and middle-income countries. International actors have dubbed this first a learning crisis, and more recently “learning poverty,” which is defined as a 10 year old child not being able to read a simple text with comprehension. These problems have been persistent at a global scale, despite occupying the international educational development agenda for the better part of the last two decades.

While progress has been made in developing interventions that work to improve some foundational reading skills, the goal of helping to significantly increase the rates of children reading with comprehension at scale remains largely unmet. Reading comprehension is “the process of extracting and constructing meaning through interaction and involvement with written language” (RAND study group, 2002, p. 11). It is the ultimate goal of reading.

In a review of evidence from structured reading programs, many interventions showed no effect in reading comprehension (see Kim et al., 2016), and more modest effect sizes were reported for reading comprehension in a meta-analysis compared to emergent literacy skills (Kim et al., 2020). Moreover, many scholars have critiqued the existing reading comprehension measurement strategy in EGRA (Hoffman, 2012; Kim et al., 2016; Zuilkowski et al., 2019; Bartlett, et al., 2015; Dowd & Bartlett, 2019), though no new designs for cost-effectively measuring reading comprehension in LMICs has emerged and gained traction (Zuilkowski et al., 2019).

In addition to measurement, too little is known about what prevents children from reading with comprehension in multilingual contexts. Specifically, a significant limitation of existing studies is the assumption that learners have adequate exposure to the language of instruction. This assumption is often invalid in multilingual contexts, where millions of children are learning to read in languages they may not speak or use at home or in their communities (World Bank System, 2021). Since learning to read in any language is critically dependent on oral language skills, it is not surprising that these skills play an even more significant role with multilingual learners (August & Shanahan, 2006; Jeon & Yamashita, 2014; van den Bosch, Segers, & Verhoeven, 2020; Raudszus, Segers, & Verhoeven, 2021). Unlike monolingual learners, it cannot be assumed that they begin school with sufficient oral language proficiency to read with comprehension (Babayigit, 2015; Melby-Lervag & Lervag, 2011).

This panel explores these issues in novel but important ways, by considering how reading comprehension in multilingual contexts can be effectively measured, understood, and deconstructed into underlying skill gaps for specific students. It features three papers that are works in progress from two inter-related studies, the LITES study (Language of Instruction Transitions in Education Systems) and the RELM pilot study (Receptive and Expressive Language Module), both funded by USAID as part of the SHARE activity.The countries included: Mali, Senegal, Mozambique, Kenya, Rwanda, and the Philippines.

This study offers a unique dataset that includes students’ language and reading skills in two languages for 3rd and 4th grade students. These unique data allow for a robust inquiry into the underlying - linguistic and literacy skill - components of reading comprehension. Additionally, beyond testing new methods of assessing students’ language skills at scale in the RELM pilot, LITES also tested three distinct measurement strategies to assess reading comprehension (building upon previous work in India from AIR).
The three presentations will focus on the following:

The first paper explores three different approaches to measuring reading comprehension and considers the distinct psychometric properties of each. The three approaches are the conventional EGRA task, based upon a 1-minute timed ORF passage with 5 questions, a modified version of this task that allows students to look back at the passage (to avoid conflating short-term working memory with reading comprehension), and a newly adapted task using one or more sentences as prompts and multiple choice picture matching, untimed but with stop-rules and an increasing level of difficulty over 7-10 items. Preliminary results suggest that the picture matching task has far fewer zero scores and is much better able to detect a range of students’ reading comprehension abilities with greater sensitivity than the typical EGRA reading comprehension task.

The second paper tests a theoretical model of reading comprehension and reading skills transfer, based upon the simple view of reading as applied to multilingual contexts. It tested numerous, novel linguistic landscapes and language pairs that have never been assessed before. Modeled upon previous research using structural equations modeling (SEM) (Proctor, et al., 2010), this presentation explores the degree to which language pairs of local languages from 6 LMIC countries and second languages (English, French, Portuguese, and Filipino) fit the theorized model of the simple view of reading for learners in multilingual contexts, exploring and noting any variety in the key relationships of interest (e.g. the degree to which decoding skills correlate across languages and the strength of the relationships between language skills, decoding skills, and reading comprehension respectively across language pairs.

A third and final paper will use the RELM and LITES data from three countries, the Philippines, Rwanda, and Kenya, to analyze sub-groups of students who are failing to read with comprehension. It uses Latent Class Analysis (LCA) or sub-group analysis, to identify distinct groups of students among non- or low- reading comprehenders, enabling the researchers to identify the nature and prevalence of the underlying skill gaps impeding reading comprehension. In other words, whether the students’ have language gaps, decoding gaps, or a combination of both, that primarily explains low reading comprehension across these three contexts.

Sub Unit

• Global Literacy SIG

Organizer

• TJ D'Agostino, University of Notre Dame

Chair

• Pooja Nakamura, American Institutes for Research

Individual Presentations

• Measuring reading comprehension in LMICs at scale and with greater sensitivity to diverse literacy abilities - TJ D'Agostino, University of Notre Dame; Jill Pentimonti; Pooja Nakamura, American Institutes for Research; Annie Conaghan, University of Notre Dame-Pulte Institute of Global Development

• The simple view of reading across 14 new language pairs: evidence from Africa and the Philippines - Annie Conaghan, University of Notre Dame-Pulte Institute of Global Development; TJ D'Agostino, University of Notre Dame; Pooja Nakamura, American Institutes for Research; Jill Pentimonti

• Unveiling the Learning Crisis: Understanding the Reading Comprehension Challenges in Multilingual Contexts - Fernanda Soares, University of Notre Dame; Caroline Freeman, University of Notre Dame; Pooja Nakamura, American Institutes for Research; TJ D'Agostino, University of Notre Dame

• Understanding the language subskills and other school and student factors predicting reading comprehension in the Mozambican bilingual education program - Carlos Joaquim Manuel, Universidade Eduardo Mondlane