Measuring reading comprehension in LMICs at scale and with greater sensitivity to diverse literacy abilities

• In Event: Understanding and measuring reading comprehension in multilingual, low- and middle-income countries: emerging research from the RELM and LITES studies

Proposal

Scholars have critiqued the existing reading comprehension measurement strategy in EGRA (Hoffman, 2012; Kim et al., 2016; Zuilkowski et al., 2019; Bartlett, et al., 2015; Dowd & Bartlett, 2019), leaving some to conclude: “Although reliable measures exist to assess emergent literacy skills in international contexts, there is little consensus on the assessment of reading comprehension” (Zuilkowski et al., 2019, p. 583). Additionally, scholars note a particular challenge in the current model of reading comprehension measurement in the capacity of measures to identify students’ skills at lower-levels of reading proficiency, where the high number of zero-scores provide little actionable diagnostic information to inform policy or instructional responses (Hoffman, 2012; Zuilkowski et al., 2019). As such, scholars have recommended that “future research should explore other options for reliably measuring [reading comprehension] assessment, particularly at very low literacy levels (Zuilkowski et al., 2019, p. 584).

This paper examines the benefits, limitations, and psychometric properties of three distinct measurement approaches to the assessment of students’ reading comprehension across six LMICs and 16 languages. These three approaches are as follows:

1. the conventional EGRA task, which is 5 reading comprehension questions added to a 1-minute, timed, oral reading fluency passage;
2. the typical EGRA task, but allowing students to “look-back” at the text after having been asked the questions, an effort to mitigate possible conflation of working memory with reading comprehension as the measured constructs, and
3. An untimed reading comprehension task using single or multiple sentence prompts of increasing difficulty and multiple picture choices on a stimuli from which the students choose the picture that most closely approximates the meaning of the text-prompt (reading comprehension picture matching). This task was adapted from previous large-scale assessments led by AIR in India (AIR, Foundational Learning Study, 2022) and is similar to approaches used in some leading U.S.-based academic testing companies (e.g. the NWEA MAP test).

Preliminary analysis suggest that the reading comprehension picture-matching task is correlated at moderately high levels with the conventional EGRA task. Most notably, the number of zero scores for the picture matching task are much lower and the distribution less skewed, suggesting that significantly higher proportions of students are able to demonstrate some reading comprehension abilities through this distinct task design.

Additional analysis of the psychometric properties of these various measurement approaches are ongoing and will be reported in greater detail in the presentation.

Authors

• TJ D'Agostino, University of Notre Dame

• Jill Pentimonti

• Pooja Nakamura, American Institutes for Research

• Annie Conaghan, University of Notre Dame-Pulte Institute of Global Development