A Method of Setting Benchmarks on Precursor Skills through Reading Comprehension

• In Event: New Technical Contributions to Measuring Foundational Skills for SDG 4.1.1a

Proposal

In most low- and middle-income countries, a significant percentage of learners do not acquire reading comprehension (RC) skills by grades 2 or 3. These learners are often at different stages of reading development, such as phonological awareness (PA), understanding alphabetic principles (AP), oral language comprehension (ORC), and decoding (DC). Reporting that nearly zero percent of learners meet RC benchmarks is not useful for making effective policy decisions or designing targeted interventions. Reporting on the percentage of learners who have mastered precursor skills offers more meaningful data in low- and lower-middle-income. This information helps countries understand where learners are on the path to achieving a benchmark on reading comprehension, guiding more informed decision-making to improve educational outcomes.

Establishing benchmarks for precursor reading skills using traditional judgmental standard-setting methods (e.g., Angoff, bookmark, body of work) can be expensive and prone to significant errors if not executed well. Instead, a three-stage data-driven benchmarking approach was employed to establish benchmarks for precursor reading skills across languages or language groups. In stage 1, benchmarks for precursor skills for a single language assessment were estimated using an Item Response Theory (IRT) based test characteristic curve (TCC) mapping approach. In stage 2, 95 percent confidence intervals (CIs) were estimated using the test information function (TIF) and standard error (SE) mapping. In stage 3, benchmarks for precursor skills across a language group were established based on meta-analysis.

Other data-driven approaches, such as cluster or regression analysis-based methods, are also available. However, these methods are correlational and population-dependent, sometimes referred to as the classical approach. Benchmarks established using the classical approach for one sample may not hold for another sample. In contrast, the IRT-based approach is population-independent, providing more robust and stable benchmarks that can be applied to assessments within the same language group. Preliminary results from a single language group show promise, with both stages 1 and 2 producing narrower CIs across assessments for the same language. Stage 3 also established benchmarks for precursor skills with low SEs.

Author

• Abdullah A Ferdous, American Institutes for Research