The Effects of Incorrect Answer Substitution in PIRLS

• In Event: Future Directions of Educational Measurement in International Large-Scale Assessments

Abstract

The present study's main objective is to analyze and measure the impact of incorrect answer substitution in the Progress in International Reading Literacy Study (PIRLS).

In international large-scale assessments, students may not be compelled to answer every test item: a student can decide to skip a seemingly difficult item or may drop out before the end of the test is reached. The way these missing responses are treated will affect the estimation of the item difficulty and the student ability, and, hence, may potentially affect the country's score. Nevertheless, it is possible to jointly estimate the generating missing data process and the cognitive process of the student via a tree-based item response model or IRTree (De Boeck, & Partchev, 2012). An IRTree can describe the individual's internal cognitive process using a sequence of interconnected steps (De Boeck, & Partchev, 2012; Jeon & De Boeck, 2016) and allowing to use a single observed response to estimate multiple latent variables. Thus, separating the probability of attempting to answer an item from the probability of a correct response becomes possible (Debeer, Janssen, De Boeck, 2017).

For the present study, PIRLS data from 2006 to 2016 were analyzed. In PIRLS, skipped and not reached items are treated as incorrect responses. These data were analyzed using IRTree models (De Boeck, & Partchev, 2012) in which a sequential tree structure is estimated to model the full response process. A Rasch baseline model and an IRTree model were estimated for the whole PIRLS sample, using a resampling technique due to the large sample size, obtaining concurrent results. With these estimated parameters, student ability was estimated and then aggregated by country.

Large-scale international assessments are commonly low-stakes for the participating students, while they may be high-stakes for national policymakers. Therefore, it is interesting to incorporate effects of low test motivation when estimating a country's ability in a specific domain. While the effects of treating missing responses as incorrect answers were relatively small in PIRLS, it appears that some countries may be able to boost their performance by just teaching test wiseness (e.g., guessing an answer instead of leaving the item blank) to their students. Hence, additional IRTree analyses on response omission may generate useful information for future international large-scales assessments.

The reestimated country means showed a strong correlation with official PIRLS results, not altering the country rankings significantly. Nevertheless, when analyzed at the country level, the impact of using an IRTree model is higher for countries with a high percentage of skipped items. Moreover, for these countries, the model selection significantly impacted their trend between cycles.

Authors

• Andrés Christiansen, Katholieke Universiteit Leuven

• Rianne Janssen, KU Leuven