Student Engagement in International Large-Scale Assessments under the PIRLS 2021 Group Adaptive Design

• In Event: Division H VP Session: Past Designs, Present Tensions, Future Paths: Reimagining Educational Assessment Across Contexts

Abstract

Objectives: International large-scale assessments (ILSAs) continue to grow and provide actionable evidence to educational systems around the world. However, there is always a concern that students might not be fully engaged for these low-stakes tests. As the number of educational systems participating in ILSAs has grown and diversified, the widening proficiency differences between participants have led to discussions of accommodations being made to better target test difficulty to student ability and increase student test engagement.

In 2021, the Progress in International Reading Literacy Study (PIRLS) used a country-level group adaptive design for this purpose (Mullis & Martin, 2019). In this design, higher performing countries administered more “difficult” test booklets, while lower performing countries administered more “easy” booklets. The design aims to provide better measurement at all achievement levels and improve student engagement. This study seeks to estimate the impact of the group adaptive design on student engagement in the PIRLS 2021 reading assessment.

Perspectives: When students are not fully engaged in the test, undesirable behaviors may arise such as item omission/skipping or quitting the test (Debeer et al., 2014; Pohl et al., 2014). The group adaptive approach aims at increasing student motivation and reducing frustration, arguing that maximum engagement is attained when sets of items are tailored to the specific abilities of examinees (Ling et al., 2017; Wise, 2014). When students perceive the test as too difficult (or easy) they may be more likely to omit responses or quit early due to fatigue or lack of motivation, highlighting the importance of aligning test difficulty with student ability.

Data: We use data from 56 countries that participated in PIRLS 2021. Of the 56 countries, seven distributed a higher share of more difficult booklets and six distributed a higher share of less difficult booklets. We use several measures to capture student engagement: item nonresponse, quitting behavior, and student test enjoyment based on self-reports.

Methods: We estimate booklet effects for each of our student engagement measures. Using these parameter estimates, we use simulations to estimate the effect of switching from a random distribution of test booklets (50/50 more difficult to less difficult distribution) to one of the group adaptive design distributions (70/30 or 30/70 for high-performing and lower-performing countries, respectively).
Results: Preliminary results indicate that more difficult test booklets had higher item nonresponse/quitting behavior and lower student enjoyment across all countries. As a result of these effects, the countries that administered a higher share of less difficult booklets experienced increases in student engagement under the group adaptive design. However, the higher-performing countries that distributed a greater number of more difficult booklets experienced decreases in student engagement.

Significance: Overall, the findings speak to the importance of designing assessments that match the abilities of test takers, a difficult task in ILSAs. This evidence has practical implications for the administration of ILSAs as it highlights the potential benefits of tailoring tests to different student ability levels but also points to areas where the assessment can be improved to provide better evidence to all educational systems.

Authors

• Alec Kennedy, IEA Hamburg

• Yuan-Ling Liaw, IEA Hamburg