Impact of Test Design Decisions on Test-Taker Engagement

• In Event: The Impact of Item, Test, and Test-Taker Characteristics on Engagement During NAEP Assessments

Abstract

Test-taking engagement (TTE) has been identified as a threat to the validity of low-stakes assessment outcomes (Wise & Smith, 2016). Thus, it is important to understand how test design decisions during the development process impact TTE. We explored the impact of design decisions within the 2019 NAEP 8th grade mathematics assessment (N=114,898), in particular, their potential impact on the consistent finding that TTE degrades as test-takers move from the beginning to the end of the assessment (Wise, 2017). TTE was identified based on response times using the Normative Method with a 20% threshold (Wise & Ma, 2012).

This NAEP assessment included several design decisions that could potentially impact TTE. We focused on three design decisions: (1) division of the assessment into two independent blocks, (2) assignment of blocks to first or second position, and (3) number of items within blocks. We investigated each design decision separately for the nine unique blocks distributed across test-takers. We also investigated how these design decisions may differentially impact test-takers at different proficiency levels (National Center for Education Statistics, 2021).

The imposed “break” between the two blocks in the assessment could potentially cause an increase in TTE from the Block 1 final item to the first item in Block 2. Table 1 shows the percent change of TTE from the end of Block 1 to the start of Block 2 overall and across proficiency levels. These findings reveal that the imposed “break” can create a “boost” in TTE (positive percent change), but this does not occur for all test-takers.

Individual blocks could occur in Position 1 or Position 2 when assigned to individual test-takers, and this assignment could impact the degree of TTE that the items within that block received. We compared the effect size difference in TTE between Position 1 and Position 2 for the nine individual blocks and found that overall, there was not a meaningful difference in TTE based on block position overall, but there was evidence for a possibly meaningful difference for test-takers below the NAEP Basic proficiency level (Table 2). Further analyses for below Basic test-takers revealed that when a block was in Position 1, it received more TTE than when in Position 2.

Each block consisted of 15 to 18 items, which is shorter than the typical assessment length in TTE research (e.g., 30+ items), which could impact the established finding that later items receive less TTE than earlier items (Wise, 2017). We conducted a series of linear regressions and found that item position consistently predicted TTE, such that later items received less TTE than earlier items (Table 3).

Overall, these findings suggest that some test design decisions did impact TTE (division into two blocks), and others did not (block position, block length) in the present context. However, the differential impact of these design decisions for test-takers at different proficiency levels highlights the importance for further research to better understand the impact of design decisions on TTE to inform assessment development and the maximization of TTE for all test-takers.

Authors

• Blair Lehman, Educational Testing Service

• Jonathan Steinberg, EurekaFacts

• Jesse R. Sparks, Educational Testing Service

• Jennifer Bailey, ETS

• Fred Yan, Educational Testing Service

• Jung Aa Moon, Honeywell