Evaluating Examinee Engagement Through Modeling the Relation Between Response Time and Item Word Count

• In Event: Assessment in the Advent of Process Data and AI

Abstract

“All things equal,” providing students with more items will lead to more reliable scores. The problem is, all things are rarely equal, and students may disengage from the testing endeavor when tests are too long. This study evaluates this possibility by modeling the relation between item word count (IWC) and response time (RT) and the moderating role of ability estimates and item position. These models allow us to use the variability in the IWC-RT relation to gauge test-takers’ engagement as a function of item position.

We tested a series of cross-classified random-intercept linear mixed effects models with data from students in grades 2 to 12 who have completed a computerized adaptive untimed test of reading (N = 717,475). In these models, RT was the outcome variable and IWC, ability estimate, and item position were the predictors. Considering the interdependence between responses in the same test as well as the responses to the same items, the model intercepts were nested in these two grouping variables. The results, across all grades, indicated a robust association between IWC and RT which was moderated by both ability estimates and item positions. Notably, the three-way interaction revealed fundamental differences in response processes of students with different ability estimates. Higher ability students spend more time on longer items, regardless of their position in the test. However, lower ability students tend to spend more time on longer items mostly in the beginning. For this group, as the test progresses, the relation between word count and response time becomes weaker, to the point that this relation approaches zero sometime after the 40th item, suggesting a disengagement pattern among lower ability students when administered longer items towards the end of the test.

These results support the notion that administering more items, especially if they are lengthy, might not always improve the accuracy of test results. Further, the negative effect of longer tests is magnified for students with lower ability estimates, who tend to show a stronger disengagement tendency in later stages of the test compared to the medium and high ability estimate counterparts.

Authors

• Pooya Razavi, Edmentum

• Sonya J. Powers, Edmentum