Evaluating the Validity of Filtering Disengaged Examinees From Low-Stakes Assessment

• In Event: 70.066 - Roundtable Session 28
  In Roundtable Session: 70.066-9 - Evaluation of Validity

Abstract

Ensuring examinee engagement is important to improve the validity of score-based inferences from low-stakes assessments. To this end, researchers often remove data from test-takers deemed to be disengaged (examinee-level filtering). However, this approach assumes that rapid-guessing (e.g., rapid-guessing) are unrelated to ability as removal of low ability examinees would artificially inflate raw score means. This study evaluated the validity of this underlying assumption as well as investigated whether rapid-responses significantly biased reporting unfiltered mean performance. In general, disengaged examinees were found to have significantly lower ability than their engaged counterparts on the SAT and a college proficiency assessment. Furthermore, when filtering at the response-level, rapid-guessing was found to have a negligible impact on mean performance.

Authors

• Joseph A. Rios, Educational Testing Service

• Hongwen Guo, ETS

• Ou Lydia Liu, ETS