Differential Effects of Process Features on Item Functioning: An Analysis of PIAAC (Programme for the International Assessment of Adult Competencies) Data (Poster 5)

• In Event: Excellence in Education Research: Early Career Scholars and Their Work Poster Session

Abstract

Highlighting the increased adoption of technology-based cognitive assessments, recent literature underscores the need to examine how test-takers’ response processes influence item functioning. This study investigates the critical relationship between variations in test-takers item response processes and item success, conditioned on test-takers’ abilities and demographic backgrounds. Utilizing machine learning methods and a generalized linear modeling (GLM) approach, this study proposes a measure to identify differential effects of process functioning (DEPF), which arise from the deviations in test-takers’ response process due to demographic differences and affects the overall item success probability for specific groups. The results suggested that variations in response processes may reveal fairness issues in technology-based items, as certain item components could disadvantage specific groups of test-takers.

Authors

• Wenchao Ma, University of Minnesota

• Mehdi Rajeb, University of Alabama

• Qiwei He, Georgetown University