Exploring Human, Empirical, and AI-Derived Item Classifications to Predict Item Difficulty in Educational Assessments

• In Event: Rasch Model Applications including AI

Abstract

This study examines influences of various item classifications on item difficulty. Data from an English Language Arts assessment from 5th grade students (N=796) are used. Human defined classifications (content standards and cognitive demand) are shown to predict item difficulty. Data driven classifications (e.g., item factor analysis) can also predict item difficulty. Themes obtained from large language models can effectively distinguish differences in item difficulty that align with theoretical and psychometric expectations. A combination of different item classifications for understanding item difficulty has the potential for enhancing the development and validation process of educational assessments. Different item classification approaches can lead to the identification of meaningful and interpretable item characteristics that support the psychometric evaluation of educational assessments.

Authors

• Jing Li, University of Georgia

• Cheng Tang, University of Georgia

• Chan Lu, University of Georgia

• Matthew J. Madison, University of Georgia

• George Engelhard, University of Georgia