Exploring Rater Accuracy Using Unfolding Models Combined With Topic Models Incorporating Supervised Latent Dirichlet Allocation

• In Event: Innovative Approaches to Modeling Educational Data

Abstract

Objectively rating extended-response items on educational assessments has long been an issue due to rater biases. Even well-trained raters using a rubric can inaccurately assess essays. Unfolding models capture the inaccuracies between criterion and operational ratings by placing essays on a difficult-to-score latent continuum where essay locations are referred to as the unfolding location. This study uses a sample of essays (n=100), along with their unfolding location, to estimate a supervised Latent Dirichlet Allocation (sLDA) model to investigate the relationship between latent topics and unfolding locations. Results show that the latent topics moderately explain (r^2=0.561) difficult-to-score essays defined by the unfolding locations. This information can be used to improve rater training by identifying sources of error.

Authors

• Jordan M. Wheeler, University of Georgia

• Jue Wang, University of Science and Technology of China

• George Engelhard, University of Georgia