When Cut Scores Impact Human Ratings: A Many-Facet Rasch Modeling Approach

• In Event: Advances in Psychometric and Machine Learning Approaches to Assessment

Abstract

The standard many-facet Rasch model (MFRM) for analyzing and evaluating rater-mediated assessments is focused on between-rater differences in overall severity or leniency across facets and rating scale categories. To examine local, category-dependent severity effects, we propose the many-facet Rasch model for rater–category threshold interactions (MFRM-RCT). In a simulation study and real-data analysis, we found that (a) differences in category-dependent severity levels impacted observed score distributions and pass–fail decisions, (b) the MFRM-RCT reliably recovered true overall and local rater severity parameters, (c) ignoring rater–category threshold interactions biased overall severity and category threshold estimates, and (d) the MFRM-RCT outperformed the traditional MFRM regarding data–model fit when applied to essay rating data.

Authors

• Kuan-Yu Jin, Hong Kong Examinations and Assessment Authority

• Thomas Eckes, University of Bochum