A Mixture Rasch Facets Model for Raters' Illusory Halo Effects

• In Event: Novel Approaches for Model Fit

Abstract

A rater's overall impression of a ratee's essay (or other assessment) can influence ratings on multiple criteria to yield excessively similar ratings (halo effect). Hence, we introduce and test a mixture Rasch facets model for halo effects (MRFM-H) that distinguishes true versus illusory halo effects and classifies normal and halo raters. In a simulation study, when raters assessed enough ratees, MRFM-H accurately identified halo raters. Also, more rating criteria increased classification accuracy. Ignoring halo effects (via a simpler model) biased parameters for evaluation criteria and rater severity but not ratee assessments. MRFM-H application to three empirical datasets showed (a) experienced raters' illusory halo effects, (b) fewer illusory halo effects with more criteria; and (c) more versus less informative survey responses.

Authors

• Kuan-Yu Jin, Hong Kong Examinations and Assessment Authority

• Ming Ming Chiu, The Education University of Hong Kong