The Relationship Between Rater Experience and Performance Ratings: A Systematic Review

• In Event: 15.016 - Advancements in Human Scoring: Bringing Together Multiple Approaches

Abstract

This research investigates the effects of rater experience on performance ratings in language testing using a systematic review of studies published from 1985 to 2017. Based on a comprehensive literature search of 14 databases, we identified sixteen relevant papers. With these we conducted a narrative review to conceptualize a theoretical framework, which we used for meta-analysis. The two-step investigation revealed that most of the studies were conducted on the assessment of writing performance and that there is no consensus for defining rater experience. This yields heterogeneous operationalization as well as mixed empirical results. Moreover, most of the studies adopted quantitative approaches, and the findings of the meta-analysis suggest that, in general, score raters differ significantly among novice and experienced groups.

Authors

• Jing Huang, The University of Hong Kong

• Gaowei Chen, The University of Hong Kong