Examining the Practical Consequences of Rater Errors and Rater Misfit in Performance Assessments

• In Event: Rasch Models: Considerations and Use

Abstract

Rater effects (e.g., severity and centrality) are persistent concerns in performance assessments. Accordingly, there is a large body of literature devoted to detecting these specific rating patterns. Despite their persistence in practice and frequent discussion in research, the practical implications of rater effects have not been fully explored. The purpose of this study is to explore the practical consequences of rater effects on student achievement estimates in performance assessments. Practical consequences are considered as they relate to student achievement estimates and rank ordering. Simulated data were used to examine the effects of rater severity, centrality, and misfit. Results suggested that these rater effects affect estimates of student achievement in a substantial way that impacts the fairness of rater-mediated assessments.

Author

• Stefanie A. Wind, The University of Alabama - Tuscaloosa