Monotonicity as a Nonparametric Approach to Evaluating Rater Fit in Performance Assessments

• In Event: AERA Poster Session 10
  In Poster Session: Applications of the Rasch Measurement Model

Abstract

Rater fit analyses provide insight into the degree to which rater judgments correspond to expected properties, as defined within a measurement framework. Parametric models such as the Rasch model provide a useful framework for evaluating rating quality; however, these models are not appropriate for all assessment contexts. The purpose of this study is to explore numeric and graphical rater monotonicity analyses as a nonparametric approach to evaluating rater fit, and to consider whether these indices lead to similar decisions as Rasch fit indices in contexts with complete and incomplete data. Using simulated and real data, results indicate that nonparametric rater monotonicity indices lead to similar conclusions about rating quality as parametric fit statistics from the Rasch Rating Scale Model.

Author

• Stefanie A. Wind, The University of Alabama - Tuscaloosa