Monitoring Rating Quality in Writing Assessment Using Mokken Scale Analysis

• In Event: Roundtable Session 28
  In Roundtable Session: How Reliable Are Raters?

Abstract

A variety of indicators of rating quality have been developed based on parametric item response theory models (Engelhard, 2013; Myford & Wolfe, 2003, 2004). However, methods for monitoring rating quality based on rater-mediated assessments with nonparametric IRT (NIRT) have not been considered. Recognizing that parametric models may be unduly restrictive for ratings, this study explores Mokken’s (1971) approach to scale analysis as a method for monitoring rating quality. Because they are less restrictive, NIRT models may provide information that can inform the use and interpretation of ratings in educational settings. This study proposes a set of rating quality indicators based on Mokken scaling for polytomous data. Implications for research, theory, and practice are discussed.

Authors

• Stefanie Anne Wind, Georgia Institute of Technology

• George Engelhard, University of Georgia