Applying Mokken Scaling Techniques to Incomplete Rating Designs for Educational Performance Assessments

• In Event: Evaluating, Optimizing, and Monitoring Rater Performance

Abstract

A major challenge in the widespread application of Mokken scale analysis (MSA) to educational performance assessments is the requirement of complete data, where every rater rates every student. I used simulated and real data to demonstrate a method by which researchers and practitioners can apply MSA to incomplete rating designs. Results suggested that when rating data are collected such that each rater rates test-takers in common with at least two other raters, it is possible to use MSA to evaluate rating quality in rater-mediated assessments with incomplete ratings. These results imply that it is possible to evaluate fundamental measurement properties in rater-mediated assessments using a nonparametric approach. Such analyses are important for evaluating the validity, reliability, and fairness of rater-mediated assessments.

Author

• Stefanie A. Wind, The University of Alabama - Tuscaloosa