Exploring the Effects of Rater Linking Designs and Rater Fit on Achievement Estimates Within the Context of Music Performance Assessments

• In Event: 72.066 - Poster Session 16
  In Poster Session: 72.066-2 - Poster Session: Practical Issues in Measurement

Abstract

This study explores the effects of linking patterns and model-data fit for raters on the ordering of student estimates in rater-mediated assessments. Linking and model-data fit were examined using a set of ratings of musical performances that was modified to represent three designs: Design One was the original data collection plan, where each performance was rated by three raters; Design Two included two ratings for each performance; Design Three only included the first rating for each performance. Using estimates of performance level and model-data fit for raters from the Many-Facet Rasch model, the relative ordering of performances was compared across designs. Results indicate that person performances cannot be adjusted for rater effects when inadequate linking or model-data fit is found.

Authors

• Stefanie Anne Wind, The University of Alabama - Tuscaloosa

• Brian Wesolowski, University of Georgia - Athens

• George Engelhard, University of Georgia