Investigating Human Essay Rating Quality in a Large-Scale Assessment Using Many-Facet Rasch Measurement

• In Event: 15.016 - Advancements in Human Scoring: Bringing Together Multiple Approaches

Abstract

The main purpose of the study is to evaluate the qualities of human essay ratings for a large-scale assessment using Rasch measurement theory. Specifically, Many-Facet Rasch Measurement (MFRM) was utilized to examine the rating scale category structure and provide important information about interpretations of ratings in the large-scale performance assessment setting. The preliminary results show that though the raters varied slightly in leniency or stringency, they were able to differentiate the three rating dimensions according to the rubrics and apply the rating scale category appropriately. The reliability indices resulted in sufficient examinee separation of 0.90 from the MFRM model. The results provide not only validity evidence for the essay ratings but also implications for quality assurance of rater training

Author

• Xiuyuan Zhang, The College Board