Calibrating and Evaluating Automated Scoring Engines and Human Raters Over Time, Using Measurement Models

• In Event: Sunday Roundtable Session 3:05 pm
  In Roundtable Session: Advancements in Measurement Models and Evaluation Methods (Table 14)

Abstract

Automated scoring engines (ASE) have gained popularity in recent years. Researchers have focused on gathering evidence to support the use of ASE or its integration with human raters in scoring procedures. The purpose of this study is to explore the combination of ASE with human raters to detect changes in rater severity (rater drift) across multiple administrations. We used simulated data to explore how measurement models can be used to incorporate ASE into rater drift analyses. Results indicated that ASE can be efficiently integrated with human raters to detect rater drift using a concurrent calibration approach with measurement models. Our results also suggested that including ASE in the estimation procedure enhanced the accuracy of drift detection for human raters.

Authors

• Yangmeng Xu, University of Alabama

• Stefanie A. Wind, University of Alabama