The Diagnostic Rating System: Rater Behavior for an Alternative Performance Assessment Rating Method

• In Event: Evaluating, Optimizing, and Monitoring Rater Performance

Abstract

To address common concerns with rubrics, a new system of performance assessment rating, the Diagnostic Rating System, has been proposed. A DRS is a logic-based rating system modeled on expert cognitive processes during evaluation of performance assessment. We applied a DRS to an established assessment of ethical reasoning abilities in a fully-crossed rating study of 12 raters and 30 student essays. Each rater rated each essay using a DRS and a traditional rubric. Many-facets Rasch modeling revealed raters 1) were less severe when using the rubric compared to the DRS rating method, 2) used the rating scale differently across the rating methods, and 3) had the largest rating method differences on the first two rating criteria.

Authors

• Allison Ames Boykin, University of Arkansas

• Nnamdi Chika Ezike, University of Arkansas at Fayetteville