Rater Certification Standards

• In Event: 32.082 - Roundtable Session 11
  In Roundtable Session: 32.082-13 - Maximizing Knowledge: Utilizing the Interplay of Rigor, Evidence, Culture, and Achievement to Inform Systems of High-Stakes Accountability

Abstract

A key part of any validation argument for observation instruments relies on raters accurately and reliably applying the instrument. Most instruments rely on a process of rater certification as a gate-keeping control to ensure high quality ratings. Rater certification effectively sets an upper bound on acceptable levels of rater error, ensuring raters at least start scoring below this bound. This bound, however, has never been rigorously tested and explored. I argue for the need for standard setting procedures to rethink certification test passing standards. I show that for many instruments, certification tests are arguably too easy and discuss ways of rigorously setting new certification standards.

Author

• Mark White, University of Michigan - Ann Arbor