A Simple and Instructive Approach to Examining Interrater Reliability

• In Event: AERA Poster Session 12
  In Poster Session: Issues in Contemporary Program Evaluation in Schools

Abstract

This paper describes an application of Rasch measurement to rubric-based observational data, arguing why a measurement approach provides a meaningful framework for analyzing inter-rater reliability in a rater training situation. A two-facet Rasch model was applied to observation data during a rater training session and formative feedback provided on ratings that were inconsistent with their level of rater severity. The output from the analysis allowed the trainers to easily identify areas of misunderstanding to both guide the training and to provide feedback to instrument developers on instrument clarity. This study serves as a model for reliability analyses useful for inter-rater training for a large number of observers or graders learning to use a scoring protocol.

Authors

• Shannon O. Sampson, University of Kentucky

• Susan Cantrell, University of Kentucky