Beyond Reliability: Reviewing Rater Performance of Single Raters With Partial Back-Readings via Latent Class Signal Detection Theory and Bayesian Methods

• In Event: Roundtable Session 28
  In Roundtable Session: How Reliable Are Raters?

Abstract

In many assessment situations that use a constructed-response (CR) item, an examinee’s response is evaluated by one rater, and only some of the responses are reviewed by second raters: namely back-readings. The present study explores a Bayesian approach for the issues that arise in sparse rater designs within the context of a latent class version of signal detection theory (LC-SDT). This approach provides a model for rater cognition in CR scoring and offers measures of rater reliability and various rater effects. Simulation results showed that the Bayesian approach gave useful results; estimation of the parameters was moderate, except for small sample sizes. The paper also showed the utility of LC-SDT models in the PIRLS USA reliability data to review rater performance.

Authors

• Sunhee Kim, Columbia University

• Lawrence T. DeCarlo, Teachers College, Columbia University