Interrater Reliability and Scoring Error in Targeted Double Scoring Procedure

• In Event: Evaluating, Optimizing, and Monitoring Rater Performance

Abstract

A portfolio assessment consisting of three extended tasks was scored with a targeted double scoring (TDS) procedure. All responses to each task were scored by rater 1; if the task score from the first rating fell in a “critical score range”, the response would be rated independently by a second rater. This study presents a method for estimating the interrater reliability coefficient and the standard error of scoring (SES) for the reported test scores from this scoring procedure. The report concludes with a numerical example showing how the estimates from this method compare to estimates from existing method and in different scoring procedures.

Authors

• Jing Miao, Educational Testing Service

• Yi Cao, Educational Testing Service

• Samuel A. Livingston, Educational Testing Service