- Browse
Search
Program Calendar
Browse By Day
Browse By Time
Browse By Person
Browse By Unit
Browse By Session Type
- Information Menu
Search Tips
Annual Meeting Housing and Travel
- Navigation and Settings Menu
Personal Schedule
Sign In
- Social Media Menu
Facebook
X (Twitter)
Evaluating, Optimizing, and Monitoring Rater Performance
Mon, April 20, 8:15 to 9:45am, Virtual RoomSession Type: Paper Session
Abstract
Optimizing the performance of human raters is a fundamental measurement concern. The papers in this session address new approaches to identify, quantify, and monitor rating quality. Topics in this session include the development of a new Rater Quality Index; a non-parametric approach to exploring measurement error associated with raters; identifying what sources of construct irrelevant variance influence raters; how raters’ scores vary depending on the scoring approach; and the performance of a double scoring procedure.
Sub Unit
Chair
Papers
Applying Mokken Scaling Techniques to Incomplete Rating Designs for Educational Performance Assessments - Stefanie A. Wind, The University of Alabama - Tuscaloosa
Interrater Reliability and Scoring Error in Targeted Double Scoring Procedure - Jing Miao, Educational Testing Service; Yi Cao, Educational Testing Service; Samuel A. Livingston, Educational Testing Service
The Diagnostic Rating System: Rater Behavior for an Alternative Performance Assessment Rating Method - Allison Ames Boykin, University of Arkansas; Nnamdi Chika Ezike, University of Arkansas at Fayetteville
An Approach to Investigating Construct-Irrelevant Variance for Contextualized Constructed-Response Assessment - Xiaoming Zhai, University of Georgia; Kevin Haudek, Michigan State University; Molly A.M. Stuhlsatz, BSCS Science Learning; Christopher D. Wilson, Biological Sciences Curriculum Study