A Hierarchical Facets Model for the Joint Analysis of Ratings and Rating Times

• In Event: Applications in Psychometric Models

Abstract

Performance assessments increasingly utilize onscreen or internet-based technology to collect human ratings. Considering rating times automatically recorded along with ratings can provide a more detailed picture of the rating process and improve the psychometric quality of the assessment outcomes. Thus, we propose a joint modeling approach, the “hierarchical facets model for ratings and rating times” (HFM-RT), which includes two examinee parameters (ability and time intensity) and three rater parameters (severity, centrality, and speed). A real-data analysis of English essay ratings revealed that raters exhibited considerably different speed measures, spent more time on high-quality than low-quality essays, and graded essays faster with increasing severity.

Authors

• Kuan-Yu Jin, Hong Kong Examinations and Assessment Authority

• Thomas Eckes, University of Bochum