Modeling Rater Effects in the Presence of Data Dependency: Recent Advancements in Hierarchical Rater Models

Session Submission Type: Symposium

Abstract

Rater effects have been widely recognized as a major threat to objective measurement in performance assessment. In recent years, hierarchical rater models that recognize the dependency of multiple ratings on the same ratee began to emerge in the literature. While many studies found compelling evidence for the need to model data dependency, further research is called for to compare model performance under different conditions and pragmatic constraints. This symposium session brings together leading scholars to engage in a critical dialogue on the latest developments in rater modeling research. The robustness, power, and software efficiency of different rater models are compared and practical guidelines for when to make the switch from nonhierarchical to hierarchical rater models are suggested.

Sub Unit

• Division D - Measurement and Research Methodologies / Section 1: Educational Measurement, Psychometrics, and Assessment

Chair

• Lihshing Leigh Wang, University of Cincinnati

Papers

• On the Multilevel Facets Model for the Analysis of Rating Data - Lidia Dobria, Wilbur Wright College

• Item Response Modeling of Inter- and Intra-Rater Variations in Severity and Local Dependence - Wen-Chung Wang, The Hong Kong Institute of Education

• Some Notes on a Hierarchical Rater Model for Constructed Responses - Lawrence T. DeCarlo, Teachers College, Columbia University; YoungKoung Kim, The College Board

• Cross-Classified Random-Effects Models for Assessing Rater Severity and Differential Rater Functioning - Susan Natasha Beretvas, The University of Texas - Austin

Discussants

• Edward W. Wolfe, Pearson

• Richard J. Patz, CTB/McGraw-Hill LLC