Detecting Rater Effects in Writing Assessment: A Multilevel Modeling Approach

• In Event: 72.066 - Poster Session 16
  In Poster Session: 72.066-2 - Poster Session: Practical Issues in Measurement

Abstract

Rasch models and variations of these are routinely employed in the analysis of ratings and detecting rater effects in writing assessments. However, these models specify local independence of the multiple ratings assigned to a single student response, an assumption that is not reasonable in many applications. A viable alternative, easy to implement in standard statistical software, is the hierarchical generalized linear modeling (HGLM) framework. In addition to properly accounting for dependencies in rating data, multilevel analyses allow researchers to appropriately model various types of designs, leading to more accurate results and conclusions. This study uses data from a statewide writing assessment to demonstrate the usefulness of multilevel models as a viable alternative to Rasch modeling in detecting rater effects.

Authors

• Mihaela Ene, University of South Carolina

• Robert L. Johnson, University of South Carolina

• Edward W. Wolfe, Pearson