Search
Program Calendar
Browse By Day
Browse By Unit
Browse By Session Submission Type
Browse By Descriptors
Help
About Vancouver
Personal Schedule
Sign In
A defining feature of performance assessments is their reliance on the expert judgment of raters for the measurement of the quality of examinee responses. By virtue of their being human, raters unavoidably introduce error into the scoring process. While rater training helps attenuate this problem to some extent, it by no means completely eradicates the presence of rater errors from performance assessment data. Of the many different types of errors raters can commit, rater severity is perhaps the most serious and common. Defined as the tendency of a rater to assign higher or lower ratings, on average, than those assigned by other raters even after taking into account differences in examinee proficiency, rater severity has been studied extensively using a variety of approaches. Within item response theory, several different rater models are available for detecting and quantifying rater severity. Among these, Linacre’s (1989) many-faceted Rasch model (MFRM) is a widely-used model, whose formulation can incorporate multiple sources of variability (facets) that intervene in the measurement process (e.g., examinees, items, raters). The MFRM adjusts the estimates of examinee proficiency for the various parameters included in the model, and in particular, for differences in the level of severity raters displayed when scoring examinee performances, thus rendering fairer measures of proficiency.
Recent developments in the study of rater effects have led to the reformulation and extension of several rater models within the multilevel model framework (Dobria, Karabatsos, & Myford, 2007; Muckle & Karabatsos, 2009; Wang & Liu, 2007; Wang, Wilson, & Shih, 2006). The present study presents the multilevel facets model (MFM) for item response data scored by multiple raters. The MFM is a reformulation of the MFRM as a two-level model within the framework of the hierarchical generalized linear models. In the MFM, examinee proficiency is represented as a random intercept parameter, assumed to be normally distributed, while item difficulty and rater severity enter the model as fixed effects parameters. The outcome variable is most often ordinal, but the MFM also accommodates binary, counts, or continuous ratings. The MFM permits the inclusion in its formulation of various covariates of interest, thus lending itself to flexible extensions that facilitate investigations of differential item and rater functioning, and enable the study of rater effects such as rater severity drift. The specification of the MFM is accompanied by a listing of relevant model features and by a discussion of model limitations. To showcase the MFM and demonstrate its utility for detecting and accounting for the presence of rater severity in performance assessment data, a sample analysis of simulated essay rating data is included.