Paper Summary
Share...

Direct link:

Evaluating Rater Accuracy With a Hyperbolic Cosine Unfolding Model

Sun, April 19, 8:15 to 9:45am, Hyatt, Floor: West Tower - Gold Level, Hong Kong

Abstract

Theoretical Framework: As the number of rater-mediated assessments continues to increase, it is important to explore new methods for evaluating the quality of ratings obtained from raters. This study explores a new family of rater quality indices based on unfolding models. Andrich and his colleagues (Andrich, 1988; Andrich & Luo, 1993) developed several unfolding models, which quantity the distance between person and item locations on a continuum. Rater accuracy can also be objectively evaluated as a distance between observed ratings from an operational rater and true ratings defined by expert raters. Engelhard (1996, 2013) proposed a Rasch model for measuring rater accuracy defined as the absolute values of the distances between the ratings of operational and expert raters. Wolfe, Jiao, and Song (2014) demonstrated that this model does not differentiate rater severity from inaccuracy. Unfolding models offer the potential to model accuracy ratings that estimate directionality in the distances between observed and true ratings.

Objectives: The purposes of this study are to (a) modify and interpret the Hyperbolic Cosine Model for evaluating the accuracy of ratings obtained from rater-mediated assessments, and (b) evaluate the correspondence between three different models of rating quality, the Rasch Partial Credit Model (RPCM), Rasch Rater Accuracy Model (RRAM), and the Hyperbolic Cosine Model (HCM).

Methods: In a preliminary simulation, we generated ratings of 20 raters with four types of rater effect statuses (14 none, 2 severe, 2 central, and 2 inaccurate) for 5000 persons. We then estimated parameters for the RPCM, RRAM, and HCM. Raters were flagged based for evidencing rater effects based on these estimates, and correct/incorrect classifications constituted the outcome variable. These preliminary results are summarized here. A full simulation in which we manipulate rater effect type, magnitude, and prevalence (% of raters exhibiting effect) will be completed prior to that AERA conference.

Preliminary Results: Results from the RPCM are consistent with previous applications of that model to the detection of rater effects. Specifically, rater location parameters were inflated for the severe raters, rater threshold parameters were tightly clustered for the central raters, and rating-ability estimate correlations were lower for the inaccurate raters. Results from the RRAM allowed us to identify rater severity and inaccuracy, but those estimates were not sensitive to rater centrality. HCM parameter estimates allowed us to identify all three types of rater effects, and we are conducting additional analyses to determine whether it is possible to differentiate the three rater effects using the HCM.

Significance: In summary, we do not know of any studies that have evaluated rater accuracy with unfolding models. Accuracy ratings can be regarded as unfolding data rather than cumulative data (Coombs, 1964). The raters are expected to be accurate with observed ratings close to those obtained from an expert rater. Unfolding models hold promise as a new family of indicators for evaluating rater accuracy.

Authors