Lambda Coefficient of Rater-Mediated Agreement: Demonstration of a Chance-Corrected Agreement Coefficient (Poster 5)

• In Event: AERA Poster Session 8
  In Poster Session: Assessment and Learning: Review and Applications (San Diego)

Abstract

In this study, the performance of the Lambda Coefficient of Rater-Mediated Agreement was compared with other chance-corrected agreement coefficients. Lambda corrects for chance agreement based on specific assumptions about raters through the application of rater mediated assessment theory, which includes rater tendencies towards strict or lenient ratings. Coefficients were calculated using simulated data by varying conditions across two-, three-, and four-point rating scales. The exact agreement ranged from 50% to 95% across simulations. The simulations adjusted prevalence and bias indices according to exact agreement specifications (Xie, 2013). Simulation results demonstrated the robustness of Lambda to data conditions that are problematic for other coefficients. Lambda produced benchmark agreement results that maintained meaning that may be diminished by other coefficients.

Authors

• T. Scott Holcomb, Duke University

• Richard G. Lambert, University of North Carolina - Charlotte

• Bryndle Bottoms, University of South Carolina