Interrater Reliability of Observed Behaviors: Comparing Kappa, Correlation Coefficient, Intraclass Correlation Coefficient, and Generalizability Theory

• In Event: 60.031 - Reliability: New Approaches

Abstract

In education and social science research, when the observed behaviors are scored by different raters, the interrater reliability among the subjective ratings is a critical issue. This study reviews major estimation approaches for interrater reliability reported in educational psychology journals in 2007-2012, showing that intraclass correlation coefficient (ICC) is the most frequently (44.78%) used approach, while generalizability (G) theory is the least used one (2.99%) in current research practices. This paper clarifies and compares different approaches by an empirical example, and develops SPSS and SAS syntax. We conclude that G theory is recommended to evaluate interrater reliability, which enables researcher to obtain adequate interrater reliability by increasing the number of items occasions or rater.

Author

• Qian Cao, Texas A&M University - College Station