Understanding Interrater Reliability in a Diverse Group of Raters: Considerations for the Frame of Reference Training

• In Event: 47.062 - Rating Quality Issues in Classroom Observation

Abstract

This autoethnographic study explores the systematic variance due to common response bias among raters by examining their individual value systems and their perceptions of value-laden constructs of the observational rubric used to assess teaching and learning processes in college classrooms. We find no significant difference between the raters’ individual values and their scoring of the value-laden rubric items. However, we find that the discrepancy among raters across the board is more likely to decrease rather than increase possibly due to the frame of reference training (FOR) provided to the raters during the certification process. This study fills in the gap in the literature on personal characteristics of raters and IRR and has direct implications for FOR training.

Authors

• Abbas Abbasov, Teachers College, Columbia University

• Corbin M. Campbell, American University