A Mixed-Methods Study of Factors Affecting Rater Agreement During Teacher Observations

• In Event: 47.062 - Rating Quality Issues in Classroom Observation

Abstract

Inter-rater reliability is notoriously difficult to establish for observational protocols designed to measure complex teacher practices. The purpose of this sequential explanatory mixed methods study is to identify factors affecting rater agreement in teaching observations and to explore these factors in-depth. In Phase 1, quantitative data indicated that as the level of complexity of a pedagogical practice increased, rater agreement for the task decreased. In Phase 2, qualitative results indicated four primary sources of variance in ratings during paired observations: (1) evaluator interpretation of instrument scoring procedures, (2) evaluator interpretation of instrument language and content, (3) evaluator expertise in content area, research area, or grade band, and (4) evaluator prior personal experiences. Implications for evaluator selection and training are discussed.

Author

• Julie Brockman Smart, Clemson University