Meta-Coding: Unpacking Undetected Disagreement Among Coders During Observational Analysis

• In Event: Interdisciplinary Techniques in Educational Assessments

Abstract

We developed a coding process, metacoding, to unpack coders’ agreements while coding classroom observations. In this paper, we describe our coding system devised from coders’ narratives – written notes that justify a coding decision. Our goal was to determine whether agreement among coders was factual (coders observed exactly the same events) or categorical (coders interpreted different events as belonging to the same categorical code). The coding system was applied to a sample of coders’ narratives collected as part of a formative assessment study. We found that the metacoding process revealed high percentages of hidden disagreement – instances where coders’ agreement was categorical but not factual. Our metacoding process may be especially helpful when training coders on a new set of codes.

Authors

• Paulina Biernacki, Stanford University

• Guillermo Solano-Flores, Stanford University

• Min Li, University of Washington

• Maria Araceli Ruiz-Primo, Stanford University

• Nixi Wang, University of Washington - Seattle

• Dongsheng Dong, University of Washington - Seattle