A Method to Assess Semantic Validity and Bias When Coding Open-ended Responses

• In Event: Advances in Machine Learning (I)

Abstract

Open-ended survey questions can provide researchers with nuanced and rich data, but content analysis is subject to misinterpretation and can introduce bias into subsequent analysis. We present a method to improve the semantic validity of a codebook and test for bias: a self-coding method where respondents first provide open-ended responses and then self-code those responses into categories. We demonstrate this method by comparing respondents’ self-coding to researcher-based coding using an established codebook. Our analysis shows significant disagreement between the codebook’s assigned categorizations of responses and the respondents’ self-codes. Moreover, this technique uncovers instances where researcher-based coding disproportionately misrepresents the views of certain demographic groups. We propose using this method to iteratively improve codebooks and, perhaps, to replace researcher-based content analysis.

Authors

• Rebecca A. Glazier, University of Arkansas, Little Rock

• Amber Ellen Boydstun, University of California, Davis

• Jessica T. Feezell, University of New Mexico