Evaluation of the Consistency of Results Derived From Topic Modeling and Human Coding

• In Event: Surveys in Action: From Technical Refinement to Real-World Influence

Abstract

The application of the topic modeling (TM) technique on open-ended survey questions has appeared in various education research, while a limited research base is available on the validity of theme identification with TM. This study aims to explore the alignment of the findings derived from manual coding and with TM analysis in open-ended survey data with a relatively small sample size. We found a strong alignment between the themes identified manually and topics generated by the TM, which provides empirical support for the validity for applying the technique to open-ended survey response analyses. Unmatched themes and the need for refinement in topic labels, however, underscore the critical role of human interpretations and domain knowledge in conducting qualitative analyses on open-ended responses.

Authors

• Xiyu Wang, Purdue University

• Yuxiao Zhang, Purdue University

• Yukiko Maeda, Purdue University

• Josiah Bansueda Banks, Austin Community College

• Rachael H. Kenney, Purdue University