Transformer-Assisted Conceptual Coding: Training and Validating AI Using Hundreds of Interviews of Scientists Worldwide

• In Event: 20458 - Qualitative Methodology

Abstract

We introduce a framework for using transformer-based models to conceptually code interview transcript data, demonstrating how artificial intelligence can assist, rather than replace, human interpretation in qualitative research. Our study emerges from a large-scale project in which we conducted 231 semi-structured interviews with scientists across more than 50 countries, exploring how globalization and politics shape their careers and perspectives. Building on computational grounded theory and recent advances in natural language processing, we trained a transformer model to apply thematic codes to interview transcripts. Our coding scheme, developed through an iterative human-led process, was the foundation for training and validating the model. We evaluated the model’s reliability by comparing its performance against human-coded transcripts, finding that it achieved an F1 score of 0.867—indicating strong alignment with our team of coders. Beyond accuracy, we examine the broader implications of integrating transformer models into qualitative research. While transformers offer significant efficiency gains in coding large interview datasets, they also raise critical questions about the interpretive nature of qualitative analysis. What is gained—and lost—when human researchers no longer manually code every text? How does automation shape the epistemological foundations of qualitative inquiry? Our framework addresses these questions, proposing a hybrid approach that retains sociology's interpretive strengths while leveraging computational tools for large-scale analysis. By systematically reflecting on both the technical and methodological aspects of transformer-assisted coding, we aim to provide sociologists with a practical guide for incorporating AI into qualitative research. Our findings suggest that, when carefully implemented, transformers can be powerful complements to traditional interpretive methods, facilitating new forms of collaboration between qualitative and computational researchers.

Authors

• Charles Jonathan Gomez, University of Arizona

• Peter Ore, CUNY-Queens College

• Lilith Green, University of Arizona

• Elina I. Mäkinen, Tampere University

• Charlie Pollard, University of Arizona

• Jeffrey Shen, University of Arizona