The Guidelines and Risks of Using Large Language Models for Coding in Social Science Research

• In Event: 50462 - Section on Science, Knowledge, and Technology Roundtables
  In Refereed Roundtable (60 minute): Table 3: LLMs, a SKAT perspective

Abstract

Large language models (LLMs) have revolutionized computational social science, particularly in automated textual analysis. In this paper, we conduct a systematic evaluation of the promises and risks of using LLMs for diverse coding tasks in social movement studies. We propose a framework for social scientists to adopt LLMs in text annotation, either as the primary coding decision-maker or as a coding assistant. Additionally, we discuss the associated epistemic risks related to validity, reliability, replicability, and transparency. We conclude by offering several practical guidelines for using LLMs in coding tasks.

Authors

• Hao Lin, Stony Brook University

• Yongjun Zhang, Stony Brook University