Using LLMs to Track Methodological and Substantive Publication Patterns in Sociology

• In Event: 50462 - Section on Science, Knowledge, and Technology Roundtables
  In Refereed Roundtable (60 minute): Table 3: LLMs, a SKAT perspective

Abstract

How has sociology research evolved over the past decade, particularly in its methodological approaches, geographic scope, and research transparency practices? This paper will use a validated large language model approach to analyse publication patterns in leading sociology journals (ASR, AJS, and Social Forces) from 2010-2024 by measuring which methods, substantive topics, and open science practices are used in each article published in this period. We will track the use of 31 different methods (covering qualitative, quantitative, and theoretical approaches) as well as the subfield (using ASA sections), open science practices (replication packages and pre-registration), and indicators of selection on significance (reporting of null and non-null results). We use human codings to validate our LLM-based approach finding that the LLMs match the performance of trained graduate student researchers. What gets published in top sociology journals both reflects and creates prestige. Through this analysis we will uncover what sociology currently considers prestigious and how this understanding has evolved over time. We ultimately intend to expand our analysis to hundreds of sociology journals to fully characterize the publication patterns of the field as a whole and understand how the work published in the highest prestige outlets compares to the work being produced across the field.

Authors

• Jonathan Andrew Mellon, West Point

• Ryan Briggs, University of Guelph

• Vincent Arel-Bundock, University of Montreal