From Codebooks to Promptbooks: Extracting Information from Text with Generative Large Language Models

• In Event: 20268 - Culture and Computational Social Science (Co-sponsored by Section on Mathematical Sociology)

Abstract

Generative AI (GenAI) is quickly becoming a valuable tool for sociological research. Already, sociologists employ GenAI for tasks like classifying text and simulating human agents. We point to another major use case: the extraction of structured information from unstructured text. Information Extraction (IE) is an established branch of Natural Language Processing, but leveraging the affordances of this paradigm has thus far required familiarity with specialized models. GenAI changes this by allowing researchers to define their own IE tasks and execute them via targeted prompts. This article explores the potential of open-source large language models for IE by extracting and encoding biographical information (e.g., age, occupation, origin) from a corpus of newspaper obituaries. As we proceed, we discuss how sociologists can develop and evaluate prompt architectures for such tasks, turning codebooks into “promptbooks.” We also evaluate models of different sizes and prompting techniques. Our analysis showcases the potential of GenAI as a flexible and accessible tool for IE, while also underscoring risks like non-random error patterns that can bias downstream analyses.

Authors

• Oscar Stuhler, Northwestern University

• Etienne Ollion, Centre national de la recherche scientifique

• Cat Dang Ton