Text To Data: Adapting Large Language Models for Public Records Research

• In Event: What Can We Learn from Large Language Models?

Abstract

Political scientists frequently rely on public records to study the inner workings of government, but the process of converting these unstructured documents into tidy datasets can be tremendously labor-intensive. Researchers often require teams of research assistants – or crowd workers recruited through sites like Amazon’s MechanicalTurk – to read through documents and summarize each one. Automating such a task would allow scholars to significantly expand the scope of their research, increasing the number of documents that could be coded for a fixed budget. Unfortunately, existing automated data extraction tools tend to be available only for texts that are structured in a predictable fashion.

In this paper, we develop new methods to extract data from unstructured texts using large language models (LLMs). These models are trained on a massive corpus of language data from the Internet and digitized books, and have transformed the way that computer scientists approach natural language processing over the past five years. To date, LLMs have made few inroads into the social sciences, but they have the potential to revolutionize the way we collect data, measure latent concepts, and work with large quantities of text.

When provided with an appropriately formatted prompt, large language models can be adapted to the task of converting unstructured text into structured datasets for statistical analysis. We demonstrate the validity and reliability of this method through a series of pre-registered replications, comparing the datasets parsed by these models against those created by careful human coders. We discuss best practices for working with these models and develop an open-source software package implementing the proposed methods.

Author

• Joseph T. Ornstein, University of Georgia