Zero-Shot Issue Position Estimation Using Generative Large Language Models

• In Event: What Can We Learn from Large Language Models?

Abstract

Many methods exist to estimate legislators' and parties' ideal points (e.g., Poole and Rosenthal, 1985; Clinton, Jackman, and Rivers, 2004) and policy positions (e.g., Laver and Garry, 2000; Slapin and Proksch, 2008; Laver, 2014). Most methods rely on either (or both) roll call data and/or text of news or legislative speeches (Lauderdale and Herzog, 2016; Kim, Londregan, and Ratkovic, 2018). These methods require gathering the necessary roll call data and appropriate corpora of text. Data is further reduced when one wishes to scale legislators along a specific issue dimension.

I propose a simple method of using a generative large language model--specifically, GPT-3 (Brown et al., 2020)--to scale legislators along a single issue or policy dimension. GPT-3 is pretrained on multiple massive corpora, such as the Common Crawl, Wikipedia, webtexts, etc. GPT-3 requires an initial prompt to generate the text that follows it. This allows users to pose questions and have the generative language model answer them. In these prompts, I name two legislators and ask for the relative issue positions of both legislators (for example, who is more likely to sponsor gun control legislation?). I record the language model's generated response. I repeat this exercise with pairs of legislators. Because the generated output differs from run-to-run, I also repeat comparisons between legislators. After repeated iterations of the above, I then scale the legislators using an Elo rating system. This approach does not require additional text data or roll call data collection.

I validate this approach using the following issues in the context of the U.S.: gun rights, reproductive rights, animal rights, campaign finance, and education issues. I scale legislators from the 114th, 115th, and 116th Congresses. I compare these issue position estimations with grades assigned by various interest groups, such as the National Rifle Association and NARAL Pro-Choice America. I also compare issue position estimations with alternative scaling approaches. I find that GPT-3 can reason about issue positions in a nuanced fashion. For example, using a sample of senators from the 116th Congress with the above methodology, the Elo rating system positions Doug Jones to the right of Dianne Feinstein and Kamala Harris on the issue of guns. I also find that prompting GPT-3 using pairs of legislators and scaling these outcomes with the Elo rating system yields a more consistent scale than prompting GPT-3 to rank a list of legislators directly.

Author

• Patrick Y. Wu, American University