Simulating Partisan Media Diets with Large Language Models

• In Event: Applications in Political Methodology Using Large Language Models

Abstract

Can Large Language Models (LLMs) learn partisan media bias from ideologically slanted news sources? The rapidly developing sophistication of LLMs in natural language processing has raised questions about these models’ tendencies to inherit latent biases present in their training corpora. In this study, we conduct a series of bias experiments on several popular LLMs. We prepare “ideological” models by fine-tuning on articles from partisan news sources, with the names of notable political figures replaced with generic signifiers. The partisan biases of these fine-tuned models are then tested in two ways. First, models are prompted to evaluate the favorability of political figures mentioned in the training corpus articles in order to determine what these models have “learned” from this fine tuning. Second, models are provided with previously unseen articles from both ideologically congruent and counter-attitudinal news sources and are prompted to evaluate the reliability of these sources. Evidence of learned partisan bias will be apparent if these ideological language models are more likely to rate co-partisan political figures as favorable and to rate ideologically congruent sources as more trustworthy. A range of models are compared in this analysis, from some of the earlier widespread Transformer models such as BERT, to more recent, open-source models like GPT-NeoX as well as commercial models such as ChatGPT. This study has important implications for how LLMs can be used as tools in political science research, and suggests real differences in how different news sources cover politics.

Authors

• Galen Stocking, Pew Research Center

• Sono Shah, Pew Research Center

• Samuel Bestvater, Pew Research Center