Detecting Political Sectarianism on Twitter Using Generative Language Models

• In Event: Better Models, Better Predictions

Abstract

This project studies how political scientists can use generative language models, such as GPT-3, to detect political sectarianism (Finkel et al., 2020) on social media. We show how different approaches to prompting GPT-3 (i.e., the questions one poses to GPT-3) yield stark differences in model performance. We compare results using GPT-3 to a RoBERTa model fine-tuned using 10,000 tweets. We find that using GPT-3 with in-context few-shot learning can be a strong alternative to fine-tuning RoBERTa while requiring much less hand-labeled data. This work makes two contributions. First, our work provides a new approach to detecting political sectarianism on social media. Second, we examine different approaches to prompting GPT-3, and show which approaches work best.

Large generative (autoregressive) decoder language models have been proposed as an alternative to fine-tuning large language models for automated classification tasks. Popular examples include GPT-3 (Brown et al., 2020) and OPT (Zhang et al., 2022). Compared to bidirectional encoder language models, such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), large generative language models are pretrained on much larger bodies of text, contain more parameters, and often do not require fine-tuning to use effectively. Without fine-tuning, a much smaller number of hand-labeled examples are needed and much fewer computational resources are required.

The current literature in political methodology has primarily focused on how generative language models can be applied to a wide range of text-as-data tasks in political science (e.g., Ornstein, Blasingame, and Truscott, 2022). There are three areas for improvement in the current literature. First, these works only look at in-context zero-shot, one-shot, and two-shot learning (providing hand-labeled training examples in the prompt), but do not look at how chain-of-thought, which refers to prompting GPT-3 to provide a set of intermediate reasoning steps that lead up to an answer (Wei et al., 2022; Kojima et al., 2022), can improve outcomes. Second, these works do not compare how GPT-3 performs against fine-tuning a bidirectional encoder transformer model such as BERT or RoBERTa. Third, most applications to social media focus only on the sentiment analysis of tweets.

We address these gaps by comparing how the model performs under in-context few-shot and chain-of-thought settings, how GPT-3 compares to a fine-tuned bidirectional encoder language model, and how GPT-3 performs on a task much more conceptually difficult than binary sentiment classification. Political sectarianism (Finkel et al., 2020) is defined as a moralized identification with one political group against another. It goes beyond affective polarization, which involves distrust of others based on their out-party membership. We proceed in two steps. First, we identify whether the tweet contains aversion against Democrats or aversion against Republicans. If the tweet contains aversion, we further identify whether the tweet contains moralization or othering.

Initial results over a test set of tweets from 2020 show that including just one hand-labeled example in the prompt can dramatically improve results; including a few examples begins to approach the performance of a RoBERTa model fine-tuned on 10,000 hand-labeled tweets that came from 2020 (Chen et al., 2022). The fine-tuned RoBERTa model has an F1 score of 0.8129 for aversion against Democrats and an F1 score of 0.7140 for aversion against Republicans. Using GPT-3 with zero in-context examples, we obtain an F1 score of 0.5143 for aversion against Democrats and an F1 score of 0.5798 for aversion against Republicans. With one in-context example, we obtain an F1 score of 0.7292 for aversion against Democrats and an F1 score of 0.6252 for aversion against Republicans. With two in-context examples, we obtain an F1 score of 0.7381 for aversion against Democrats and an F1 score of 0.6557 for aversion against Republicans. With three in-context examples, we obtain an F1 score of 0.7282 for aversion against Democrats and an F1 score of 0.6747 for aversion against Republicans. With only two to three in-context examples, GPT-3 approaches the performance of the fine-tuned RoBERTa model.

The paper will first describe how performance changes when providing an increasing number of in-context training examples. We then build on these improvements by including chain-of-thought prompting, which will improve results and force the model to explain its reasoning, improving model interpretability. Finally, we examine how ensembling classification predictions across various prompts can further improve the performance of detecting political sectarianism on social media.

Authors

• Patrick Y. Wu, American University

• Haohan Chen, The University of Hong Kong

• Zhanna Terechshenko, New York University

• Richard Bonneau, NYU

• Jonathan Nagler, New York University

• Joshua A. Tucker, New York University