Large Language Models Can Argue Convincingly about Politics: Experiment Evidence

• In Event: What Can We Learn from Large Language Models?

Abstract

Rhetoric and debate are considered vital civic arts, and central to the success of professional politicians. We ask whether and how the rise of artificial intelligence algorithms for text production might affect this purported "unique" human capacity for argumentation. In particular, we study the ability of Large Language Models (LLMs)---that can generate realistic human-like text---to produce convincing rhetorical arguments in response to political prompts. We show that, at their most persuasive, such LLMs can routinely outperform humans in rendering convincing cases for particular positions on several contemporary debates ---including on abortion, immigration and gun control. LLMs are also capable of providing coherent responses to open-ended requests for "new" areas, implying some potential issue entrepreneurship. But we also demonstrate that LLMs often fail to produce anything coherent, and their "average" argument quality is less than that of human coders. In contrast to previous approaches, we provide large sample estimates (and bounds) on how often one should expect high and low quality responses---and we characterize their nature. Finally, we provide experimental results on the susceptibility (or not) of humans to information about the origins of arguments. We comment on the potential ethical consequences of such machine intelligence in politics.

Authors

• Arthur Spirling, Princeton University

• Alexis Palmer, Dartmouth College