Quantifying Narrative Diffusion Across Languages

• In Event: 20858 - Reimagining How We Measure and Model the Social World: Causation, Flows, and Artificial Intelligence

Abstract

How can one trace the diffusion of ideas across texts? This is a key measurement problem in sociological inquiry, from the study of how interest groups shape media discourse, to the spread of policy across institutions, to the diffusion of organizational structures and institution themselves. To measure diffusion, we seek to identify texts that make the same set of specific claims about the same events, a phenomenon we call "narrative commonality." We use large language models to distill texts to their core ideas and then compare them at the level of the claim instead of words, phrases, or sentences. The result is an estimand much closer to "narrative commonality" than is possible with exact text matching, which returns lexically similar documents; topic modeling, which returns topically similar documents; or an array of alternative approaches. What's more, descriptions of these approaches generally do not provide out-of-sample measures of performance (precision, recall, F1); we devise an approach to measure performance and show that our approach outperforms past approaches by a large margin. We apply our approach to an important case study: the spread of Russian claims about the development of a Ukrainian bioweapons program in U.S. mainstream and fringe news websites. While we focus on news in this application, our approach can be applied more broadly to the study of propaganda, misinformation, diffusion of policy and cultural objects, amongst other topics.

Authors

• Hannah Waight, University of Oregon

• Solomon Messing

• Anton Shirikov

• Margaret Roberts

• Jonathan Nagler

• Jason Greenfield

• Megan Brown

• Kevin Aslett

• Joshua Tucker