Labeling Online Disinformation: Experts vs. Crowdworkers

• In Event: Misinformation and Disinformation Online

Abstract

Existing empirical literature on disinformation in political science relies mainly on the datasets released through governmental or media investigations. Many of them focus on just one (albeit important) distribution channel: social networks, as postings there conveniently encapsulate information for structural and dynamic comparisons; others quickly become outdated or are impossible to replicate as article-level factchecking is costly and slow. Most importantly, both the sampling and classification into various categories of disinformation are often ad-hoc, a black box for researchers, and is not checked for robustness. To create a rigorous methodology for detecting disinformation online and to build a comprehensive dataset of U. S. disinformation websites, we use both information verification experts and crowdworkers to evaluate websites’ credibility based on a rich set of features. In this paper, we utilize both expert and crowd judgments to create novel empirical classifications of types, methods, and targets of disinformation; to establish which features (e.g., the degree of social media integration, clickbait headlines, or low-quality ads) differentiate disinformation from credible sources; and to evaluate if crowdworkers could judge disinformation as well as (or better than) experts. We also use crowdworkers as a proxy for the general public to assess if its ability to detect disinformation is under- or over-estimated by experts.

Authors

• Sergey Sanovich, Princeton University

• Andy Guess, Princeton University

• Benjamin Kaiser, Princeton University

• Jonathan Mayer, Princeton University