A Critical Analysis of Bot Detection Methodologies

• In Event: Methodological Challenges to Studying Misinformation and Disinformation in Data-Driven Politics: Fake News, Bots, and Digital Campaigns

Abstract

In the past year, a growing amount of attention has been paid to the role that automation, and politically motivated automation, in particular, may be having on social media platforms. Communications scholars have recently joined computer scientists in becoming interested in the effects of ‘bots’— automated social media accounts — on platforms like Twitter. To detect bots, researchers have analyzed various aspects of the Twitter-based communication, from simple sender-receiver relationships to complex behavioral patterns, used a wide array of tools and methods including machine learning, network analysis, and linguistic approaches, and deployed honeypot accounts. Almost all studies relating to bots hinge on the accurate detection and classification of bot accounts; however, detecting bots with publically available data is a very difficult computational puzzle. In this paper, we provide the first comprehensive literature review of various bot detection methodologies that critically assesses the strengths and limitations of each method. We introduce some major puzzles for current approaches, including the “ground-truth problem” (the lack of reliable training data for machine learning algorithms), and the “cyborg problem” (the issue of hybrid accounts that produce a combination of automated messages and human curated content). We then suggest some avenues which could potentially be harnessed to yield more accurate detection in the future.

Authors

• Robert Gorwa, University of Oxford

• Bence Kollanyi, Oxford U

• Douglas Richard Guilbeault, The Annenberg School for Communication at the U of Pennsylvania

• Philip N Howard, Oxford Internet Institute, Oxford University