Using Neural Networks to Classify Based on Combined Text and Image Content

• In Event: Deep Learning in Political Science

Abstract

The use of social media data in political science is now commonplace. Recent work in election forensics uses Twitter data to capture
people's observations of incidents during the 2016 U.S. Presidential Election. Findings suggest that partisans report different types of
election incidents on Twitter and are involved in partisan communication silos. That work uses automated text-based classification, but
such tools do not use all available content -- in particular the machine classifiers ignore images. For instance, some Tweets feature
pictures of long lines or of smiling voters wearing "I voted" stickers. Human coders use such image information, but the machine
algorithms do not. In this paper we introduce machine-learning methods to automate the coding of combined text and image content. Layers
in a deep neural network combine and learn from features extracted from text and, where present, images. The method is used to examine
incident observations from the Twitter Election Observatory using data from both the 2016 Presidential and 2018 Midterm elections.

Authors

• Walter R. Mebane, University of Michigan, Ann Arbor

• Alejandro Pineda, University of Michigan