Republicans See Red, Democrats Feel Blue: Labeler Issues in Image Analysis

• In Event: Images as Data

Abstract

Image analysis studies, big or small, rely on human labelers to annotate materials. Human annotations inform researchers about what is in their raw materials (e.g. pictures of cats, dogs or protests) and also about reactions to those materials (e.g. evoked sadness, disgust). Human-generated labels are treated as ground truth for analysis. However, as we demonstrate in this paper, what labelers see and what their reactions are varies based on their personal characteristics. Republicans and Democrats, for example, have very different emotional reactions to our set of test images, which come from 1,841,886 tweets (and 947,005 attached images) relating to thirteen left-leaning social movements in the United States. Beyond emotional differences, partisans also on occasion disagree about which images contain protest. We show how an analysis of retweets using these data changes when different labelers are used. Our results emphasize that image analysis researchers must understand the differences in their labeler populations to draw accurate inferences.

Authors

• Nora Webb Williams, University of Illinois at Urbana-Champaign

• Andreu Casas, Vrije Universiteit Amsterdam

• Kevin Aslett, New York University

• John D. Wilkerson, University of Washington