Decoding Politicians’ Emotions on Facebook and Instagram with Computer Vision

• In Event: Understanding Emotions

Abstract

Despite strong public interest in the electoral implications of social media, little remains known about the emotional valence of politicians’ posts on these platforms. This lack of knowledge is problematic, since prior research suggests that the emotional cues of political messages can influence citizen mobilization, partisan-motivated reasoning, and information-seeking (Brader, 2005; Marcus, Neuman, & MacEuen, 2000). Moreover, as social media platforms trend towards prioritizing images and video, there is a pressing need to understand how digitally-mediated forms of visual media affect voters. Visual media are particularly persuasive in a political context because they convey non-verbal behavior such as facial expressions and emotional displays, which humans prioritize when processing information (Grabe & Bucy, 2009).

In this paper, we reveal the emotions that politicians convey through their facial expressions in images posted on Facebook and Instagram. Using the 2020 primary elections as a case, we collect images posted on the two platforms from Jan 1, 2019 – March 31, 2020 (n=8,183) for eight candidates (Biden, Bloomberg, Buttigieg, Klobuchar, Sanders, Steyer, Trump, Warren). Then, we classify facial expressions in these images by emotion using Amazon’s Rekognition API, a computer vision tool not yet applied in political research. In addition to uncovering the emotions that politicians express, we also test if these emotions vary by platform and whether certain emotional appeals translate into high performing posts. Guiding our analysis are the following research questions:

RQ1: What emotions do candidates display in images from their Facebook and Instagram accounts?

RQ2: Do these emotions differ by platform or according to different stages in an election campaign?

RQ3: Does emotional self-presentation help explain high performing social media posts?

Theory and Hypotheses
Our theoretical approach is guided by affective intelligence frameworks in political psychology. Affective Intelligence Theory (AIT) posits that emotional responses are generated by two systems – the disposition system and surveillance system – that exhibit differential effects on voters (Marcus, Neuman, & MacEuen, 2000). The disposition system refers to emotions that relate to enthusiasm (happy-sad) and aversion (anger-disgust). The surveillance system, meanwhile, refers to behavior relating to threats and comprises the emotional dimension of anxiety (calm-fear). Extant evidence suggests that disposition system emotions drive mobilization and reify partisan preferences. By contrast, emotions in the surveillance system induce demobilization but encourage new information-seeking. The AIT framework has largely confirmed by a number of studies analyzing political ads in experimental designs (Brader, 2005), studies combining surveys and experiments (Jones, Hoffman, & Young, 2012; Weber 2012), and agent-based modelling (Scheller, 2019). Yet, no study has applied this framework to chart the emotions that politicians actually convey in their messaging.

Following AIT theory, we expect that candidates will seek to communicate disposition system emotions in their images on Facebook and Instagram when encouraging supporter mobilization (e.g. calls for action) (Hypothesis 1). Conversely, we expect candidates to post images conveying surveillance system emotions when attacking opponents, since these messages aim to persuade (Hypothesis 2). In terms of post performance, we hypothesize that disposition (i.e., mobilizing) emotions will generate more online engagement than surveillance (i.e., demobilizing) emotions.

Method
To collect and classify images by emotion, we deploy two open science software tools that we developed. The first, Facebook Toolbox, harvests images from links collected via Crowdtangle. The second, Pykognition, is a wrapper for Amazon Rekognition API and detects faces and emotions in images. The emotion categories, which lend to a smooth operationalization of AIT theory, are: Happy, Sad, Angry, Fear, Disgust, Calm Surprised, and Confused. To validate this tool, we crowd code 3,000 images with 3 MTurkers each to gauge whether algorithmically-derived classifications align with human-labelled annotations. Once validated, we then descriptively report our findings and utilize OLS regressions to reveal whether disposition emotions correlate with high performing posts, as measured by reactions data (likes and favorites).

Our findings will contribute theoretically to political communication research by providing the first study of emotions conveyed in the facial expressions of politicians across social media platforms. Methodologically, we contribute by providing two open science tools for future research, as well as provide an assessment of whether Amazon’s computer vision technology is accurate enough for political communication studies.

Authors

• Michael Bossetta, Lund University, Sweden

• Rasmus Schmøkel