Facial Expression Recognition APIs for Political Science Usage: An Evaluation

• In Event: Image Processing for Political Research

Abstract

The measurement of emotions in facial expressions of politicians as part of their communication goes back to the 1980s. Until recently, the measurement had to be conducted by hand in a time consuming manor, clearly limiting the scope of studies. Machine learning approaches nurture the idea of automating the classification of facial expressions into the six basic emotions. APIs make the existing techniques accessible to non-computer scientists and thus, first papers are published in social sciences making use of these techniques. However, the APIs’ underlying models and training procedures generally remain undisclosed. The mechanism of the classification remains a black box. This study evaluates two of the most widely used facial expression recognition (FER) APIs: Face++ and Microsoft Azur Face API. I examine how well the APIs classify data from (1) images and (2) videos and discuss the conclusions implied for their usage in political science. In a first step, I explore the performance on the large standard datasets from computer vision research as a general benchmark. For images, I use the FER-2013 dataset and EmotiW-2018 for videos. In a second step, I specifically address a political science context. Based on data from parliamentary speeches, I compare the APIs output to two cheap student-coding methods, thereby setting a different benchmark of comparison. Furthermore, I evaluate the APIs’ performances on video material of acted political speeches specifically addressing the most studied emotions in political science: anger and fear. First results confirm accurate measurement of the general image dataset by the APIs. However, systematic biases evolve for video data of political speeches, e.g. “surprise” as an emotion is overestimated by the APIs, calling for a cautious use of these new techniques.

Author

• Theresa Küntzler, University of Konstanz