Unsupervised and Interpretable Feature Extraction for Political Image Data

• In Event: Image Processing for Political Research

Abstract

The development of a general methodology for the analysis of image data is an important avenue for applied political science research. This is because image data from political and social contexts of theoretical interest is increasingly more available, but there is still a lack of a general-purpose methodology to integrate such image data with conventional quantitative analyses: methods have to be tailored to each specific application, often by hand-labelling large amounts of training data, which is a costly and lengthy process.

We introduce an unsupervised methodology to extract real-valued features from sets of images that are easily interpretable. The features produced with our method can be employed in traditional quantitative analyses such as regression. The methodology is fully unsupervised and does not require labelled training images. Examples of features that can be extracted include the presence/absence of certain sub-images, as well as pure graphical elements like color, position or scale.

Our approach is based on deep learning and specifically tailored to social science image data. Image sets analyzed in the social sciences often contain images that are graphically different but contain similar concepts. For example, social media posts from a political party will often be quite diverse in terms of graphic content, but will likely all contain similar underlying concepts. Current interpretable deep learning methods are unsuited for this setting as they are geared towards isolating differences from sets of otherwise graphically similar images. We introduce an interpretable variational autoencoder that automatically also clusters similar images together, and extracts different features for every cluster of images. Optional auxiliary image-level covariates can also be used with our methodology.

We assess the performance of our methodology on standard datasets and on socio-political data, showing that our method performs at least as well as other state of the art approaches, while being more interpretable.

Author

• Marco Morucci, New York University