Co-Detecting Visuals and Texts: A Deep Learning Model for Image Classification

• In Event: Images as Data

Abstract

In this paper, we introduce a novel deep learning model of image classification that allows political scientists to not only extract visual representations of images but to exploit scene text—that is, text within visual images. Image classification techniques are now more often used in political science to analyze the political phenomenon, such as social protests and voting behavior. However, at the level of coarse-grained image classification, which political scientists currently use, we cannot fully extract valuable political information contained in visual data. For example, coarse-grained image classification does not allow us to exploit scene text, such as words that appear in placards held by protestors. Without such information, we can still detect the rise of social protests but cannot classify their type. Image classification models that allow researchers to pursue fine-grained image classification can extract valuable information from visual data including scene text for more elaborate research. To show the models’ validity and advantage, in this paper, we modified a deep-learning-based image classification model and applied it to research on contentious politics. Using actual visual data, we show how our model can utilize information obtained from scene text to 1) better detect social protests; and 2) classify types of protest in visual data.

Our model combines three sub-models. The first is a Visual Encoder (VE) model that generates visual representation from images. The second is a Visual-Enhanced Scene Text Recognition (VESTR) model that we developed from a Scene Text Recognition (STR) Model. The VESTR model directly recognizes scene text within visual images and, at the same time, recognizes words and sentences from visual-enhanced textual representation by the VE model. And the third is a Textual and Visual Image Classification (TVIC) model that classifies images into fine-grained categories by combining visual and textual representations obtained from the VE and VESTR models. The combination of these three models allows us to recognize hand-written, distorted, and irregular texts that often appear at social protests and during electoral campaigns. Our model can extract valuable information from such scene texts and use it to classify types of political activity. We thus regard our model as a visual-enhanced scene recognition model for fine-grained image classification.

We applied our model to actual visual data to test its validity. The dataset we used was the UCLA protest image dataset containing 1,911 images with placards showing hand-written, distorted, and irregular texts. We assigned two different image classification tasks to our model: protest detection and protest demand detection. The latter detects the types of social demands (e.g., racial or gender equality) being made at each protest. Compared with the purely visual representation model proposed in the UCLA paper, our model outperformed the F-score evaluation for the protest detection task. We believe that information extracted from scene text led to better detection. As for the second task, a benchmark model is not available for validation because this is an initial attempt. Nevertheless, our experiment shows that our model succeeded in classifying the demands of each protest with more than 75% accuracy.

In this paper, we focused on detecting and classifying social protests through scene text in order to test the validity of our model. However, our model can be applied to visual data for fine-grained image classification in a variety of political situations. We believe our model displayed promising results for more elaborate research of visual data in political science.

Authors

• Xuan Luo, The University of Tokyo

• Sota Kato, International University of Japan

• Budrul Ahsan, Philips Japan

• Takafumi Nakanishi, Musashino University