Images with Texts: Multimodal Framing Analysis via Hypergraph Learning

• In Event: Images as Data

Abstract

Big data revolution has created unprecedented research opportunities for political scientists to delve deeply into large amounts of multimedia objects. Compared with the rapid growth of unstructured data like images and texts, quantitative approaches that deal with the volume, variety, and complexity of such data have not been introduced sufficiently in political science. A growing number of scholars realize the need to develop a data-driven method to combine various types of unstructured data in a unified model and investigate complex relationships between them. To address such challenging issues and search for new methods, this paper introduces a hypergraph combined with big data techniques and learning algorithms as a reliant method to discover multimodal patterns in narratives. Unlike conventional graph structures that capture exclusively pairwise relationships, a hypergraph constructs rich patterns of high-order relationships to better convey the underneath geometrical structure in various types of data. This paper shows two applications of a hypergraph learning algorithm for multimodal framing analysis of news articles. First, I develop a multimodal topic model via hypergraph learning that fuses the visual, textual, and other multimedia features simultaneously to find the latent topic representation in media coverage during the European refugee crisis and Black Live Matter protest. Despite the increasing significance of multimedia, including texts, images, or video in media, empirical models of nonverbal framing in political communication have lagged behind text analysis. To address this issue, I provide a tool to examine what stories are reported and how stories are described visually and verbally. Second, I introduce a hypergraph-based image retrieval technique that measures the relevance score of specific words to images to analyze how social minorities are frequently paired with stereotypical visual imagery and whether such visual patterns are different across partisan media outlets. By generating joint visual and text representation of social minority groups in a large-scale dataset of news articles, I demonstrate the connotative representation of social minority groups portrayed in verbal and visual information. This is the first research to introduce the concept of a hypergraph in political science to move beyond pairwise relationships among the object of interest, leverage the rich multimodal information from text and images simultaneously, and examine complex relations among them.

Author

• Seo Eun Yang, Ohio State University