VecOpt: A Word Embedding-Based Optimization Approach to Extracting Text Features

• In Event: New Approaches to Measuring Emotions

Abstract

Political and social scientists are often interested in studying latent content features, such as emotional appeals, language sophistication, and more recently, moral appeals, within messages like social media posts, congress speeches, and news articles. The challenge for measuring latent features is non-trivial. For example, manual human coding cannot easily scale up to process “big message data”, or in some cases, shown to be suboptimal to alternative strategies such as crowdsourcing when intercoder reliabilities fall below conventional threshold even after repeated training. The rise of computational content analysis (e.g., text-as-data approaches) has given rise to the popularity of constructing dictionaries as a low-cost, quick-to-use measurement strategy for processing large-scale textual data. That said, dictionary-based measurement suffers from a set of known shortcomings, including insensitivity to context-specific applications, difficulty in extracting signals from short-format texts (e.g., tweets), and inefficiency for neglecting contextual information in the documents.

In this case study, we introduce a new method for classifying latent features using moral appeals as an example – VecOpt, that integrates information from validated dictionaries with word embeddings through a nonlinear optimization model. Specifically, our model uncovers the coordinates of a target moral foundation through an iterative optimization algorithm that optimally matches the moral axes with the corresponding dictionary information in a shared vector space. VecOpt can then construct valid semantic axes in a vector space, onto which documents can be projected to measure the latent features of interest. In our preliminary analyses, we used the word2vec tool which contains 300-dimensional vectors for 3 million words and phrases to create the vector space. We report a case study applying VecOpt to measure moral appeals within a large-scale corpus of COVID-19 related tweets to demonstrate its advantages, conceptual foundations, and implementation protocols.

We focus on moral appeals in the current study because moral appeals represent a type of latent message features, invoking and appealing to one or more fundamental moral values within the framework of moral foundations theory (MFT). MFT posits the existence of six universal moral foundations: Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, Sanctity/Degradation, and Liberty/Oppression. Each moral foundation is like a “taste bud” that enables individuals to respond to situations that either uphold or violate the foundation. A burgeoning literature has applied MFT in different contexts to explain how individual or cultural differences in the relative importance of moral foundations underlie divides such as disagreement over culture war issues between the liberals and the conservatives. Studies have also shown that expressions relevant to the Fairness/Cheating foundation (i.e., appeals to Fairness/Cheating) increased support for liberal policy among those who endorsed the foundation, across partisan divides. Other studies found that moralized language can predict hate speech on social media. These findings suggest the potential of latent moral appeals in natural language to influence individuals’ attitudes and behaviors across a variety of issue contexts.

Our full manuscript in in-progress and we proceed in seven sections. As extracting latent message features such as moral appeals become more important for social scientists, Section 1 and 2 provide an overview about the strength and weakness of existing computational methods for classifying latent traits, such as the limited flexibility in incorporating contextual-level information using the dictionary approach (e.g., eMFD), and the performance gaps between classifying long vs. short texts with the word embedding method (Section 3). In Section 4, we then introduce a new method for classifying latent features using moral appeals as an example – VecOpt. VecOpt combines dictionary method and word embedding approach through optimization models. Mathematic and conceptural definitions have been clarified and examined in this section. Section 5 assesses the external validity of the VecOpt output by comparing it to other moral measurements such as eMFD, using human evaluation as the baseline through a pilot validation study. In Section 6, we applied VecOpt to extract moral appeals for more than 2 million Twitter posts (“tweets”) and associated the quantified moral appeals with user engagement data such as number of comments, and Section 7 concludes.

We aim to develop and release a R/Python package to help interested researchers more easily implement VecOpt. We will be able to present the full work introducing this novel computational content analysis approach at the APSA conference.

Authors

• Zening Duan, UW-Madison

• Sijia Yang, University of Pennsylvania

• Anqi Shao, University of Wisconsin, Madison

• Kaiping Chen, University of Wisconsin-Madison

• Yicheng Hu

• Yoo Ji Suh

• Jisoo Kim

• Heysung Lee, University of Wisconsin-Madison

• Xining Liao, University of Wisconsin-Madison