The Manual Identification of Topics by Different Groups

• In Event: Preprocessing, Annotation, and Interpretation in Computational Social Science

Abstract

It has been well identified that despite their name, the structures described in topic models may or may not correspond well to what humans naturally identify as topics. This has led to statistical innovations in the field of topic modeling that introduce features such as regularization to improve the coherence and interpretability of topics. While the differences between human and model identification of topics are important, insufficient work has been done on how different groups of people interpret or identify the same topics. The degree to which individuals associate certain words with each other depends on the cultural context in which individuals have lived. This sort of learning is evident in how respondents identify topics for purely methodological work, but it also has significant implications within political communication, which we illustrate with a motivating example.

We use campaign policy statements given in both English and Spanish as a corpus of documents. As direct translations of one another, the English language and Spanish language corpuses may be expected to contain identical topics with identical coherence, with coherence is measured with word and topic intrusion. Previous work on the subject has preliminarily indicated that while they may be faithful translations of the original English policy statements, the Spanish language policy statements do not reflect the common language and framing used by Spanish language voters. This provides an important methodological point about the complexity of using human coding to verify topic modeling results as well as a potent example of how this complexity plays out in politics with real implications in political communication.

Authors (listed alphabetically): Alejandro Flores and Jacob Jaffe

Authors

• Jacob Jaffe, Massachusetts Institute of Technology

• Alejandro Flores, Massachusetts Institute of Technology