Analyzing Text Data Indexed by a Hidden Characteristic: Method and Application

• In Event: Machine Learning and Text Analysis

Abstract

Many text data in political science, including but not limited to corpuses of news articles, have the feature that an observation is indexed by its author (individuals or organizations) and a characteristic that is often hidden in the text and unorganized without large-scale hand-coding. For instance, researchers often observe a stream of news articles from each media organization but don’t know which pairs or groups of articles are covering the same event without close inspection of each article. Ideally one would want to read and compare all pairs or groups of articles and recover the underlying data structure, but this may be infeasible in large corpuses.
On the other hand, many quantities of substantive interest depend crucially on the hidden index. American politics scholars may be interested in measuring differences in ideological leanings between media organizations in the coverage of same underlying political events. Comparative politics scholars may be interested in whether state-run media restrict discussion on sensitive news coverage more heavily than non-state counterparts in authoritarian countries. In both cases, estimating quantities that necessitate controlling for event-level heterogeneity is challenging.
In this paper, I propose a novel method to estimate quantities of interest which are conditional on a characteristic hidden in the text. Instead of extensive hand-coding, the method proceeds in the following four steps. It starts by forming candidate pairs of observations that potentially (but not necessarily) share a common index. After this initial pairing procedure, I require a sample of these candidate pairs to be labelled by researchers or human subjects as training data. Utilizing the training data, I estimate predictive models of the probability of a pair of observations having a common hidden index (1) as a function of a summary measure of text similarity and (2) as a function of the similarity measure and the difference in the outcome variable. Finally, I construct an estimator of the quantities of interest using the fitted predictive model, the similarity of all pairs and metadata (if available).
The method overcomes difficulties surrounding existing methods that are sensitive to critical tuning parameters and produce biased estimates when applied to estimate such quantities. Applying the method to the study of media bias, I find the ideological difference between media organizations is not always driven by how events are covered. Instead, the contribution of presentation bias to the total difference in ideological leanings between news articles published by media organizations varies in ways that are consistent with the structural difference between different media types.

Author

• Yimeng Li, California Institute of Technology