What Topic Model Should I Choose? Measurement Error Induced by Model Choice in Automated Text Analysis

• In Event: 4377 - Computational Sociology

Abstract

The dream of unsupervised automated text analysis is that we become able to identify valid and
reliable topics at scale without polluting data with arbitrary human judgements. In this paper we
show that current criteria for labeling and accepting topic models include arbitrary judgements and that they lead to nontrivial measurement errors. Model labeling and choice relies on three unwarranted assumptions 1) semantic coherence between the most predictive documents within topic cluster and the rest of the documents 2) insignificant variation between topic models classified as the same topic 3) explicit performance measures are not necessary.
Using data from Facebook 46.000+ posts from Danish politicians we compare simulations of 1600
topic models to a labels created by a pre-defined dictionary of 15 topics and find 8969 matches. To
investigate the varying performance of possible choices we narrow this down to a set of 2960 equally plausible choices. Our results strongly suggest that topic models are not 1) semantically coherent and 2) do not treat all topics – or the topic articulation of different actors – equally, leading to differential bias in the results. To illustrate the consequences for research in political communication we simulate 1600 end-to-end research projects(model labeling, model acceptance and conclusions) of opportunism in political communication. We show that political opportunism can be assigned to all political parties depending on choice of model. We end by pointing at alternative research strategies for automated text analysis.

Authors

• Hjalmar Bang Carlsen

• Snorre Ralund, University of Copenhagen