A Comparison of Complex Language Processing Models in Political Analysis

• In Event: Embeddings in Multiple Languages

Abstract

Most recent analyses on political ideology using word vectors or word embeddings focus on systems with clear distinctions between major parties, such as in the US (Democratic Party, Republican Party) and the UK (Conservative Party, Labour Party). In these instances, more complex natural language processing (NLP) relying on neural networks seem to perform better than simpler models, given that the text corpora are sufficiently large (see, for example Rheault and Cochrane, 2020). Rodman (2020) suggests that word vectors can perform rather well even on ideological concept analyses over time based on relatively small data. However, few previous studies discuss the potential cross-country differences between political systems that use plurality rules compared with proportional systems that tend to produce coalitions. Potentially, simpler statistical semantic models that employ co-occurrence techniques, perform better in systems where party representatives are more oriented toward collaboration and where there are a greater number of parties. Moreover, the tendency of party representatives to align on a “left-right” political spectrum, versus a more ideologically complex system, could play a role independently of the electoral system.

This paper elaborates on the pros and cons of different NLP approaches ranging from simpler n-gram analyses to more complex neural networks, in order to map ideological positions of party representatives. Machine learning models can produce numeric vector representations of word meaning. The contribution of this study is to compare the representations produced by different models that have been trained with datasets from politicians from the two main parties in the US (Democratic party and Republican party) and Sweden (Social Democrats and Moderates). This comparison will allow us to identify words that are represented differently in the used datasets, i.e., when actors use the same words but in different contexts/meanings. Comparing quotes on crime from political representatives of main parties in the two different contexts allows us to test and elaborate different text and computer-based methods to characterize and evaluate the meaning of words and model usability in these contexts. In that manner, it is possible to capture different meanings of politically-charged words, such as “liberal”, between the main parties in these contexts.

Distinctions in the meaning of words between politicians from different parties are more likely to arise in the US context, where there are usually well-defined debate standpoints between the two major parties. In contrast, in political systems with proportional representation, simpler word clouds and co-occurrence analyses are likely to perform better. In these systems there are most likely differences in terms of nuance. At the same time, the overlap between parties is greater, which is exemplified by coalition formation between parties in Sweden. However, these patters may also change over time under proportional representation systems, as populist parties or fringe parties enter the arena.

We find that simple co-occurrence matrices often do not suffice to capture ideological content in political speeches, but that normalizing the results by setting co-occurrence in relation to single occurrence of words (the so-called “point-wise mutual information”) helps the model to perform better. Neural networks, however, often also require fine-tuning to obtain good results, and we discuss in which context and languages it is possible to leverage pre-trained word embeddings.

Authors

• Annika Fredén, Karlstad University

• Moa Johansson, Chalmers University of Technology

• Pasko Kisic Merino, Karlstad University

• Denitsa Saynova