Interpreting Turning Points in Longitudinal Text Corpora with Machine Learning

• In Event: Methodological Developments in Text-as-Data

Abstract

Text is a shared medium for interpretivists and machine learning analysts alike. Most new works in computational text analysis mention in passing that computational and interpretive specialists should work in tandem. However, little has been said about how such a collaboration could take place and what the payoffs may be. One reason why is because of a non-intuitive gap between substantive intuition about discursive significance and the quantities and ordering of language that make up a text corpus. How might this gap be closed? We argue that the disjuncture between discursive significance and material structure in a body of political texts can be reconciled through a union of machine learning and interpretive methods. To do so, we introduce a novel method for using supervised classification – a technique for machine learning text analysis – to discover rhetorical turning points in longitudinal text corpora. We illustrate how this machine learning approach can be used to suggest when to look for significant shifts in structures of meaning, and what to look for in characterizing them, even without prior substantive knowledge of the corpus. However, these rhetorical changes cannot be made meaningful absent an interpretivist moment, based on emic knowledge of the discursive community. We demonstrate the payoffs of our method on the most widely used International Relations text dataset - the UN General Debate Corpus (UNGDC), which covers 50 years of UN General Assembly floor speeches. Our text analysis identifies four salient rhetorical turning points over that span (1971-1981, 1982 – 1991, 1992 – 2007, and 2008 - 2020), when structures of meaning in global governance discourse shifted quite dramatically. We then discuss these transformations based on in-depth interpretation of contemporary diplomatic history, showing how these two methodological streams not only converge in their findings, but also enlighten each other in making sense of them. Overall, our approach contributes to ongoing methodological debates about the strengths and limitations of machine learning for political science, exploring along the way new frontiers in the dialogue between quantitative and qualitative methods.

Authors

• Scott Robert Patterson, McGill University

• Vincent Pouliot, McGill University