Beyond Topics: Semi-Supervised Learning for Texts from a Measurement Perspective

• In Event: Advances in Measurement

Abstract

This project proposes a new methodological framework to use text data as a measurement in political science. In most political science empirical settings, researchers are not necessarily interested in topics of the text data. Instead, empirical researchers convert a large amount of text data into a numerical measurement of a concept of interest compatible with statistical analyses. This project fills this gap by introducing and improving semi-supervised learning models, with which researchers only hand code a small set of texts and let the computers learn the rest, taking into account that such labelling would then be in a causal context. Recent literature, such as Miller et al. (2019), promotes the use of active learning for automatic labelling of unlabeled data, but has not yet brought in the possibility of measurement errors into the model, which limits the potential use of the method in causal contexts.

Egami et al (2018) summarizes that there are two types of text-as-data research in political science. One is text as outcome, while the other is text as treatment. For both of these types of research, researchers either have to rely on hand coding to develop a codebook and operationalize the text-as-data outcome into a numeric measurement compatible with statistical analyses (Roberts et al, 2014), or can only makes use of topic models to project high-dimensional texts into low-dimensional treatments (Fong and Grimmer, 2016).

Yet, for many political science subfields, researchers may be interested in variables other than topics. For example, in the area of authoritarian politics, to study whether Chinese court has local protectionism, Long and Wang (2014) hand codes hundreds of cases published by the Supreme Court of China on its Gazette, into a dataset containing ownership structure and location of parties involved in the disputes, as well as the results of the dispute. Starting from 2014, the Supreme Court of China requests all courts in China to publish their verdicts online. Up until the year of 2014, in Henan province alone, over 1 million cases were published (Liebman et al, 2019). Such magnitude of number of cases makes it prohibitive for researchers to hand code the winner of the cases any more.

On the other hand, in international political economy, to study how immigration policy and visa restrictions have an effect on trading policies, Peters (2017) hand codes immigration policies across various periods and jurisdictions for her regression analyses. Nowadays, IATA maintains a public database, though in the form of texts, containing more than 40,000 entries of policies on whether a traveler travelling with a specific passport would need a visa for her destination country. The texts contained in these databases are not usually in a clear-cut sense of yes or no. Instead, they are usually in the form of “visa required, except that (a) (b) (c)”. Some of the conditions for visa exemptions are easy to satisfy, which makes the visa requirement essentially a visa exemption. A close reading and hand coding of these policies would be time-consuming. The intransparent coding process would make it harder for readers and researchers to discover coding errors should there were some.

A research gap emerges between the demand of empirical researchers in using texts as a measurement beyond topics and methodological toolkit available. Based on ideas proposed in machine learning literature on semi-supervised learning (eg. Zhu et al 2002) and split-sample methods in interval inference (Athey and Imbens, 2016; Egami et al 2018), this project proposes a framework where empirical researchers could rely on to semi-automatically label their text data and then make use of these labels in their causal analyses.

This project will make three contributes. First, it enables researchers to measure complicated concepts with text data, beyond topics. In semi-supervised learnings, a researcher would only hand code a small set of the data, the computer would learn and make the rest of the coding automatically. Second, this framework helps make the coding process more transparent and replicable. In semi-supervised learnings, the same input coded cases would guarantee the same prediction for unlabeled data. Publishing the small set of hand-coded data by the researcher would enable the reviewer to examine the possible coding errors in the labelling process. Third, this project bridges the gap between Miller et al (2019) and Egami et al (2018). Miller et al (2019) indicates that the prediction performance for a model trained on a hand-coded data randomly sampled from the population is bad because some categories of events may have rare appearance. They then thus proposes several methods to improve the sampling process. This project extends Miller et al (2019) by integrating both the sampling procedure and prediction model with the framework of causal inference.

Author

• Shiyao Liu, New York University Abu Dhabi