A Semi-Supervised Method to Extract Political Identities from Texts (Pre-Recorded)

• In Event: Methods to Study Diversity

Abstract

As political science turns its focus towards issues of identity, diversity and culture, we note the deficiencies of existing computational techniques in classifying texts based on such concepts. Specifically, such concepts contain an inherent multidimensionality, which make them difficult to uncover via unsupervised methods, yet vulnerable to narrow operationalizations based on researcher bias in supervised techniques. Additionally, such concepts can be quite rare in a target corpus (specifically categories related to minority or unobtrusive groups), and thus challenging to pick up via existing methods. With these issues in mind, we outline a semi-supervised method to utilize multiple iterations of human researcher input and computerized concept expansion to allow for the systematic classification of such categories.
Categories such as ideologies, political identities and culture, span multiple semantic and syntactic dimensions, including thematic topics, tone and styles – making them difficult to uncover via traditional supervised and unsupervised machine learning approaches. For example, while a topic modeling algorithm will successfully identify thematic topics such as ‘war’ or ‘elections’, it will have difficulty in picking up on categories such as political identities or ideologies. A researcher trying to measure categories like ‘liberalism’ or ‘conservatism’ is essentially searching texts for a concept that integrates economic stances, social values, and foreign policy attitudes (among others). Unsupervised techniques will have a difficult time uncovering such categories in their clustering of texts. Additionally, unsupervised methods are prone to ‘missing out’ on rare categories. Such limitations imply the need for a supervised approach; however, in preparing a training set for a supervised classifier, human researchers are vulnerable to personal bias when attempting to define and delineate such complex and multifaceted concepts, making the annotation process difficult and error-prone.
However, such concepts are increasing in significance to public discourse as well as theory. Thus, we propose a semi-supervised method that complements researcher domain expertise with computational concept expansion to overcome limitations of researcher bias, while annotating a training set used to fit a classifier capable of empirically and systematically measuring the many dimensions of target concepts. Our method begins with the researchers, who collect a representative group of documents for each concept (after formulating an initial operational definition of each). Then, we transform these base documents to document-level embeddings in a shared semantic space (with the target corpus), allowing us to detect in an unsupervised manner unidentified theoretical dimensions of the target concepts.
Through multiple iterations of human and computer concept expansion (including a number of benchmarks and validation steps), we gradually refine a training set for supervised classification. We utilize this training set to train a deep learning classification model to analyze texts and identify such concepts within a corpus. The multiple steps and procedures taken are meant to ensure a comprehensive method that considers syntax, semantics, context and human bias in the final classifier. Utilizing this technique, we were able to successfully extract and measure symbolic political identities in a corpus of nearly one million documents, made up of the general mass media coverage for a period of over a year and a half (spanning three elections).
The symbiosis of human and computer is increasingly being recognized as necessary to improve the use of computational methods. The process described here provides one form of standardized procedure to reduce the influence of bias in both the human researchers and the algorithms. We believe that this is but a first step in what could promise to be an important endeavor – continuing to cement this interaction and ensuring machine learning methods remain accurate, valid, and robust.
Additionally, this method offers a significant contribution to political science – the ability to handle multidimensional and rare categories. While it can be argued that all concepts contain multiple dimensions, for some this inherent complexity is more blatant. The complexity and amorphousness of concepts such as ideologies, identities and culture have previously limited the empirical exploration of such areas. We offer a quantitative method for studies of concepts that had previously been undertaken solely through the qualitative prism, offering exciting research opportunities.

Authors

• Dror Kris Markus, Hebrew University of Jerusalem

• Guy Mor, Hebrew University of Jerusalem

• Vered Porzycki, The Hebrew University of Jerusalem

• Alon Zoizner, Hebrew University of Jerusalem

• Effi Levi

• Avishai Green, Hebrew University of Jerusalem

• Tamir Sheafer, Hebrew University of Jerusalem

• Shaul Rafael Shenhav, Hebrew University of Jerusalem