Machine Learning and the Deliberative Quality Index in Canada’s Territories

• In Event: Advances in Mixed Methodology

Abstract

This paper describes a new methodology for studying discourse quality using machining learning techniques for the study of discourse quality in Canada’s Northern parliaments, in the Yukon, Nunavut and the Northwest Territories. It builds upon DelibAnalysis, a tool that was developed to automatically analyze the quality of online political discussions using the Discourse Quality Index (DQI) (Fournier-Tombs 2018).

The original Discourse Quality Index was developed in order to quantify the discourse quality of parliamentary speeches (Steenbergen et al, 2003). Although widely used, it was also criticized for being difficult to code manually, coding being a very time-consuming and somewhat subjective task (King, 2009).

When DelibAnalysis was published in 2018, it addressed the manual component, allowing researchers to only code a small fraction of deliberations manually and use machine learning techniques to predict the quality of other speeches, based on their characteristics (word count, phrases used, etc.). However, the framework simplified the DQI considerably, predicting only three categories of discourse quality – low, medium or high. While this was considered appropriate for much shorter online political discussions, it was found overly simplistic for the current study.

While Yukon has a party-based adversarial type legislature, Nunavut and the Northwest Territories are consensus-based, where independent representatives work together to find solutions to territorial concerns. This use-case is very appropriate for discourse quality analysis, but, given the volume of data, also complex to address using manual methods.

This study therefore proposes an enhanced DelibAnalysis framework, based on speeches in the three territorial parliaments of Canada, which uses machine learning to predict each indicator in the Discourse Quality Index. It shows not only the cross-sectional applicability of machine learning to enrich qualitative research, but also the adaptability of the DQI to various contexts and methodological improvements.

The machine learning algorithm used is primarily based on random forests, a methodology consisting of grouping multiple decision tree algorithms and averaging their results. This algorithm is relatively straightforward to implement and has the advantage of also being easily interpretable. This interpretability also allows for a broader qualitative understanding of the data, due to the model’s feature (or variable) selection.

The paper outlines a new methodology that can be used by researchers who are interested in combining qualitative and computational analyses of complex, difficult to code, interactions such as parliamentary debates.

Authors

• Eleonore Fournier-Tombs, McGill University

• Michael Kenneth MacKenzie, University of Pittsburgh