Can a Machine Tell The Difference? A New Way to Measure Congruence

• In Event: Advances in Measurement

Abstract

Measuring and understanding mass-elite congruence---the degree to which elected elites mirror their constituents---is a critical and growing endeavour. While new methods of analysing congruence overcome several issues with previous methods they still suffer from substantial limits. In this letter we suggest a new measurement strategy that overcome these limits while also offering several new insights into the nature of congruence: Making use of the accuracy of machine classifiers, that is, measuring to what extent a machine classifier can tell the difference between elites and voters. Intuitively, if a set of classifiers cannot tell which are elites and which are voters, we claim that they can be considered congruent. By contrast, when they can tell, they are not congruent; and critically, the degree to which they are not congruent is exactly proportional to the percent of elites the classifiers can correctly identify.
Our proposed strategy is to employ a diverse ensemble of predictive models---including logistic regression, random forest, and support vector machines---and use cross-validation to measure their out-of-sample area under the curve (AUC) to try to predict whom the elites are, using as features whatever variables we wish to investigate mass-elite congruence. Based on statistical learning theory and simulation we show how this serves as an intuitive, coherent, and valid measure of many-to-many congruence.
The merits of this approach is not deriving a valid measure of congruence for variables of any type and scale, but rather letting practitioners simultaneously grasp where this congruence is coming from. For instance, variable permutation importance---i.e. how much worse the ensemble would predict if a specific variable is permuted---shows how much non-congruence is uniquely attributable to each variable. In addition, marginal predictive performance can show the level of non-congruence at any specific value of any specific variable. It also opens up for measuring many-to-one congruence by estimating individual marginal accuracy for each elite, letting practitioners figure out which elites that stand out. We demonstrate the usefulness of these new quantities of interests with an example from a classical debate in the mass-elite congruence literature.

Authors

• Troy Saghaug Broderstad

• Mikael Poul Johannesson, University of Bergen