Evaluating Democracy: Conceptualizing and Measuring Standards of Democracy

• In Event: Poster Session: Democracy & Autocracy

Abstract

Diffuse support for democracy is a kind of evaluation (e.g., Easton 1975, Norris 2011, Ferrín and Kriesi 2016a, Heyne 2019) and all evaluations imply a standard of comparison. When citizens perceive the actual performance of democracy to fall short of their ideal standards for democracy (a “democratic deficit”) we expect diffuse support to fall.

Recently, scholars have directed increased attention to the implied democracy standard, seeking to empirically estimate the magnitude of democratic deficits in countries, groups and individual citizens.

This paper introduces a new method for measuring democracy standards. Following Easton and more contemporary works, we conceptualize democracy standards as a kind of value. Values are most often measured by ranking methods (e.g., Rokeach 1973, Inglehart 1991, Schwartz and Bilsky 1987, Schwartz et al. 2010, 2015), but ranking methods have many disadvantages – not least among these is the inability of survey respondents to rank more than a small number of elements.

We combine content insights from the sixth round of the European Social Survey (Ferrin and Kriesi) with a pairwise ranking system originally proposed by ranking Bilskey et al. (2015). Bilskey’s method is impractical, forcing respondents to make scores of pairwise comparisons. To make pairwise rankings tractable, we create a hybrid questionnaire that combines features of conjoint experiments, ranking methods, and rating scales.

In this method, respondents are shown a pair of statements side by side. For example:

==============================================================================================
Which of the two values below is more important to you and to what extent is it more important?

(Presented on the left) Individual Rights: All adults enjoy the rights to worship, think, and travel as one pleases, along with the rights to own property, to privacy and the right to be left alone.

(Presented on the right) Political Rights: All adults enjoy the rights to assembly, petition and protest government decisions, along with the right to form groups to advocate for specific goals.
==============================================================================================


They are then asked to rate the pair on an ordinal scale ranging from 1 (left value is much more important) to 6 (right value is much more important). This method allows a high degree of specificity that allows respondents to consider a wide range of elements of liberal democracy and permits them to rank them closely (“the left values is slightly more important”) or not.

The resulting data allow the calculation of value rankings for countries and subgroups for a large number of elements, even though the typical respondent ranks only a small subset of possible pairings. At the individual level, latent value scores can be calculated for each respondent. This then allows the valid and reliable calculation of democratic deficit for individual citizens, politically relevant social groups within a society, and nations as a whole.

We demonstrate this using data from a recent nationally representative survey of US citizens. Half answered questions in the format of the European Social Survey and half were randomly assigned to rate the same elements of democracy using the paired comparison method. We assess the two approaches in terms of measurement quality: reliability, validity, and respondent satisficing.

Preliminary findings suggest distinct value groups in the United States, with some engaged citizens especially valuing aspects of political equality, some emphasizing individual rights and liberties, and some reflecting an overall feeling of disaffection. The method draws on question wordings developed outside the US, and therefore should be adaptable to many countries and cultures.

Authors

• Eric Plutzer

• Michael B. Berkman, Pennsylvania State University