Satisficing and Measurement Accuracy of List Experiments

• In Event: Methods for Survey Research

Abstract

The increasing use of list experiments in political science is premised on the assumption that, by masking individual responses on a sensitive topic, such questions reduce sensitivity bias (i.e., strategic measurement error) and thereby increase reporting accuracy. We argue that, compared to direct questions, list experiments also have multiple ‘non- masking’ effects on reporting, such as satisfying (i.e., non-strategic measurement error). We show formally that non-strategic measurement error biases prevalence estimates towards 0.5, which invalidates the dominant validation approaches. We employ a new validation approach to list experiments which, unlike existing approaches, simultaneously detects false positives and false negatives. Focusing on turnout as a sensitive topic and verifying survey respondents’ true turnout using official records, we find for two different survey methods (paper-pencil and online) in two different contexts (Britain and New Zealand) that even a relatively small number of satisfiers can reduce a list experiment measurement accuracy compared to a direct question. Formal analysis indicates that such satisficing is likely to have nontrivial negative consequences for the efficacy of list experiments more generally, suggesting that their range of useful applications is much more limited.

Authors

• Patrick Michael Kuhn, Durham University

• Nick Vivyan, Durham University