Choosing Survey Respondents

• In Event: Advances in Expert-Coded Data

Abstract

A fundamental task in any survey project is determining the theoretical population of respondents and the questions this population is capable of answering. This task is complicated because respondents possess knowledge of the issue areas affecting their lives, or in which they have received formal training, but their level of knowledge is both relative and dependent on the cases and concepts the enterprise investigates. Survey enterprises must devote substantial attention to appropriately matching respondents and their expertise to concepts and cases, and weigh the advantages and disadvantages of recruiting different types of respondents. Understanding these trade-offs is crucial to identifying sampling strategies that will reduce the need to improve comparability with post-hoc adjustments. In this paper, I first present a typology of actors – experts, trained coders, crowd workers, citizen respondents – who provide survey data. After developing this typology, I then theorize about the types of questions and tasks that are best answered or completed by each category of respondents, and the characteristics of the questions/tasks, respondents, and the incentives they face that may condition data quality and cross-case and respondent comparability in responses. Drawing on a series of experiments conducted among Amazon Turk and Crowd workers, as well as Varieties of Democracy (V-Dem) experts, I test these expectations regarding the determinants of data quality, both within and across the respondent groups. I also compare one common survey question type – answering Likert-scale questions – to another type of question that has been employed in other surveys: paired comparisons of country-year cases. I assess whether asking respondents to compare two cases rather than answer a Likert-scale question about a case in isolation could be used to improve survey data validity and reliability. Next, I use the V-Dem confidence ratings to analyze variation in expert confidence, the level of certainty that experts self-report regarding their codings at the observation level. This analysis provides evidence regarding the degree to which experts are confident in their expertise; further regression analyses provide insight into what makes some experts more confident than others. Finally, I analyze variation how error-prone experts are, and discuss how to identify “bad” respondents and specific coding instances. Together, my results provide empirical insight into how respondent selection and recruitment affects downstream data quality and measure comparability.

Author

• Brigitte Seim, University of North Carolina-Chapel Hill