Can Crowd-Sourced Data Substitute for Expert-Coded Data?

• In Event: Methodology in the Study of Political Institutions

Abstract

This paper examines the degree to which crowd-sourced data can substitute for expert-coded data. Political science research has increasingly used expert indicators to measure of complex political phenomena. At the same time, scholars have criticized expert surveys in terms of reliability, validity, cost-efficiency and replicability. For example, Benoit et al. (2016) compare crowd-sourced and expert coding of party manifestos and argue that crowd-sourced estimates of party policy positions should be preferably used over expert estimates. This paper furthers this line of research. First, we compare expert-coded data on key political issues from the V-Dem data set to crowd-coded data covering the same topics. Second, we use experiments to assess the degree to which manipulations can influence the substitutability of crowd-sourced data for expert-coded data. The experimental manipulations take two forms. First, we randomly assign half the participants to receive a lower or higher payment for the task to determine the degree to which incentives can increase crowd reliability. Second, we vary the type of task to assess how question framing affects crowd reliability. More precisely, half of the crowd-source sample receives the same questions as experts in the V-Dem data, while second group is asked to rank-order selected cases.

Authors

• Constanza Sanhueza Petrarca, University of Gothenburg

• Michael Bernhard, University of Florida

• Staffan I. Lindberg, University of Gothenburg

• Brigitte Seim, University of North Carolina-Chapel Hill

• Kyle L. Marquardt, University of Gothenburg

• Daniel Pemstein, North Dakota State University