Visualizing and Using Expert-Coded Data

• In Event: Advances in Expert-Coded Data

Abstract

Making their data accessible to a wide audience has been the focus of much effort on the part of expert-coded data-gathering enterprises. Much of this effort is due to two concerns. First, the data are often of interest to a wide range of audiences, from policymakers to quantitative social scientists. Such widespread appeal means that the technical skills of audiences diverge widely, as do their desired applications. Maintaining methodological integrity while also ensuring that the data are readily interpretable is thus a difficult needle to thread. The second, related concern is that measurement uncertainty is an integral element of expert-coded data. Presenting this uncertainty to a variety of audiences is a similarly difficult task. The Varieties of Democracy (V-Dem) Project has been an ideal laboratory for threading these needles, as its data have been used in projects that range from cutting-edge quantitative research, to policy-oriented country rankings on different metrics, to undergraduate classroom presentations. This paper draws on this experience to demonstrate how to: 1) leverage a Bayesian IRT modeling framework to tailor outputs to different consumers, and 2) effectively communicate measurement uncertainty when presenting estimates and conducting downstream analyses. Though the Bayesian IRT framework that V-Dem employs to aggregate expert-coded has many advantages when correcting for expert error, it also has a key disadvantage relative to more traditional techniques: it is complicated. Most importantly, it necessarily produces estimates that are on a different scale than the original questions. This scale is amenable to some consumers, particularly academic researchers who are comfortable working in latent spaces, and for whom an interval-level scale is methodologically convenient. But other consumers, especially policymakers, are less comfortable working with this abstract output. In this paper I show how to work within the modeling framework to produce a variety of outputs/rankings, ordinal estimates on the original question scale, and probabilistic case comparisons that meet the needs of a variety of audiences, and allow one to simultaneously produce quality estimates from surveys while communicating to audiences with varying needs, and different levels of statistical literacy.

Author

• Kyle L. Marquardt, University of Bergen