Survey Data Transparency in an Era of Nonprobability Samples

• In Event: Innovations in Survey Sampling and Questionnaire Design

Abstract

Many social scientists employ survey methods to explore the contours of human behavior. Public opinion studies are especially reliant upon interviews with human subjects, but due to technological changes and broader social science trends, less is known about survey sample origins than in the past. While scholarly attention typically focuses upon mode differences (e.g., internet vs. in-person or telephone interviews) or measurement alternatives (e.g., whether one attitudinal scale outperforms another), variations in sample quality receive much less introspection. Crucial details about the sampling frames of nonprobability (i.e., “opt-in”) surveys are unknown or unknowable. While there are tangible benefits when it comes to low-cost convenience samples, there are also risks and assumptions that are unacknowledged or portrayed in a misleading fashion. For instance, while a survey can plausibly claim to be national in scope with interviews from different states or regions, calling a nonprobability survey “representative” is potentially problematic. Opt-in sample optimism rests upon assumptions related to (A) homogeneous treatment effects, (B) the efficacy of weighting schemes, and (C) evidence pointing to similar findings across different opt-in samples or, less often, with comparisons to true probability samples. In the 21st century, there has been a movement toward greater transparency when it comes to analytical decisions in social science research through pre-registration of research questions, hypotheses, and analytical plans. Based upon a review of past practices over the last few decades, we argue that the same scrutiny should be applied to survey data origins.

Authors

• Jennifer Jerit, Dartmouth College

• Jason Barabas, Dartmouth College