Doubly Random, Doubly Robust: Accounting for Non-ignorable Non-response

• In Event: Drawing Better Inferences from Biased Surveys

Abstract

Non-response presents a major challenge in contemporary survey research. While significant progress has been made in using weights, multiple level regression and poststratification and quota sampling, all of these methods assume that non-response is ignorable, meaning that there is no relationship between response and opinion once we have controlled for covariates. Assuming non-response is ignorable means that (1) there is no omitted variable (such as social trust) that affects response and opinion (2) opinions cannot directly affect probability of response (e.g., people with strong opinions on a topic cannot be more likely to respond).

When non-response is non-ignorable, small correlations of response propensity and opinion can lead to major errors (Meng 2018; Bradley, Kuriwaki, Isakov, Sejdinovic, Meng and Flaxman 2021), a problem that is exacerbated by low-response rates and the use of non-random samples. Therefore many scholars are interested in analyzing surveys with models that directly incorporate the possibility of non-ignorable non-response (Mercer 2018). Researchers working across multiple fields including survey research, economics and epidemiology have produced many such models, including the Heckman mode (1979), copulas (Marra, Radice, Barnighausen, Wood and McGovern 2017), control functions (Das, Newey and Vella 2003) and bespoke maximum likelihood models (Peress 2010). These models have not been widely implemented by survey researchers due to concerns about the parametric assumptions embedded in some and a recognition that these complex models can be unreliable without high quality data.

This paper presents an approach that deals with both concerns. First, in order to produce high quality data, this paper uses a randomized response instrument, which is a randomly assigned treatment that affect an individual’s probability of response but does not affect the individual’s opinion. Implementing this is relatively easy for many survey designs as pollsters have considerable experience in identifying protocols that increase or decrease response propensity.

Second, in order to rely on minimal parametric assumptions, this paper implements the doubly robust estimator developed by Sun, Liu, Miao, Wirth, Robins and Tchetgen Tchetgen (2018). This estimator is based on a model in which the opinion be measured can directly influence response propensity. It is consistent if one (but not necessarily both) of the response and outcome models are correct.
This paper analyzes multiple surveys on U.S. politics, showing the similarities and differences in conclusions when using methods that do or do not assume non-response is ignorable. It also explores how within party difference vary across models as previous research has shown that the Democrats who respond are more liberal than Democrats who do not respond and the Republicans who response are more extreme than the Republicans who do not respond. These biases can offset each other when analyzing the full population but can lead to exaggerated estimates of polarization.

References
Bradley, Valerie C., Shiro Kuriwaki, Michael Isakov, Dino Sejdinovic, Xiao-Li Meng and Seth Flaxman. 2021. Unrepresentative big surveys significantly overestimated US vaccine uptake. Nature 600: 695-700.
Das, Mitali, Whitney K. Newey and Francis Vella. 2003. Nonparametric Estimation of Sample Selection Models. The Review of Economic Studies 70(1): 33-58.
Heckman, James J. 1979. Sample Selection Bias as a Specification Error. Econometrica 47(1): 153–161.
Marra, Giampiero, Rosable Radice, Till Barnighausen, Simon N. Wood and Mark E. McGovern. 2017. A Simultaneous Equation Approach to Estimating HIV Prevalence with Non-ignorable Missing Responses. Journal of the American Statistical Association 112(518): 484-496
Meng, Xiao-Li. 2018. Statistical Paradises and Paradoxes in Big Data (1): Law of Large Populations, Big Data Paradox, and the 2016 Presidential Election. The Annals of Applied Statistics. 12, (2): 685-726.
Mercer, Andrew. 2020. Decomposing Selection Bias in Nonprobability Surveys. Pew Research Center.
Peress, Michael. 2010. Correcting for Survey Nonresponse Using Variable Response Propensity. Journal of the American Statistical Association 105(492): 1418-1430.
Sun, BaoLuo, Lan Liu, Wang Miao, Kathleen Wirth, James Robins and Eric J. Tchetgen Tchetgen. 2018. Semiparametric Estimation with Data Missing Not at Random Using an Instrumental Variable. Statistica Sinica 28: 1965–1983.

Author

• Michael A. Bailey, Georgetown University