Variable Selection and Sensitivity Analysis for Survey Weighting

• In Event: Improving Survey Experiments and Analysis

Abstract

Dramatic declines in response rates and recent failures in public opinion polling have reduced trust in survey results and analysis. While recent advances in survey methodology aim to improve the way in which auxiliary data is used to construct survey weights or model-based estimates, concern remains that there are unmeasured or unobservable factors that affect survey response. This paper proposes two techniques to evaluate sensitivity to remaining confounding in survey estimates. Researchers often have substantive knowledge about factors theoretically related to survey nonresponse. The first technique assumes these variables can be measured among survey responses, and uses a Markov Random Field to estimate a set of auxiliary variables, within the experiment, that are sufficient for unbiased estimation of population quantities; this set can be constrained by what is measurable in the target population. When required variables are measured among the survey respondents but not the target population, a sensitivity analysis is proposed. The second technique proposes a two parameter sensitivity analysis for confounders unmeasurable among both the survey respondents and target population, defined by the correlation between a discrepancy between estimated and true survey weights with the outcome and the variance of the discrepancy in the two sets of weights. Combined, these techniques allow researchers to evaluate sensitivity of their survey estimates to confounders that are unmeasurable in the target population or to those that are inherently unobservable.

Author

• Erin Hartman, UC Berkeley