Collecting National and State-Level Political Data Using Hybrid Online Samples

• In Event: Developments in Survey Methods

Abstract

With continued declines in response rates and escalating costs of traditional random-digit-dial (RDD) samples, researchers are searching for new approaches to obtaining representative survey samples of the U.S. population. This challenge is particularly acute when, as is often the case in political research, researchers must balance cost control with the need for a rapid turnaround, the control of selection bias, and/or the production of state-level or other sub-national estimates.

Samples from online panels offer one relatively affordable solution, but there are different types of online panels, each with its own advantages and limitations. Probability-based panels offer the traditional advantages of probability sampling, but available sample sizes are limited, particularly for state-level polling. Nonprobability panels offer much larger sampling pools and even lower costs, but are likely subject to selection bias, raising questions about their suitability for applications (such as election polling) in which highly accurate population estimates are needed. This raises the question of whether a “hybrid” approach—combining samples from both probability and nonprobability panels—provides a workable methodology for online polling in the context of political research.

This paper will use the 2022 Collaborative Midterm Survey to assess the utility and tradeoffs of hybrid online samples for political research that requires externally valid population estimates, including but not limited to pre-election polling. The Collaborative Midterm Survey included samples from three different survey organizations, each using a different methodology. One of the samples, fielded by SSRS, used a hybrid online methodology, blending a probability sample from the SSRS Opinion Panel with nonprobability samples from several opt-in panels. The study questionnaire included numerous items with high-quality external benchmarks to facilitate post-collection evaluation of sample representativeness. Several key states were oversampled, allowing assessment of both national and state-level estimates. The probability and nonprobability samples were blended and weighted using SSRS’s Encipher Hybrid methodology, which leverages the probability sample to adjust the nonprobability sample on non-demographic characteristics that might otherwise drive selection bias.

This paper will evaluate the accuracy of the estimates produced from this hybrid online sampling and calibration methodology, with an emphasis on comparing the appropriateness of this methodology for national vs. state-level estimation. This will include comparison of estimates of voting intention to certified election results; comparison of attitudinal measures to gold-standard external collections; and exploration of the extent to which additional weighting adjustments are needed to obtain acceptably accurate estimates. Results will provide insight into the viability of hybrid sampling and calibration as a scalable methodology for collecting political measures from representative samples of the U.S. population.

Authors

• Michael Jackson, SSRS

• Kristen Leigh Conrad, SSRS