The Utility of Non-probability Samples for Estimating Public Opinion

• In Event: Innovations in Survey Sampling and Questionnaire Design

Abstract

Under what conditions might non-probability samples yield valid and reliable inferences about public opinion within a population of interest?

The methods used to evaluate a non-probability sample generally involve the comparison of its estimates with those derived from a probability sample (Yeager et al. 2011, Berrens et al. 2003, Chang and Krosnick 2009). These research designs generally employ non-probability samples which are approximately equivalent in size to the probability samples against which they are being benchmarked.

In probability and non-probability samples of equivalent size, the former is almost certainly more likely to reflect the actual distribution of opinion within the population of the interest. In the context of a probability sample, assuming the sample to be approaching randomness, there is a threshold at which additional observations should produce negligible statistical power. This is not necessarily the case for the non-probability samples involved in a comparative analysis. I argue in this paper that, under particular sampling conditions, much larger non-probability samples can in fact produce estimates of public opinion that are on par with if not superior to their probabilistic counterparts.

To support this argument I analyze the electoral forecasts generated from survey data collected by national election studies conducted during federal elections in Canada and Australia—as well as general elections in New Zealand—between 2015 and 2021. I contrast the distribution of the vote modelled using these robust probability samples and compare it with extraordinarily large non-probability samples generated from a voting advice application called Vote Compass, which operated during the same elections. Drawing on the actual results for each election, I calculate the difference in the forecasting error between the national election study and the voting advice application samples.

Author

• Clifton van der Linden, McMaster University