An Alternative Method for Understanding Public Support for Policy Proposals

• In Event: What Determines Views on Contemporary Political Issues?

Abstract

Surveys are an essential tool for understanding the extent to which the public supports policy proposals. But the usual way researchers test the effectiveness of arguments for and against a particular policy position has limitations. With the standard approach of randomly assigning a single message to each survey respondent, researchers sacrifice statistical power in order to provide clean estimates of public opinion. The lack of statistical power is particularly problematic when theory suggests heterogeneity in attitudes across important subgroups and as the number of viable arguments increases. To the extent that the public would encounter multiple arguments through the media or other channels, research is also constrained with regard to how the public weighs various arguments against one another. In this paper, we evaluate an alternative method for testing the effectiveness of multiple arguments that leverages random assignment to preserve sample size. We do so via an experimental design that allows us to compare the traditional approach to the alternative in a single survey that randomly assigns respondents to one of the methods. In the traditional approach, a control group reports their degree of support for a policy without hearing any arguments for or against a particular position, while the treatment group receives just a single argument (either for or against) and then reports their level of support. The alternative approach first asks all respondents to report their level of support for the same policy, without hearing any arguments. It then provides to all respondents each argument, one by one, in random order, with a question asking about their level of support after each argument. In an experiment examining arguments for and against granting Washington, DC statehood, we find that respondents are willing to update their answers after hearing each argument, and that the overall differences in the level of support across the two approaches are minor. We discuss the circumstances under which any biases presented by treating each respondent with multiple arguments is outweighed by the greater precision offered by preserving the full sample. Given that policy makers and interest groups also engage in survey research on policy proposals, this work has uses beyond academic research.

Authors

• Jared McDonald, Stanford University

• Michael J. Hanmer, University of Maryland, College Park