Harmonizing Survey Measurements of Public Opinion on Climate Change

• In Event: Innovations in Survey Sampling and Questionnaire Design

Abstract

For decades, News media attention has focused relentlessly on natural science findings suggesting that the earth has been warming and that this warming poses significant threats to modern and developing societies. Yet despite cries for intervention by the U.S. government to reduce greenhouse gas emissions in the country, ambitious legislation has not passed, and the country continues to burn fossil fuels and perpetuate warming. One possible explanation for this is that perhaps the American public does not support such efforts to transform energy generation to rely on renewable sources. Perhaps the caricature of the nation as divided 50/50 on every issue applies to climate change as well.

But is this really true? Perhaps the best way to answer that question is from surveys done of truly random samples of American adults asking them questions about various aspects of the issue, including: whether the earth has been warming, whether warming has been caused by humans, whether warming poses a significant threat to people, whether effort should be devoted to reducing future warming, whether particular policy approaches to greenhouse gas emissions reduction should be implemented, whether people are willing to pay the financial costs of transforming modern and developing societies to rely on renewable energy sources, and what priority people would place on this issue.

According to some surveys of high quality samples, large and even huge majorities are on what might be called the “green” side of this issue, acknowledging the existence and threat of climate change and calling for remediation efforts. But according to other surveys done with high quality samples, American is much more divided and less supportive of those efforts. This discrepancy between survey results can have at least two important effects: (1) it may lead politicians to disregard public opinion when making policy-making decisions, because the public seems not to have crystallized and compelling views on desirable pathways, and (2) it may lead politicians and the general public to doubt the validity of survey research, concluding that modern surveys are incapable of producing accurate results. Both of these outcomes would be undesirable for the functioning of representative democracies.

This paper sets out to make sense of the discrepant results produced by various survey organizations. The process began with the creating a huge archive of survey results collected over 30 years by a wide array of survey organizations interviewing random samples of American adults, including the Gallup Organization, the Pew Research Center, and researchers at many academic institutions, including the University of Michigan, Yale University, Stanford University, and others. By thoroughly cataloguing numerous measurements, the current paper was able to demonstrate the extent to which different organizations seemed to be measuring the same opinion (e.g., has the earth been warming?) and the extent to which those measurements have been in agreement or disagreement across organizations in terms of the cross-sectional distributions of opinions at single time points and trends over time.

Having documented similarities and differences thusly, the project turns to exploring whether discrepant results can be explained by differences in question wordings employed by the different survey organizations. A remarkable number of potentially important differences in wordings were identified, leading to the development of a random of hypotheses about how different question wordings might explain differences in obtained results.

The project then tests these hypotheses by conducting experiments in national surveys where respondents were randomly assigned to be asked one version of a question or another, breaking down the differences between questions used in past studies into parts and evaluating whether each wording difference is responsible for different obtained results in past research. These studies are being conducted with large national samples of American adults (N = 1,500 each). The results yield support for a series of principles regarding best practices of survey question wording, illustrate how important wording differences can be in yielding seemingly discrepant results that are really not contradictory, and document the scientific robustness of survey measurements and the sensitivity of respondents to subtleties in how issues are posed to them in surveys. In the end, we are able to make a coherent case about what large and huge majorities of Americans believe on this issue and what point of view are less prevalent.

Thus, this project is about public opinion and about political communication. The process of communicating between the public and legislators is shown to be powerfully shaped by survey methods. And the process of communication between survey researchers and their respondents via specific wordings of questions is illuminated as well.

Authors

• Jon A. Krosnick, Stanford University

• Bo MacInnis, Stanford University