Asking about Complex Policies

• In Event: Drawing Better Inferences from Biased Surveys

Abstract

Survey researchers have long worried about the responses they obtain from asking about complex policies, concerned that respondents are providing non-attitudes or data riddled with measurement error (e.g., Converse 1964; Alvarez and Brehm 2002). These concerns have only increased as the world--and political issues--have increased in complexity. For instance, pollsters asked about President Joseph Biden’s Build Back Better legislation, a complex piece of policy with many components (e.g., Prokop 2021). More generally, every two years, the Cooperative Congressional Election Study (CCES) asks people how they would vote on various pieces of legislation in an attempt to jointly scale the ideological locations of the public and members of Congress. Pollsters also have asked about complex legal issues concerning the role of the Supreme Court in U.S. politics (e.g., Jessee, Malhotra, and Sen 2022). Finally, as the information environment has increased in complexity, public opinion scholars are increasingly interested in measuring levels of misinformation and disinformation--the theme of this year’s conference. These issues are compounded by worries that inattentive survey respondents populate Internet survey panels, which have emerged as a dominant mode of data collection in social science research (e.g., Berinsky, Margolis, and Sances 2014).

The literature has generally argued that task difficulty can compromise data quality because it is positively related to survey satisficing (e.g., Krosnick 1991; Krosnick 1999). One sign of satisficing is response order effects, or when respondents simply pick the top response option from a list in visually administered questionnaires. Such a practice can introduce both bias and error in survey data (Schuman and Presser 1996; Krosnick and Alwin 1987). The empirical literature is somewhat mixed, with some studies finding that order effects are more present in complex questions (e.g., Holbrook et al. 2007) with other studies finding the reverse relationship (e.g., Malhotra 2009).

In this paper, we assess the correlates of response order effects by analyzing survey responses to complex questions about cases before the Supreme Court and related political issues. We leverage randomized response order experiments embedded in three surveys conducted in 2020, 2021, and 2022. Across these three waves of data collection, 6,291 respondents provided their opinion on 41 upcoming Supreme Court decisions, providing over 84,000 observations for analysis.

On average, we observe that respondents were 2.7 percentage points more likely to select a response option when it was listed in the first compared to the second position. However, we also find substantial heterogeneity in this effect across questions and respondents. Questions containing more words exhibited higher response order effects, while respondents with more domain-specific knowledge about the Supreme Court were less likely to exhibit response order effects, even controlling for education. Additionally, these two variables exhibited a significant interaction: the relationship between question length and response order effects was strongest for the least-knowledgeable respondents. Although response order effects are estimated to be close to zero for shorter questions and for higher knowledge respondents, we find that among the most sensitive subpopulation—low-knowledge respondents answering long questions—the predicted response order effect was a whopping 16.9 percentage points.

Our findings have important implications for survey researchers trying to understand public opinion on complex policy issues. First, a tradeoff exists between providing detail and response quality. Adding more words to a question may make the description of the issue or policy at hand more informative or more precise, but it comes at the expense of possibly greater response order effects. Researchers should attempt to capture the essence of a policy question in as few words as possible. This is especially important because respondents with low substantive knowledge about the topic are most likely to exhibit this effect, meaning that reported differences across subgroups can be confounded by survey artifacts. Importantly, substantive knowledge is not simply captured by respondent education, meaning that weighting by demographics is not a silver bullet. Second, although our findings underscore the importance of randomly rotating response options, they also sound a note of caution. Random rotation reduces bias but does not eliminate measurement error. Nonetheless, variance can be reduced via increasing sample size whereas bias is unaffected.

Authors

• Stephen Jessee, University of Texas at Austin

• Neil Malhotra, Stanford University

• Maya Sen, Harvard University