Testing Assumptions in Experiments on Political Cognition

• In Event: Mechanisms Explaining Policy Attitudes

Abstract

Recent research that examines the relative influence of party cues and policy-relevant information in the attitude formation process concludes that most people are not “cognitive misers” ---rather, when they have policy-relevant information, they incorporate it into their attitudes. Unlike research from the Michigan school, which relied heavily on observational studies and painted a dark picture of “responsibility” in the American electorate, these new studies make heavy use of survey experiments. Why experiments? Briefly, in real-world politics, Republicans usually endorse conservative positions, and Democrats endorse liberal positions. How does one estimate the independent effects of party endorsements and policy content if Democrats always publicly favor liberal policies and Republicans favor conservative policies? The existence of such a problem pushes researchers toward survey experiments.

How do these party cue/policy content experiments work? Generally, these experiments randomly vary party cues and policy content such that treatment combinations include all possible combinations of party cues (neutral/Democratic/Republican) and policy content (liberal/conservative).

Respondents are randomly assigned to a treatment combination and asked their attitude on the policy in question. Party cues and policy content are completely uncorrelated so the estimation of treatment effects is straightforward.

Though this type of experimental design is simple, it operates on the assumption that liberal respondents prefer liberal legislation and conservative respondents prefer conservative legislation. Does this assumptions always hold?

In this paper, I argue that this assumption does not always hold. More specifically, I argue that people who think a particular issue is the single most important issue in contemporary American politics may think about the issue in a fundamentally different way than people who think the issue is of less importance. Take health care, for example. People that treat health care as a run-of-the-mill issue may have clear preferences regarding whether the government should spend more or less on it, and further, it may be the case that these preferences align with their general ideological preferences. People who think health care is the most important issue, however, may think differently about it ---they may think that health care should be more affordable to individuals, or that everyone should have access to quality care. And, this may hold for liberals and conservatives alike. In other words, one can imagine a subset of conservatives (e.g. those who think health care is the single most important issue in government today) that actually prefer a liberal policy solution. If this subset exists and a researcher ran a study on health care attitudes using the above-mentioned experimental design, the subset of conservatives that prefer a liberal policy solution would appear to be have “incorrect” (i.e. incongruent) preferences, and one would not conclude that these conservatives considered policy content when forming their attitudes. Of course, in analyzing this hypothetical data, one would underestimate the relative influence policy content on attitude formation.

Here, I test the aforementioned assumption using a two-wave study. In the first wave, I ask respondents to indicate their ideological identity, and I ask them to indicate what they believe to be the most important problem in American politics. I then take a random sample of respondents from the first wave (but I oversample conservatives that believe health care is the most important issue), and invite them to participate in the second wave where they are randomly assigned to a treatment combination (as detailed above). I then estimate the relative influence of party cues and policy content separately for liberals and conservatives, and I include interaction terms to assess the extent to which believing health care is the most important issue moderates the effects of party cues, on one hand, and policy content on the other. Briefly, I find that liberals react favorably to liberal policy content, and liberals who believe the issue is of particular importance react very favorably (the interaction between liberal policy content and issue salience is significant and in the expected direction). Conservatives react unfavorably to liberal policy content, but conservatives who believe the issue is of particular importance react to liberal policy content in a fundamentally different way than their counterparts. More specifically, the results on the interaction term between liberal policy content and issue importance is significant and positive, indicating that these conservatives react in a very favorable way to a liberal health care policy.

I argue that these results cast some doubt on the whether the above-mentioned assumption holds and I discuss implications for how these experiments should be interpreted.

Author

• David Ciuk, Franklin & Marshall College