New Evidence of Question Wording Ambiguity Suggests Revising ANES Affect Battery

• In Event: Measurement in Survey Research

Abstract

In a proposal on revising the affect battery questions for the 2018 ANES pilot study, Redlawsk, Roseman, Katz, and Mattes (2018) discussed ambiguity in the 2016 Time Series question wording.

The 2016 wording is “Think about [candidate]. How often would you say you’ve felt [emotion term] because of the kind of person [candidate] is or because of something [candidate] has done? [never, some of the time, about half the time, most of the time, or always]?”

A problem with this wording is that it is ambiguous whether the respondent feels the specified emotion toward the candidate named in the question or whether something the candidate said or did led the respondent to feel the emotion toward someone else (e.g., an opposing candidate). For example, Clinton supporters could report feeling hopeful because thinking about the kind of person Trump is led them to feel hopeful Clinton would win the election; Trump supporters could report feeling angry at Clinton because of things Trump said or did (e.g., at rallies).

Evidence of ambiguity in interpretations of affect battery questions comes from an analysis of 2016 ANES Time Series data. Separate logistic regressions were run to distinguish the strongest supporters of each candidate (those reporting voting for Clinton/Trump and giving her/him the highest rating on feeling thermometers; “idolizers”) from those voting for the candidate but giving lower thermometer ratings (“non-idolizers”).

When the affect battery terms for both candidates were used as predictors, results unsurprisingly indicated that Clinton idolizers (CIs, n=162) were distinguished from Clinton non-idolizers (n=1116) by more frequent pride and hope and less frequent anger she elicited. But these CIs were also distinguished from her non-idolizers by more often feeling hope when thinking about Trump (all ps<.01), which is surprising indeed—unless at least some of them are feeling hopeful about Clinton’s prospects because of the kind of person they believe Trump is or something he has done.

Were Clinton idolizers who reported some hope when thinking about Trump (“Trump-hopeful CIs”; n=10) really favorable to Trump? Compared those who never felt hope when thinking about Trump (“Trump-never-hopeful CIs”; n=152), t-tests showed Trump-hopeful CIs reported feeling more often afraid (p<.05) and marginally less often hopeful (p=.05) when thinking about Clinton—which could indicate more fear that she would lose and less hope that she would win. Notably, Trump-hopeful CIs rated the positive trait “speaks his mind” as less descriptive of Trump (p<.05) than Trump-never-hopeful CIs.

Indeed, Trump-hopeful CIs rated every positive candidate trait as more descriptive of Clinton than Trump. They also felt negative emotions more often than positive emotions when thinking of Trump, and positive emotions more often than negative emotions when thinking of Clinton. We converted the emotion and trait variables into z-scores to conduct t-tests comparing the emotions and traits about Clinton vs. Trump for the Trump-hopeful CIs. They felt pride about Clinton more often than about Trump, and felt disgust, anger, and fear about Clinton less often than about Trump (ps<.01). They also thought the traits of “strong leader,” “cares,” “knowledgeable,” “honest,” “speaks mind,” and “even-tempered” were all significantly more descriptive of Clinton than Trump (ps<.05).

Additional support for this hypothesis is found in correlations between hope elicited by Trump and other felt emotions for Trump-hopeful CIs. Hope elicited by Trump was significantly correlated with fear (r=.65) and disgust (r=.65) when thinking about him (ps<.05). Among the larger group of Trump-ever-hopeful Clinton non-idolizers (n=159), hope when thinking about Trump was correlated with disgust (r=.17) and anger when thinking about him (r=.17), and pride when thinking about Clinton (r=.23), although it was also positively correlated with pride (in Clinton?) when thinking about Trump (r=.26) and negatively correlated with hope when thinking about Clinton (r=-.23; ps<.05). The majority of these results again suggest that at least some respondents reported feeling hopeful when thinking about Trump because the ambiguous question wording allowed them to report thinking that “because of the kind of person he is or because of something he has done” Trump would lose and Clinton would win the election.

Overall, these results support changing the wording of the ANES affect battery questions. Redlawsk et al. (2018) proposed asking “How often would you say you’ve felt [emotion term] toward [candidate name] because of the kind of person [candidate name] is or because of something [candidate name] has done?” Specifying the object of the emotion is likely to improve the validity of the affect questions (whether asking about hope, anger, or other emotions) and enable better understanding of the relationship between emotions toward candidates and political outcome measures such as vote choice.

Authors

• John Crowell, Rutgers University, CCAS

• Ira Roseman, Rutgers University, CCAS

• David P. Redlawsk, University of Delaware

• Kyle Mattes, Florida International University