Contingent Conjoint Conclusions? How Information Format Shapes Cognition

• In Event: Causal Inference: Post-treatment Bias and Conjoint Experiments

Abstract

The “conjoint experiment,” long popular in market research, is recently ascendant in the study of political behavior. And for good reason: subject to often-reasonable assumptions, it offers the ability to analyze the impact on citizen choice of many different attributes of a candidate or policy. A conjoint is essentially a many-factor, fully-factorial, forced-choice experiment in which many features of a pair of choices are randomly (and independently) manipulated to create vignettes, and in which respondents are asked to repeatedly rate vignettes or choose between pairs of vignettes.

Following Hainmueller and colleagues’ (2014) assimilation of conjoint methods to the potential outcomes causal inference framework, there has been an explosion in the use of conjoints. There has also been active methodological research on various aspects of its design and analysis, including the impact of varying numbers of attributes and vignettes; the analysis of subgroups of respondents; the analysis of interactions among attributes; and more.

There has been little attention, however, to the tabular format typically used to present information about the choice alternatives. There is reason to think this may matter, however, because different presentation formats can lead respondents to engage in different sorts of cognitive strategies. In particular, tabular display formats can favor certain cognitive heuristics, and can encourage the use of meta-cognitive strategies that shape decision-making. When the same information is presented in narrative form, on the other hand, respondents are more likely to engage in “narrative transportation” (i.e., be carried away by the story), and more apt to form gestalt impressions of candidates as whole people rather than lists of attributes. All of this affects how the information presented is perceived, organized, remembered, evaluated, and processed into summary evaluations. This has important implications for the validity of conjoint experiments that present treatments in tabular format, because that format does not, in general, correspond with the ways voters encounter information about candidates in real elections.

In our paper, we present a content analysis of recent conjoint experiments in political science journals, to demonstrate the technique’s increasing popularity and document the presentation of information in tabular format in almost all cases. Then, we draw on several literatures in social psychology—on person perception, narrative reasoning, and dual process models of persuasion—and in market research—on “narrative transportation”—to develop a series of hypotheses on the ways that tabular and narrative presentation of candidate information will shape cognitive processing differently. In the heart of the paper we present results from several experiments with mTurk respondents, in which we replicate several recently-published candidate choice conjoint experiments. We randomly assign respondents to receive information on candidates either in standard tabular format, or in narrative paragraph form. In each case, the paragraph convey the same information as the table in the original study, but organized into a narrative structure.

We find that presentation format has subtle, but noteworthy, impact on decision-making. When respondents face tabular information, they are more apt to weigh a single attribute heavily across the various vignettes; this effect is strongest among those with lower political sophistication. In addition, when presented with narratives, respondents make more inferences about the candidates that go beyond the information given. These results do not invalidate tabular-style conjoint experiments for political behavior research, but they do suggest some limits in their interpretation. We conclude by considering the implications of our findings for experimental methods in political behavior, and also for the study of political cognition and persuasion more broadly.


Hainmueller, Jens, Daniel J. Hopkins, and Teppei Yamamoto. 2014. “Causal Inference in Conjoint Analysis: Understanding Multidimensional Choices via Stated Preference Experiments.” Political Analysis 22(1): 1–30.

Authors

• Nicholas Winter, University of Virginia

• Leah Malkovich, University of Virginia

• Simonas Cepenas, University of Virginia