Abstraction and Detail in Experimental Design

• In Event: Advances in Experimental Methods and Designs Across the Subfields

Abstract

Experimentalists in political science often face the question of how abstract or concrete their experimental stimuli should be. Typically, this question is framed in
terms of a tradeoff between experimental control and generalizability, on the one hand, and realism and richness on the other: the more political context you introduce into your studies, the less control you have, and the more difficulty you have generalizing your results. Yet we have conceptual reasons to question this framing of the tradeoff, and there is relatively little systematic evidence experimenters can rely on when calibrating the degree of abstraction of their studies. We seek to make two contributions with this project. First, we provide a conceptual framework that better characterizes the advantages and disadvantages of adopting a particular level of abstraction or concreteness, including whether a study is hypothetical or not. Second, we replicate a range of classic vignette-based survey experiments from the international relations literature,
varying three key elements of abstraction: whether the scenario is described as hypothetical or not, whether the actors in the vignette are generic and unnamed, fictional, or real, and the degree of context added or subtracted to the studies, to explore how, and for whom, the results change. In this sense, we hope to offer guidance experimentalists can turn to when designing their studies.

Authors

• Ryan Brutger, University of California, Berkeley

• Joshua D. Kertzer, Harvard University

• Jonathan Renshon, University of Wisconsin, Madison

• Dustin Halliday Tingley, Harvard University

• Chagai Weiss, University of Wisconsin - Madison