Stimulus Validity

• In Event: Advances in Survey Experiments

Abstract

Experimental or survey research designs in political science very often require the use of stimuli supposed to represent some aspect of real-world information environments. For example, scholars studying how people process misinformation typically select some (fake) news articles actually circulating. Researchers in persuasion draft messages they believe to be representative of some larger phenomenon. Those studying negative campaigning select or construct political ads they deem typical of real-world cases.

This raises the question: What real-world information environment is represented? Striving for external validity when researching human subjects is received wisdom. To date, there is less concern about stimulus validity, i.e. the degree to which chosen stimuli represent some population of information. The issue has received some attention in adjacent disciplines, namely psychology and communication (e.g., Jackson and Jacobs 1983; Wells and Windschitl 1999; Slater, Peter, and Valkenburg 2015; Reeves, Yeykelis, and Cummings 2016;
Harris et al. 2022), but it is still mostly neglected in political science. In a world providing an abundance of information, and thus a practically infinite choice of stimuli, we argue that stimulus validity should be key concern for political scientists using experiments and surveys.

We approach the issues on multiple levels. First, we formally define stimulus validity in the language of estimands, estimates and inference and point to the parallels to external validity, accounting for modern frameworks such as M-STOUT (Findley, Kikuta, and Denly 2021). We discuss how concepts like “generalizability of treatments” or ecological validity do not cover what we mean by stimulus validity. We elaborate stimulus validity in formal terms both in settings of descriptive and causal research.

Second, we examine several popular designs to explore the issue of stimulus validity, re-analysing and replicating three peer-reviewed studies. Re-analysing a study by Allcott and Gentzkow (2017) about misinformation, we show how results would differ had the authors chosen slightly different sets of stimuli. Replicating Fridkin and Kenney (2008) about the impact of incivility, we illustrate how choosing particular values of a continuous independent variable when crafting stimuli matters for treatment effects. Finally, we replicate a study by Brooks (2011) to show that supposedly manipulating stimuli on just one dimension—a politician’s gender with two different names—may actually vary other, unobserved factors.

Third, we discuss some practical ideas how to avoid a lack of stimulus validity. First and foremost, we recommend to conceptualize a population of information, and if possible draw a random sample of stimuli from it. This may often not possible, and we discuss other methods that may help to better reflect the real-world information environment, such as factorial experiments that account for profile distributions (e.g. De la Cuesta, Egami, and Imai 2022), or pre-study content analysis.

We finish the paper with a discussion of limitations of our approach and open questions. To what extent may the problem be subsumed under the umbrella of external or ecological validity? In which research areas is the problem of stimuli selection most relevant? Although some scattered attempts to recognize the issue of stimulus validity exist, this projects aims to contribute by formalizing existing intuitions and illustrating the problem empirically. Overall, we hope to prompt researchers to think more carefully about stimuli and provide practical ideas how to make designs more valid.

Authors

• Bernhard Clemm von Hohenberg, GESIS Institute for the Social Sciences

• Lukas Otto, GESIS Leibniz Institute for the Social Sciences