Where to Place Sensitive Items in an Experiment? Pre- or Post-Treatment?

• In Event: The Methodology of Experiments

Abstract

Common practices in the use of experiments in political science are ever evolving, but one particular issue has continued to be the subject of debate: the use of post-treatment variables as covariates in regression analyses. Some argue that post-treatment variables should never be used as covariates because the treatment could bias the results of the post-treatment questions (e.g., Montgomery, Nyhan, and Torres 2018). Others, however, argue that sensitive questions should be used as post-treatment covariates in order to avoid priming effects before experimental treatment (e.g., Mendelberg 2008). Experimental studies on racial attitudes that include measures of racial resentment (RR) as covariates are not consistent in their timing of the RR battery, reflecting the lack of consensus on this debate. We seek to contribute to this debate by demonstrating, experimentally, the different biases that are introduced due to the timing of the RR battery in an experiment that uses treatments sensitive to racial priming effects. We argue that the results of the experiment will successfully adjudicate this longstanding debate in the literature and provide much-needed guidance for scholars in the future who want to study phenomena prone to bias from measuring racial attitudes and perhaps other sensitive measures.

Authors

• Tabitha Bonilla, Northwestern University

• Amanda Sahar d'Urso, Northwestern University

• Sam Gubitz, Northwestern University