Priming Bias versus Post-treatment Bias in Experimental Design

• In Event: Causal Inference: Post-treatment Bias and Conjoint Experiments

Abstract

It is now common knowledge that conditioning on post-treatment variables when estimating causal effects has serious pitfalls. In experimental contexts, however, collecting covariates pre-treatment can also introduce bias if respondents are primed by the covariate measurement in a way that interacts with the treatment effect. This paper assesses the tradeoffs between pre- and post-treatment measurement of covariates in experiments. First, we derive bounds for conditional average treatment effects and interactions for covariates measured post-treatment. Second, we propose a novel experimental design that randomizes whether covariates are measured pre- or post-treatment. Under this design, researchers can diagnose both priming bias and post-treatment bias, while allowing for unbiased estimation when only one is present. We conclude with practical recommendations for scholars designing experiments.

Authors

• Matthew Blackwell, Harvard University

• Jacob Brown, Harvard University

• Sophie Hill, Harvard University

• Kosuke Imai, Harvard University

• Teppei Yamamoto, Massachusetts Institute of Technology