Robustness to External Validity Bias

• In Event: Advances in Experimental Design

Abstract

External validity of experimental results is essential in political science. Existing methods aim to generalize experimental results to a particular well-defined target population by re-weighting the experimental data to approximate the population data. However, these methods are sometimes difficult to implement either because it is infeasible to obtain data for the target population or because there is no target population that both analysts and skeptics can agree on. In this article, we consider a different goal — quantifying how robust an experiment is to external validity bias in general. In particular, we propose an index of external robustness by estimating how much re-weighting of the experimental data would be required to explain away an estimate of the causal effect in any target populations. If an estimated index is large, this implies stronger robustness to external validity bias; substantive results stay the same unless whatever target population data is significantly different from the experimental data. Our proposed approach has several advantages. First, estimation of the proposed index only requires the experimental data and does not require any target population data. Second, we provide simple benchmarks to help interpret whether an estimated index is large or not in each application. Finally, the proposed framework can incorporate recent machine learning estimators to improve efficiency. We illustrate the proposed approach by analyzing various randomized experiments, including a multi-site experiment.

Authors

• Naoki Egami, Columbia University

• Martin Devaux, Columbia University