A Generalized Sensitivity Analysis for Observational Designs

• In Event: Methods to Study Accountability and Criminal Justice

Abstract

We develop a design-based sensitivity analysis that is motivated by recent debates on racially biased policing. Scholars argue that estimates of racial bias in police use of force under the assumption that police are equally likely to stop Blacks and Whites likely understate evidence of racial bias. While scholars have alluded to formal sensitivity analysis as a way to reason through this possibility, existing tools are surprisingly ill-equipped to do so. The statistical machinery of sensitivity analysis is oriented to cases in which scholars reject a given null hypothesis: Given a particular value of a sensitivity parameter, design-based sensitivity analyses assume the worst-case scenario of unobserved confounding, which implies that p-values only increase over different values of the sensitivity parameter. But in cases where researchers worry that unobserved bias leads them to understate a causal effect, such a worst-case scenario is nonsensical. We develop an agnostic sensitivity analysis that uniformly draws from the set of possible configurations of unobserved confounding, subject to certain researcher-specified constraints. We also enable scholars to draw on domain knowledge about plausible configurations of bias instead of having to assume a perfect storm of unobserved confounding.

Authors

• Thomas Leavitt, Harvard University

• Luke Miratrix

• Jacob Bowers, University of Illinois, Urbana-Champaign