The Devil is in the Counterfactual: Alternative Strategies for Evaluation Studies

• In Event: Analytic Innovations in Violence Reduction

Abstract

There are diverse ways to evaluate treatment effects with non-experimental data. At least part is due to examining different questions with different counterfactuals. Data from a study of the effects of an ignition-interlock device on DWI recidivism, where convicted offenders used the device during (1) a pre-probation period only, (2) a probation period only, or (3) both a pre-probation and a probation period, were used to create different matched comparisons by propensity score matching. For example, offenders who used the device only during a pre-probation period were matched with a control sample comprised of convicted DWI offenders who never used the device. These same offenders who used the device only during a pre-probation period were also matched with a sample of offenders who used the device only during a probation period. There are many possible comparisons of this type, addressing different questions and different counterfactuals. Seemingly similar comparisons can produce different conclusions about treatment effects, and no particular type of comparison is more valuable than another. However, some questions and counterfactuals are more useful for policy decisions, while others are more academic. These issues are considered in terms of their implications for understanding evidence about treatment effects with non-experimental data.

Authors

• Victoria Terranova, University of Northern Colorado

• Mark Stafford, Texas State University