Estimating Dynamic Treatment Effects: Three Methods Compared

• In Event: Advanced Methods for Causal Inference and Power Analysis

Abstract

Summary: This article compares three popular approaches for estimating dynamic treatment effects: event studies (or dynamic difference-in-differences), synthetic controls (SCM), and generalized synthetic controls (GSCM). These methods are widely employed in applied microeconomics and other social science domains to estimate the causal impact of exogenous policy shocks, such as changes in state-level legislation. The article is intended as an accessible guide to practitioners; the working principles behind each method are explained, and their performance tested with both real and synthetic data. The major software packages implementing these methods in R and Stata are also compared.

Methods: While all three methods discussed in the paper estimate a dynamic treatment effect, the way they arrive at this estimate differs considerably. We begin with a high level summary of the methods, with a focus on minimizing jargon and elucidating the underlying principles and assumptions of each model. We then test each model using both real and synthetic data. The real-world data comes from the legalization of Marijuana in Colorado; we choose this example because the treatment effects are sufficiently large as to be easily identified even from a prima-facie analysis. We also test the models against two forms of synthetic data: AR1-time series, and time-series generated via. a Bai (2013) factor model.

Results: Our initial results suggest that all three methods perform well when the treatment effect is large relative to the underlying variance of the time-series in question. In noisier data, the method of generalized synthetic controls is prone to over-fitting, with the estimated treatment effect driven by spurious fluctuations in the synthetic counterfactual. Synthetic controls and event studies both perform well on real data, but SCM does considerably better when the number of available control units (the "donner pool") is small. Other issues, such as inference, are considered and we suggest best practices for practitioners.

Author

• Zhaochen He, Christopher Newport University
  Presenting Author