Inference with Modern Difference-in-Differences Methods

• In Event: Threats to Validity in Impact Evaluation Research

Abstract

Several newly proposed difference-in-differences (DiD) estimators address biases arising from staggered treatment adoption and heterogeneous treatment effects. The reliability of the standard errors of these new methods has yet to be fully explored, particularly in settings with few treated units. This paper examines the finite-sample performance of seven leading DiD estimators under various simulated conditions, including placebo policies in state-level unemployment data and synthetic data that vary the number of units, time periods, and treatment heterogeneity. Results show that most estimators tend to over-reject when the number of treated units is small. Treatment heterogeneity induces its own bias, reducing rejection rates.  However, these biases do not perfectly balance out.  An imputation-based estimator paired with a wild bootstrap generally performs well across all simulation designs, maintaining rejection rates close to nominal levels when there is treatment homogeneity and providing conservative inference otherwise. These findings highlight the importance of carefully selecting both the DiD estimator and the inference procedure in practical applications.

Author

• Yuji Mizushima, RAND School of Public Policy
  Presenting Author