Approaches for Estimating Treatment Effects in Quasi-Experiments: A Comparison of Three Methods

• In Event: 25.068 - Roundtable Session 8
  In Roundtable Session: 25.068-11 - Educational Statistical Research: Accuracy in Analytics and in the Context of Quasi-Experimental Designs

Abstract

A variety of statistical approaches have been proposed to aid in causal claims through selection bias reduction, but results can lead to different estimates and it is unclear which method should be trusted. The present study is a case study of two realistic simulated scenarios in which treatment effects were estimated using three approaches, multiple linear regression (MR), matching on propensity scores (matching), or inverse propensity weighting (IPW). Both simulated scenarios induced selection bias, and group size balance was also manipulated. The three bias reduction approaches showed similar treatment effect estimates. For balanced conditions, MR and IPW were better at reducing bias, while IPW had smaller (more precise) standard errors. Naïve regression (ignoring covariates) not recommended for any quasi-experimental situation.

Authors

• Elizabeth A. Dietrich, University of Washington - Seattle

• Elizabeth A. Sanders, University of Washington