Evaluating Different Covariate Balancing Methods: A Monte Carlo Simulation

• In Event: Poster Session: Political Methodology & Political Science Education & Civic Engagement
  In Poster Session: Poster Session: Political Methodology

Abstract

The purpose of this paper is to examine the effectiveness of different matching and weighting methods in improving covariate balance and correcting for bias in treatment effect estimation using Monte Carlo simulations (hereafter referred to as simulations).

Matching is a method that aims to reduce bias in treatment effect estimates by making the effects of factors other than treatment more homogeneous between the treatment and control groups. One of the most commonly used matching methods, which have been developed rapidly since the 1980s, is propensity score matching (PSM).

The bias correction effect of PSM has long been questioned by experimental researchers (Arceneaux et al., 2006). Recently, in addition to that, researchers such as King and Nielsen (2019), who do not deny the effectiveness of the matching methods per se, have also harshly criticized PSM. According to King and Nielsen’s simulation analysis, PSM can worsen rather than improve covariate balance and increase the bias in estimates of treatment effects (PSM paradox). Therefore, King and Nielsen (2019) suggest that researchers should use other matching methods such as Mahalanobis distance matching (MDM) and Coarsened exact matching (CEM) to avoid the PSM paradox. On the other hand, Guo et al. (2020) argue that the results of their simulation analyses do not allow us to conclude that MDM and CEM are more effective than PSM in reducing the bias of the estimated treatment effects.

Puzzlingly, neither of them shows how the bias in the estimates of treatment effects changes as the covariate balance changes. They both emphasize the importance of post-matching covariate balance checks. However, King and Nielsen (2019) only show the relationship between the numbers of observations removed by matchings and the maximum estimate biases of the treatment effect, or the relationship between the numbers of observations removed and the covariate balance. Guo et al. (2020) also only show the relationship between the numbers of observations after matchings and the mean values of the estimated biases (real values) of the treatment effect. In addition, King and Nielsen (2019) do not analyze the effects of unmeasured confounders that applied researchers often worry about in real-world empirical analyses.

Thus, we examine the relationships between covariate balance, measured/unmeasured confounders, and average biases in treatment effect estimates through an extension of the simulation in Guo and Fraser (2014).

We examine five major covariate balancing methods: PSM, MDM, CEM, Inverse probability weighting (IPW), and Entropy balancing (EB). The results suggest that PSM, MDM, CEM, IPW, and EB effectively improve covariate balance. The PSM paradox pointed out by King and Nielsen (2019) is not observed in our simulation setting in the relationship between the number of removed observations and covariate balance. Rather, our simulation analyses suggest another paradoxical relationship between covariate balancing and the average biases in the estimates of treatment effect: improving covariate balance by PSM, MDM, and CEM in settings with measured or unmeasured confounders can lead to the increased average biases in the estimates of treatment effect at some point (the paradox of improving covariate balance).

Our simulation results also suggest that the performances of five covariate balancing methods in bias correction are not necessarily better than those of OLS estimation using raw imbalanced data in settings with measured or unmeasured confounders. The average biases of the treatment effect estimates obtained by OLS are generally lower than those observed in estimates obtained by IPW, EB, and MDM and as low as those from PSM and CEM in both settings with measured or unmeasured confounders.

Unfortunately, our simulation results do not provide clear guidance on the extent to which the covariate balance should be achieved to minimize the estimation bias. In addition, it is unclear whether the phenomenon, which should be called the paradox of improved covariate balance, is a phenomenon observed generally in settings with measured/unmeasured confounders and whether it would be observed in different simulation settings and with different bias correction methods. Further investigations are necessary.

References
Arceneaux, K., Gerber, A. S., and Green, D. P. (2006), “Comparing experimental and matching methods using a large-scale voter mobilization experiment,” Political Analysis, 14(1): pp.37-62.
King, G., and Nielsen, R. A. (2019), “Why propensity scores should not be used for matching,” Political Analysis, 27: pp.435-454.
Guo, S., and Fraser, M. W. (2014), Propensity score analysis: Statistical methods and applications, 2nd edition, Los Angeles, SAGE publications.
Guo, S., Fraser, M., and Chen, Q. (2020), “Propensity Score Analysis: Recent Debate and Discussion,” Journal of the Society for Social Work and Research, 11(3): pp.463-482.

Author

• Hideki Fukui, Ehime University