A Measure of Variable Importance for Sources of Treatment Effect Heterogeneity

• In Event: Causal Inference: Accounting For Heterogeneity

Abstract

Many social science questions require an understanding of heterogeneous treatment effects (HTEs). Causal forests (Wagner and Athey 2018) provide a systematic, honest way to evaluate HTEs over high-dimensional functions of covariates, even in the presence of irrelevant covariates. However, the resulting permutation-based variable importance measure is unreliable when the covariates are highly correlated or vary in their scale of measurement or when the covariates, both common characteristics of social science data. We propose a conformal inference method for estimating variable importance as a source of HTEs with asymptotic confidence intervals. We use a leave-one-covariate-out approach by first sample-splitting the data and using the training set to train a model with the full set of covariates and another model with all except the covariate whose importance is being estimated. Then, we fit the test set into both models to estimate the increase in prediction error by not having access to the covariate of interest. We show that we can construct asymptotic confidence intervals around this increase in prediction error for a measure of variable importance for inferences. We evaluate the performance of the proposed method through simulation and apply the method to two experimental datasets.

Authors

• Zenobia Chan, Princeton University

• Marc Thomas Ratkovic, Princeton University