Modelling Heterogeneity Using Complex Sparsity

• In Event: Machine Learning for Measurement

Abstract

Data analysis starts by making theoretically informed assumptions about the structure of the world. Yet, we also typically have reasons for expecting certain types of heterogeneity in our analysis (e.g. temporally or spatially varying effects). This paper provides a unified framework for modelling the most common sorts of heterogeneity by leveraging the concept of "complex sparsity" from machine learning. This framework bridges two existing traditions (classical "random effects" and "machine learning") to suggest that if we are willing to use our existing knowledge to make assumptions about the patterns of heterogeneity in the data (e.g. that neighbouring units share similar heterogeneity), then we can use this framework to estimate highly flexible models for this heterogeneity (unlike most classical random effects models) while retaining interpretable results (unlike most machine learning methods).

"Complex sparsity" combines two types of sparsity from computer science (generalized and group LASSO) that penalize linear and quadratic combinations of coefficients. This allows for more complicated types of restrictions to be enforced at the posterior mode / penalized maximum likelihood estimate and thus mimic, inter alia, change point models and splines. In this paper, I focus on what I call "geographic" or "categorical" sparsity that fuse elements of a vector to each other (e.g. collapsing levels of a categorical variable). I show, as outlined later in this abstract, that this type of sparsity has important relevance to the estimation of models with fixed effects.

To implement this idea for social scientific questions, however, it is necessary to develop a new inferential procedure as existing methods cannot quantify uncertainty nor estimate non-linear models without approximations. This paper addresses this by deriving a new Bayesian representation of complex sparsity and associated inferential procedures. Specifically, I unify and extend the generalized and group LASSOs in a Bayesian framework; thus, a fully Bayesian procedure exists for sampling this model. I provide novel (and non-trivial) results to characterize the prior exactly in terms of its normalizing constant and hyper-parameters and thus allow for simple fully Bayesian inference on the strength of regularization as well as the parameters of interest. I also discuss extensions to other global-local forms of shrinkage (e.g. horseshoe; LASSOplus) and show that complex sparsity does work in those cases, although it raises some more subtle issues about sampling hyper-parameters that requires further investigation.

Further, as fast inference is important for prediction and starting a Gibbs Sampler at reasonable values, I leverage recent work on data augmentation to derive an EM estimation procedure that is guaranteed to converge to a posterior mode. This allows for exact estimation of non-linear models in this framework (an improvement upon existing approaches). These point estimates could be used for prediction, refitting the model given the estimated pattern of sparsity, or starting a Gibbs Sampler at very "good" initial values.

I apply this to the thorny question of fixed effects: Despite being a very common empirical strategy, there are many ambiguities and researcher degrees of freedom in how to include fixed effects (e.g. at what level; to interact with other FEs). I re-analyze an important recent paper on the impact of residential segregation on political outcomes and public goods provisions (Trounstine 2016). I find that while the use of FEs in the published paper does not survive the usual robustness checks (i.e. varying the fixed effects specification in plausible ways), using complex sparsity—with the assumption that some units likely share the same underlying FE—allows for the original results to be recovered at typically significant levels. Thus, complex sparsity allows for efficiency gains and flexible modelling while providing interpretable results if we are willing to use our case-specific knowledge about the patterns of underlying heterogeneity.

Author

• Max Goplerud, Harvard University