Towards Robust Prediction and Classification of Cluster-Specific Effects in Hierarchical Linear Modeling

• In Event: Advances in Multilevel Modeling for Complex Data and Study Designs

Abstract

Hierarchical linear models (HLMs) are widely used for high-stakes school ranking, but the impact of violating statistical assumptions on classification accuracy is understudied. Using empirical data and Monte Carlo simulations, we compare partially pooled HLM estimates to unpooled estimates and examine the utility of alternative distributional priors. We find that a school’s classification relative to a meaningful cutoff is model-dependent, and that shrinkage can obscure violations of the normality assumption in diagnostic tests. While non-normality has a small but observable negative impact on classification accuracy, simple fixes like changing the random effects' prior offer limited improvement. Ultimately, this work challenges the perceived objectivity of accountability systems, revealing the danger of making policy decisions based on what may be statistical artifacts.

Authors

• Graham Ward Buhrman, University of Wisconsin - Madison

• Jee-Seon Kim, University of Wisconsin - Madison