Improving Clinical Decision Utility beyond XGBoost using Variance Adaptive Gradient Boosted Bootstrap Modeling (VAGBBM)

• In Event: Machine Learning and Fairness: Applications in Lending, Benefits, and Clinical Decision-Making

Abstract

Background: Substance use treatment outcomes reflect complex interactions among clinical, behavioral, and structural factors, producing substantial heterogeneity that challenges conventional predictive modeling. Standard approaches such as gradient boosting prioritize global discrimination but often overlook subgroup variability, limiting stability, interpretability, and clinical relevance. This study evaluates a novel Variance Adaptive Gradient Boosting Bootstrap Modeling (VAGBBM) approach to determine whether it improves predictive robustness, captures nonlinear structural risk, and enhances clinical decision utility relative to standard methods.

Research Design and Methods: A comparative predictive modeling design was conducted using Treatment Episode Data Set data with a binary outcome of treatment completion. Models included standard XGBoost and VAGBBM, which incorporates bootstrap aggregation with variance informed adaptive weighting. Performance was evaluated across discrimination, probabilistic accuracy, calibration, predictive stability, interpretability, and clinical utility. Subgroup analyses examined structural heterogeneity defined by housing instability and insurance status. Data management was performed in Stata v18, and modeling was conducted in RStudio.

Results: Standard XGBoost demonstrated marginally higher discrimination (AUC = 0.648; AUPRC = 0.566) relative to VAGBBM (AUC = 0.645; AUPRC = 0.561), although absolute differences were small (ΔAUC = 0.004). Probabilistic accuracy was comparable across models, with VAGBBM yielding a Brier score of 0.226 and log loss of 0.643, closely aligned with XGBoost (Brier = 0.226; log loss = 0.642). Subgroup analyses revealed greater predictive stability under VAGBBM, with lower bootstrap prediction variance in high heterogeneity groups (prediction SD = 0.014) compared to low heterogeneity groups (prediction SD = 0.01643), alongside consistent discrimination across strata (AUC range = 0.624 to 0.645). Calibration diagnostics indicated minimal global bias (intercept = -0.005) and moderate probability scaling distortion (slope = 1.123), with significant Hosmer Lemeshow miscalibration (χ² (8) = 48.854, p < .001), reflecting sensitivity to large sample size rather than substantive misfit. Distributional properties of predicted probabilities showed a mean of 0.426, median of 0.443, and range from 0.189 to 0.797, with moderate outcome prediction correlation (r = 0.273). Decile based calibration demonstrated strong alignment in mid to upper risk strata (e.g., decile 10: predicted = 0.656, observed = 0.667), with modest underestimation in lower deciles. Feature attribution analyses using SHAP indicated that structural determinants dominated predictive influence, with living arrangements, employment status, education, and insurance status consistently ranked among the top predictors. Nonlinear effects were evident across these variables, particularly within structurally vulnerable subgroups. At the optimal classification threshold (0.394), VAGBBM achieved high sensitivity (0.849) with lower specificity (0.352), reflecting a decision profile prioritizing detection of high-risk cases. Decision curve analysis demonstrated that VAGBBM yielded greater net benefit than treat all and treat none strategies across clinically relevant threshold probabilities (approximately 0.05 to 0.45), supporting its decision relevance despite comparable discrimination performance.

Conclusion and Implications: Variance-adaptive ensemble modeling enhances predictive stability, interpretability, and clinical utility while maintaining competitive discrimination relative to standard boosting methods. These findings support a shift in predictive modeling toward multidimensional evaluation frameworks that prioritize robustness and decision relevance in heterogeneous clinical populations.

Author

• Graham Zulu, University of Denver
  Presenting Author