Individual Submission Summary
Share...

Direct link:

Predicting Recidivism Using Lasso Regression Models

Thu, Nov 17, 11:00am to 12:20pm, L404, Lobby Level

Abstract

Efforts to improve recidivism prediction have involved techniques using both inferential statistics and machine learning. The latter category includes lasso, an estimator of coefficients in a regression model that incorporates a penalty term to address over-fitting. Because lasso selects relatively few out of many variables that may affect an outcome, it is useful for both prediction and model selection.

To develop my predictions for the NIJ Recidivism Forecasting Challenge, I used lasso to fit logit and probit models on subgroups of training data categorized by race and gender. I added several newly-constructed variables to the data, including proxy measures relating to age while incarcerated, plea bargaining, and missingness. Lasso included these constructs among the variables selected for their predictive power, suggesting that versions of these constructs could prove useful in predicting recidivism with other methods.

However, none of the lasso models achieved high accuracy for out-of-sample predictions, possibly due to over-fitting or problems relating to data sparseness. Nevertheless, when compared to “black box” methods, the simplicity and interpretability of lasso models may provide an acceptable trade-off for slightly lower accuracy in recidivism forecasting, particularly in cases where inherent data limitations may not be surmountable with more complex and resource-intensive methods.

Author