How Accurately Can Machine Learning Algorithms Predict a Person’s Future?

• In Event: 30802 - Machine Learning and Classical Statistical Approaches: Trade-offs, Integration, and Debate (Co-sponsored by Section on Mathematical Sociology)

Abstract

Predictive models are increasingly used in public services, including in child maltreatment risk assessments in child welfare, pretrial risk assessments in criminal justice, and targeting of homelessness prevention services. The use of these tools raises questions about how accurately a person’s future can be predicted. We investigate the predictability of hundreds of life outcomes for youth and their families, and identify patterns in what types of outcomes are most (and least) predictable. Specifically, we use in-depth survey and observational data collected from a youth’s birth to age 9 in the Future of Families and Child Wellbeing Study to predict 493 outcomes collected at the youth’s age 15. We test dozens of machine learning methods for every outcome to ensure a reasonable estimate of the best possible predictive performance that can be achieved with these data. Our main finding is that most outcomes in our set are not very predictable. In fact, for 25% of the outcomes, our algorithms do not make predictions any better than a null model that predicts everyone’s outcome to be equal to the mean of the training data. However, some outcomes are much more predictable than others, and we identify patterns in what types of outcomes tend to be more predictable. These findings emphasize the need for caution about predictive performance when implementing models that make predictions about individuals’ futures. Furthermore, by systematically assessing the limits of prediction across hundreds of outcomes, this work contributes to the growing literature on the use of machine learning to study model goodness-of-fit with nonlinear models.

Authors

• Emily M. Cantrell, Princeton University

• Pranay Anchuri

• Hanzhang Ren, Stanford University

• Matthew J. Salganik, Princeton University