Towards Individualized Risk Predictions with LLMs

• In Event: Frontiers in Criminal Justice Policy Methods: Discrimination, Incarceration Dynamics, and Consumer Choice

Abstract

Empirical audits of police practices increasingly inform legislation and litigation. However, important contextual information is often confined to unstructured text narratives, hindering auditors who generally base statistical analyses on more limited structured post-stop forms. Using data from over 300,000 California police stops collected under the Racial and Identity Profiling Act (RIPA), we use LLMs to incorporate information from stop narratives into individualized estimates of contraband risk. To ensure risk models only use information available to officers prior to a search, we first use an LLM to redact post-stop and post-search outcomes from the narratives. We then structure narrative information using both state-of-the-art text embeddings, and prompt LLMs to answer a set of questions about narrative contents to generate a separate interpretable feature set. Using these representations to train risk models predicting contraband recovery, we find that narrative information materially improves predictive accuracy, and individualizes risk estimates by capturing contextual information not present in structured administrative data. As a result, we find that narrative-informed risk estimates alter common audit quantities like the prevalence of low-risk searches, suggesting that prior audits based solely on structured data may have overlooked important contextual information. Our work demonstrates how LLMs can be used to obtain more accurate and individualized risk estimates, in turn enabling improved oversight and audits of police practices at scale.

Author

• Madison Elisse Coots, Harvard University
  Presenting Author