Building a human-in-the-loop AI system for tracking fatal police violence in the US

• In Event: Police Violence 10 Years After Ferguson: What Do We Know and Where Do We Go Next?

Abstract

Mapping Police Violence (MPV) is a public database that tracks and documents instances of police killings, as reported in local news media. Over time, the methodology has evolved to incorporate automations while keeping human review as an integral component. Presently, MPV employs automated keyword searches and filtering of news articles, which are then manually reviewed by humans to categorize potential police violence incidents. The articles contain graphic depictions of violence, creating a mental health burden for the human reviewers, which adds to the benefit of increased automation in this process. Recent advances in large language models (LLMs) present more effective tools to automate content extraction and semantic processing of text. We test the efficacy of fine-tuning a pre-trained LLM to predict whether a news article describes an incident that matches MPV’s definition of police violence, using previously human-labeled data. We examine LLM model performance and its effect on human reviewer workload. Our aim is not only to equip platforms like MPV with AI tools for comprehensive tracking of these incidents, facilitating near-live incident detection, but also to contribute insights to other organizations navigating the complex landscape of criminal justice data analysis.

Authors

• Maria Gabriela Ayala, Campaign Zero

• Andrew D. Zaharia, Campaign Zero

• Abdul Nasser Rad, Campaign Zero