Benchmarks As Proxies: Monitoring Racial Bias in Police Traffic Enforcement with Noisy Signals of Driver Behavior

• In Event: Discrimination, Fairness, and Algorithmic Bias in Machine Learning Applications

Abstract

Decades of work has sought to assess whether police traffic enforcement is discriminatory by comparing the racial composition of those who are stopped to some external benchmark, such as the composition of local residents in a police jurisdiction. Such "benchmark analyses" have been criticized by statisticians and law-enforcement officials alike for yielding misleading results that fail to accurately represent the population at risk of police traffic stops: namely, individuals actually engaged in dangerous driving behavior.

In this paper, I provide a causal framework that formalizes these concerns and clarifies what benchmark analyses can and cannot identify. I show that any valid benchmark must act as a race-neutral proxy for the underlying behavior that justifies a stop, and I derive the conditions under which commonly used estimators in practice recover a well-defined, but hitherto unspecified, causal estimand: the relative likelihood of being stopped among drivers who are genuinely subject to enforcement. In particular, I demonstrate that standard approaches implicitly assume the absence of false charges, by both officers and the proxy—assumptions that are not only strong, but empirically testable and frequently violated in practice.

Building on this framework, I develop a partial identification strategy that treats benchmark datasets as imperfect measurements of underlying behavior. Leveraging ideas from the negative control outcomes literature, I construct bounds on the causal effect of race on the probability of being stopped among drivers who are truly at risk of enforcement as a function of proxy quality. I demonstrate the generality of this approach by applying it with three benchmark datasets: traffic cameras, collisions, and sobriety checkpoints, addressing the specific elements of the framework that are unique to each. We further address several complications which arise due to the real world realities of policing data, including: missing or mismeasured race, high-dimensional driving behavior, and the substitution effects of camera and officer enforcement. I then use this framework to evaluate bias in mvoing violation and speeding stops in the jurisdictions of the Chicago Police Department and the Texas Highway Patrol.

Author

• Kai Reis Darius Cooper, University of Pennsylvania
  Presenting Author