Quantifying Discrimination in Outcome Tests

• In Event: Frontiers in Criminal Justice Policy Methods: Discrimination, Incarceration Dynamics, and Consumer Choice

Abstract

Outcome tests, which compare the success rate of decisions across different groups, are a long-standing and widely used approach to detecting discrimination in lending, hiring, policing, and beyond. The robust outcome test (Gaebler & Goel, 2025) is a simple and statistically sound variation on this basic statistical strategy: In the context of lending, if lenders approve fewer loans to one group, but that group repays its loans more often, it suggests lenders subjected that group to a stricter standard, only lending to its most qualified members. However, the robust outcome test is inconclusive when the same group receives and repays loans more often, and only measures the presence—not magnitude—of discrimination. The threshold test (Simoiu et al., 2017; Pierson et al., 2018) quantifies discrimination by directly estimating decision standards, but is computationally challenging and relies on poorly understood statistical assumptions. It also imposes restrictive data requirements and models decision makers—often implausibly—as perfectly rational, limitations it shares with other popular methods. Here we introduce a modified robust outcome test that quantifies discrimination by transparently incorporating prior beliefs about how predictable the outcome (e.g., loan repayment) is. Our hybrid approach combines the robust outcome test's simplicity with the threshold test's quantification of discrimination. Our method depends transparently on its main statistical assumption—prior beliefs about plausible outcome model AUCs—and accommodates bounded rationality—the varying sensitivity of real-world decision makers to risk. Using our method, we analyze 68 million traffic stops from across the U.S., finding evidence of widespread discrimination—including many instances the robust outcome test would have missed. Across 34 U.S. cities and states, we estimate that 10% to 25% fewer searches of minority drivers would occur under uniform evidentiary standards. This methodological improvement provides researchers and policymakers with a tool for understanding discrimination in lending, policing, and other high-stakes domains that is both informative and statistically transparent.

Author

• Johann Demetrio Gaebler, New York University
  Presenting Author