A Simple, Statistically Robust Test of Discrimination

• In Event: Saturday Poster Luncheon

Abstract

In observational studies of discrimination, the most common statistical approaches consider either the rate at which decisions are made or the success rate of those decisions. For example, in the context of lending, if one group of borrowers repays their loans more than other groups, that suggests the bank held them to a more stringent standard, only lending to the most creditworthy applicants from that group. Such "outcome tests" are one of the most common empirical approaches for detecting double standards in lending, hiring, criminal justice, and other domains. Despite their simplicity, outcome tests suffer from well-known statistical issues and often miss discrimination or even mistake discrimination for favorable treatment. "Benchmark tests"—where, for example, researchers directly compare lending rates across different groups—are the most common alternative, but suffer from their own deep limitations. These statistical limitations have attracted considerable attention, prompting several attempts to place outcome tests, in particular, on firmer statistical footing. Despite the fallibility of the benchmark and outcome tests individually, here we prove a surprisingly strong statistical guarantee: under a common non-parametric assumption, at least one of the two tests must be correct; consequently, when both tests agree, they are guaranteed to yield correct conclusions. We present empirical evidence that the underlying assumption holds approximately in several important domains, including lending, education, and criminal justice—and that our hybrid test is robust to the moderate violations of the assumption that we observe in practice. Drawing on data from 2.8 million police stops across California, we find that the standard outcome test indicates discrimination against White individuals in a third of the state's large law enforcement agencies---defying expectations and calling into question the test's findings of discrimination against racial minorities in many of the remaining agencies. In contrast, the robust outcome test uncovers consistent evidence of police discrimination against racial minorities across California. Our simple methodological improvement promises to strengthen the quality of evidence in statistical studies of discrimination.

Author

• Johann Demetrio Gaebler, New York University
  Presenting Author