Human Judgement in Algorithmic Loops

• In Event: Politics, Ethics and AI Technology: The State of the Discipline

Abstract

Algorithmic systems are increasingly involved in the exercise of power by state and private actors, prompting concerns about justice. One safeguard, recognised in data protection law, is the right to demand a human review of the decision. Arguments for such a safeguard tend to focus on certain capacities which are allegedly unique to human decision makers, including: articulating and recognising reasons; providing and evaluating justification; and exercising discretion. Another set of safeguards against algorithmic injustice, proposed in recent work on fairness-aware machine learning, involve constraints on the system to ensure it adheres to certain formalizations of equality.

These two approaches reflect different aspects of justice. A human reviewer is capable of 'Einzelfallgerechtigkeit' (justice in each particular case), of considering each person as an individual. By contrast an algorithmic system will act with unwavering consistency; given the same set of inputs, it will always produce the same output. This reflects a tension within our understanding of justice, between consistency (in Aristotle's formulation, 'treating like cases alike') and discretion - the ability to treat cases differently even if they appear equivalent according to the rules. According to the traditional view, discretion, 'like the hole in a doughnut, does not exist except as an area left open by a surrounding belt of restriction' (Dworkin 1977).

This paper explores different ways in which humans and algorithmic systems can be implicated in both the doughnut itself and the hole within it. Algorithmic decision systems may be more proficient than humans at maximising formally defined derivations of distributive justice across a series of decisions, but humans are seen as essential to procedural justice. While we valorise the human capacity to treat each case individually, we fear our lack of consistency, bias and prejudice. While it does not aim to resolve the dilemma, this paper attempts to partially dissolve it, by reflecting on the relation between these aspects of justice, the extent to which they are zero-sum, and the variety of possible divisions of labour between human and algorithmic decision-making.

Author

• Reuben Binns, University of Oxford