Three fairness conditions that do not meet
On 19 September 2016 Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan submitted a paper that formalised three fairness conditions for risk scores and proved that no assignment of scores satisfies all three at once, except in two degenerate cases: when prediction is perfect, or when the groups have equal base rates. The paper names the argument over COMPAS as its occasion.
Why it matters
Until then each side of an argument about algorithmic bias could hold that the other had made a mistake. After this theorem it was known that neither had: the incompatibility sits in the definitions themselves and does not depend on how a score is computed. Choosing a fairness criterion stopped being an engineering question and became something that has to be said out loud.
The three conditions as the paper states them: (A) calibration within groups; (B) balance for the negative class; (C) balance for the positive class. Theorem 1.1: if a risk assignment satisfies (A), (B) and (C), then the instance must either allow perfect prediction, with every probability 0 or 1, or have equal base rates. The paper also proves an approximate version: anything that satisfies the conditions to within an error must approximately look like one of those two cases. The record does not claim that (B) and (C) are equality of false positive and false negative rates. The paper itself notes that they are distinct from those: they compare the average score assigned to members of a class rather than a rate of error, and generalise them. The version the record is dated to was the one read: v1 of 19 September 2016, twenty-two pages. A second version followed on 17 November that year. The affiliations on page one are Cornell, Harvard and Cornell.