Face recognition accuracy by sex, race and age
In the December 2012 issue of IEEE Transactions on Information Forensics and Security, Brendan Klare and colleagues measured six face recognition systems on 102,942 police photographs split into eight demographic cohorts. The most accurate of the three commercial systems matched 88.7 per cent of black subjects against 94.4 per cent of white ones, 89.5 per cent of women against 94.4 per cent of men, and 91.7 per cent of 18-to-30-year-olds against 94.6 per cent of 30-to-50-year-olds.
Why it matters
Here the gap is set out as a table of numbers by cohort rather than as an observation: one metric, one gallery, six systems. That gave later audits a shape in which a claim of bias can be checked, and it is in exactly that shape that the audit of commercial systems returned six years later.
The metric is the true accept rate at a fixed false accept rate of 0.1 per cent. The systems: three commercial (Cognitec FaceVACS 8.2, PittPatt 5.2.2 and Neurotechnology MegaMatcher 3.1, anonymised in the results as COTS-A, B and C), two non-trainable (local binary patterns and Gabor) and one trainable (4SF). The gallery is mug shots from the Pinellas County Sheriff's Office, two images per subject, with training and test sets disjoint. The cohorts are sex, race (Black, White, Hispanic) and age (18-30, 30-50, 50-70). A second result: training exclusively on a cohort raises accuracy on it, by nearly 2 per cent for the Black cohort and 1.5 per cent for the White one against balanced training. It did not work for women: 4SF trained only on females scored 72.8 per cent on females against 73.4 per cent for the balanced version, and the paper concludes from this that the female cohort is inherently harder. The record does not claim this was the first such measurement. The paper cites earlier work on the gap between White and East Asian faces and an earlier finding that women and younger subjects are harder to recognise; what it calls its own novelty is the question of whether an algorithm can be trained to exploit a demographic cohort. The database held too few Asian subjects for that cohort to be measured at all. What was read is the accepted manuscript, not the journal version: it runs six pages and is headed To appear, where the journal article occupies pages 1789-1801. The numbers come from the manuscript and the date from the DOI registration record. The affiliations, from the footnote on page one: Klare at Noblis, Burge and Klontz at The MITRE Corporation, Vorder Bruegge at the Science and Technology Branch of the FBI, and Jain at the Department of Computer Science and Engineering of Michigan State University and at Korea University. The incoming report credited the work to the University of Michigan, a different institution.