Face recognition measured on 37,437 people
In March 2003 NIST published the FRVT 2002 report. In July and August 2002 ten commercial face recognition systems had been tested under the organisers’ supervision on 121,589 photographs of 37,437 people from the US State Department’s visa archive. On comparable tests the error had fallen by about half since FRVT 2000.
Why it matters
Face recognition was measured on a database of real-world size: in FERET and FRVT 2000 the largest experiment had 1196 people. The report showed what accuracy depends on — time between photographs, lighting, a person’s sex and age, the size of the database — and gave agencies a way to ask whether a system suits a particular use rather than whether it "works" at all.
NISTIR 6965: P. Jonathon Phillips (DARPA, NIST), Patrick Grother, Ross Micheals, Elham Tabassi (NIST), Duane Blackburn (a Defense Department programme), Mike Bone (NAVSEA Crane). The main test took 121,589 images of 37,437 individuals, a subset of a collection of about 6.8 million photographs of about 6.1 million applicants for Mexican non-immigrant visas, 1996-2002. The participants included Cognitec, Identix, Eyematic, Imagis, Viisage, VisionSphere, C-VIS and Dream Mirh. The report’s conclusions: indoor accuracy had improved substantially since FRVT 2000; it falls roughly linearly with the time between photographs; men are easier to recognise than women, and younger people harder than older; outdoor recognition "needs improvement"; in search against a database accuracy falls linearly with the logarithm of its size; three-dimensional morphable models help with non-frontal faces, while video gave no advantage over stills. The request behind this record called FRVT 2002 the first independent measurement after FERET; the report itself names FRVT 2000 as a predecessor as well. The record gives no verification rates: they appear only in the report’s graphs, not in its text.