A blind evaluation for face recognition
In September 1996 the third and final FERET evaluation was run. The protocol required matching a set of 3323 images against a set of 3816 — approximately 12.6 million matches — over a database of 14,126 images of 1199 individuals, gathered independently of the algorithm developers. Results were reported separately for images taken on the same day, on different days, and over a year apart.
Why it matters
Face recognition acquired a number that was not reported by the author of the method. Until then each group measured itself on its own sample by its own rules and there was nothing to compare two methods with; after FERET, testing against a withheld set became a condition, and every large computer vision contest since has repeated the arrangement.
The database was collected over 15 sessions between August 1993 and July 1996: 1564 sets of images, 14,126 images in total of 1199 individuals, including 365 duplicate sets — second sittings of a person already in the database, usually on a different day. For some individuals over two years passed between first and last sitting, and that is what first made it possible to study how a person's appearance changes over a year. The earlier rounds: August 1994 with a gallery of 316 individuals, March 1995 with a gallery of 817. The 1996 evaluation was run in two versions — for partially automatic algorithms, given the coordinates of the eye centres, and for fully automatic ones, given only the images. NIST states plainly what came before: a large number of papers reported outstanding results, usually above 95 percent correct, on databases of usually fewer than 50 individuals. Total funding for the programme was in excess of 6.5 million dollars. The record gives no specific drop in accuracy for images a year apart. The methodology and results paper appeared in IEEE TPAMI in October 2000, but its full text could not be read, so the figures here are only those NIST's own page states.