A contest where everyone is scored the same way
On 11 April 2005 in Southampton the results of the first PASCAL Visual Object Classes challenge were announced. Four classes — motorbikes, bicycles, people and cars; two tasks — say whether an object is in the image, and put a box around it. Twelve teams entered and six presented at the workshop.
Why it matters
Evaluation in computer vision became an annual event with rules announced in advance, rather than a section in somebody's paper. Detection acquired an unambiguous criterion of correctness, and the contests of the following decade inherited that criterion unchanged.
The rule by which a detection counts as correct: the area of overlap between the predicted box and the ground truth box must exceed 50 percent of their union. Lower-ranked repeat detections of the same object count as false positives, so every participant had to add a way of arbitrating between them. Classification was scored off the ROC curve by two measures at once — equal error rate and area under curve — so as to favour neither regime. The first image set: 684 images and 812 objects across training and validation, 689 images and 833 objects in the first test set, counted per class. A second test set was gathered separately through Google Images so as not to come from the same distribution: 1072 images and 1713 objects. Images were contributed by TU Darmstadt, the University of Illinois, Caltech, MIT, Graz University of Technology and INRIA. The record does not claim "nine teams" and does not claim "1578 images and 2209 objects". The challenge report names twelve teams of which six presented; neither of the two dataset tables yields the totals quoted elsewhere. The challenge page names the organisers as Luc van Gool of Zurich, Chris Williams of Edinburgh and Andrew Zisserman of Oxford; Mark Everingham of Oxford runs the challenge as technical contributor and is the first author of the report.