TREC: a shared test for retrieval
On 4-6 November 1992 in Gaithersburg, NIST and DARPA held the first Text REtrieval Conference: 25 groups ran their systems on a shared collection of about two gigabytes of text and shared topics, and the results were scored by one procedure. 92 people attended.
Why it matters
For the first time retrieval systems from universities and companies could be compared on the same data, by the same measures, and on a collection orders of magnitude larger than the usual few megabytes. Judging a pool of the top documents from every run made such a collection possible to evaluate.
The documents went out on two CD-ROMs of about a gigabyte each, training D1 and test D2; topics and judgements by email. Two tasks: ad hoc (a new query against an archive) and routing (a standing query against a stream). Relevance was judged on a pool built from the top 200 documents of each run for each topic. Category B allowed 25 topics and a quarter of the documents. TREC-1 used the same collection as DARPA's TIPSTER project. Donna Harman's overview itself calls these results a baseline. The record does not claim a document count: 740,000 does not occur in the texts read.