Benchmark · November 4 – 6, 1992

TREC: a shared test for retrieval

On 4-6 November 1992 in Gaithersburg, NIST and DARPA held the first Text REtrieval Conference: 25 groups ran their systems on a shared collection of about two gigabytes of text and shared topics, and the results were scored by one procedure. 92 people attended.

Why it matters

For the first time retrieval systems from universities and companies could be compared on the same data, by the same measures, and on a collection orders of magnitude larger than the usual few megabytes. Judging a pool of the top documents from every run made such a collection possible to evaluate.

The documents went out on two CD-ROMs of about a gigabyte each, training D1 and test D2; topics and judgements by email. Two tasks: ad hoc (a new query against an archive) and routing (a standing query against a stream). Relevance was judged on a pool built from the top 200 documents of each run for each topic. Category B allowed 25 topics and a quarter of the documents. TREC-1 used the same collection as DARPA's TIPSTER project. Donna Harman's overview itself calls these results a baseline. The record does not claim a document count: 740,000 does not occur in the texts read.

Event record

Event date

November 4, 1992 – November 6, 1992

Timeline date

Event date

Verification

Sources gathered automatically · September 24, 2026

Lines

ID

evt-0693

Sources

primary The First Text REtrieval Conference (TREC-1), NIST Special Publication 500-207: preface
National Institute of Standards and Technology · Published February 20, 1993
primary Overview of the First Text REtrieval Conference (TREC-1)
National Institute of Standards and Technology · Published 1993

Related events

Related The document as a vector of terms
The TREC-1 overview gives as its reason that evaluation on small collections such as Cranfield (1,400 documents) does not show how systems behave on large full text; Chris Buckley of Cornell is on the programme committee in the preface.
Overview of the First Text REtrieval Conference (TREC-1)

Records that link to this one

Builds on BM25
The paper traces how City University's Okapi system changed from TREC-1 to TREC-3 and gives recomputed results for its TREC-1 run.
Okapi at TREC-3
Builds on Searching recorded speech: the SDR track
The spoken document retrieval track was part of TREC: the same queries and measures, but the documents were automatic transcripts of news broadcasts.
The TREC Spoken Document Retrieval Track: A Success Story (John S. Garofolo, Cedric G. P. Auzanne, Ellen M. Voorhees), RIAO-2000 Content-Based Multimedia Information Access, Paris, 12-14 April 2000
Builds on The TREC video track: a shared test for video search
The video track began inside TREC 2001 and carried its method, shared data, queries and measures, from text over to video.
The TREC-2001 Video Track Report (Alan F. Smeaton, Paul Over, R. Taban), in NIST Special Publication 500-250, The Tenth Text REtrieval Conference (TREC 2001); report dated 18 April 2002
Related MUC-6: named entity recognition as a task of its own
The same model DARPA and NIST applied to retrieval in TREC: participants receive shared data, the run takes place before the conference, and the conference discusses the results.