Benchmark · April 12, 2000
Placed by the contemporary primary publication. The exact event date is not known; its documented interval appears below.
Searching recorded speech: the SDR track
In 1997-1999 NIST ran a spoken document retrieval track at TREC: systems recognised news broadcasts and searched the transcripts for queries. The collection grew from 50 hours in 1997 to 557 hours and 21,754 stories in 1999. NIST presented the summary at RIAO 2000 in Paris on 12-14 April 2000.
Why it matters
For the first time it was measured how much a recogniser's errors hurt search in audio: the relation turned out near-linear and gentle, so an automatic transcript with one word in four or five wrong was already good enough to search hundreds of hours of broadcasts. The speech recognition and retrieval communities worked on one collection for the first time.
TREC-6 (1997): 50 hours, 1,451 stories averaging 276 words, a known-item task; IBM's baseline recogniser got 50% of words wrong, 40% on average per story. Speech recognition test corpora before it had been under three hours. TREC-7 (1998): 87 hours, 2,866 stories, ordinary ad hoc retrieval. TREC-8 (1999): 557 hours of ABC, CNN, Public Radio International and Voice of America news from February-June 1998; NIST's baseline recognisers had 27.5% and 26.7% word error. There were 13, 11 and 10 participating groups. In TREC-8 systems also searched without marked story boundaries for the first time.
What this record does not assert: the paper does not say that searching recognised text equals searching a human transcript. Its caveat is that question answering and spoken queries, where one wrong word ruins the result, had not yet been tried.
Event record
- Event date
- November 1997 – April 14, 2000
- Timeline date
- Primary publication date
- Verification
- Sources gathered automatically · September 25, 2026
- Lines
- ID
- evt-0825
The track ran at TREC-6, TREC-7 and TREC-8, each in November, from November 1997; the interval ends with NIST's summary at RIAO 2000 in Paris on 12-14 April 2000.
Related events
- Builds on TREC: a shared test for retrieval
The spoken document retrieval track was part of TREC: the same queries and measures, but the documents were automatic transcripts of news broadcasts.
The TREC Spoken Document Retrieval Track: A Success Story (John S. Garofolo, Cedric G. P. Auzanne, Ellen M. Voorhees), RIAO-2000 Content-Based Multimedia Information Access, Paris, 12-14 April 2000 - Related Sphinx
NIST first took SPHINX-III as its baseline recogniser, but it ran at nearly 200 times real time, and a faster one had to be found for 87 hours.
The TREC Spoken Document Retrieval Track: A Success Story (John S. Garofolo, Cedric G. P. Auzanne, Ellen M. Voorhees), RIAO-2000 Content-Based Multimedia Information Access, Paris, 12-14 April 2000 - Related The TREC video track: a shared test for video search
The 2001 video track report names spoken documents as one of TREC's few excursions beyond text before video.
The TREC-2001 Video Track Report (Alan F. Smeaton, Paul Over, R. Taban), in NIST Special Publication 500-250, The Tenth Text REtrieval Conference (TREC 2001); report dated 18 April 2002