Resource Management: a shared speech recognition test
From March 1987 NIST, for DARPA, ran tests of speech recognition systems on a shared Resource Management database: read sentences for a naval resource management task, a vocabulary of about a thousand words, over 21,000 recordings from 160 talkers. The database was described at ICASSP-88 in April 1988; by February 1989 four tests had been held.
Why it matters
Laboratories funded by DARPA began recognising the same sentences from the same talkers and counting errors with one program. From then on a number on a shared test, not a demonstration, answered whose system was better.
The database's authors came from BBN, Texas Instruments, SRI International and the National Bureau of Standards, later NIST. It is split into training and test parts and serves speaker-dependent, speaker-independent and speaker-adaptive systems. NIST wrote a single scoring procedure and software for it; tests were held in March and October 1987, June 1988 and February 1989, the early ones with BBN, Carnegie Mellon, MIT Lincoln Laboratory and SRI. Carnegie Mellon's Sphinx, on the official 1987 sentences (150 sentences from 15 talkers), recognised 96 percent of words with the word-pair grammar (perplexity 60) and 82 without a grammar (perplexity 991); on the 1988 and 1989 tests without a grammar, 78.1 and 76.4 percent. What the record does not claim. The full ICASSP papers were not read, so the record gives no results for other systems and does not say who won any test.