MUC-6: named entity recognition as a task of its own
On 6-8 November 1995 in Columbia, Maryland, the Sixth Message Understanding Conference, funded by DARPA, reviewed an evaluation that for the first time measured named entity recognition separately: names of people, organisations and places, dates, sums of money and percentages. The best of 20 systems from fifteen sites, SRA's, had an F-measure of 96.42; two human annotators scored against each other had 96.68.
Why it matters
Extracting facts from text was broken into smaller tasks that could be measured separately and applied regardless of subject, and the first of them, finding and classifying names, proved solvable almost at the human level at once. Named entity recognition has existed as a language processing task of its own since.
The tasks were defined by a meeting DARPA convened in December 1993; a dry run took place in April 1995 (two systems of nine exceeded 90%). The training and test sets were 100 Wall Street Journal articles each; 30 of them were the named entity test. Markup used the SGML tags ENAMEX, TIMEX and NUMEX. Half of the systems exceeded 90% F-measure. On the same text in capitals only, SRA's system reached 85% recall and 89% precision, nearly 10 points lower. Besides named entities, MUC-6 measured coreference, template elements and a scenario on changes in company management. What the record does not claim. The test is small and uniform: 30 financial news stories without a single time expression. Parity with people holds for this sample only.