Statistical part-of-speech tagging

At the ANLP conference in Austin in February 1988 Kenneth Church of Bell Laboratories presented a program that assigns parts of speech in unrestricted text by multiplying two probabilities estimated on the tagged Brown Corpus of about a million words: how often a word takes a part of speech, and how often that part follows the two before it. It was 95-99% correct, depending on the definition of correct.

The lexical probability is the share of cases in which a word takes a part of speech in the Brown Corpus (I is a pronoun in 5,837 cases out of 5,838); the contextual one is a tag trigram frequency divided by the bigram frequency. Church estimates the uncertainty at about 0.25 bits per tag when the word is known and about 2 bits when only the two neighbouring tags are. A 400-word sample was judged 99.5% correct. The same approach found simple noun phrases, their probabilities estimated from 40,000 words (11,000 phrases). What the record does not claim. Church was not the first to apply n-grams to tagging: the paper itself names Mercer's experiments at IBM in 1982 and the Lancaster group that tagged the LOB corpus with 96.7% success, and says it developed independently of that work. The 95-99% figure is the author's estimate, not a measurement on held-out text.