Lipreading cuts a recogniser's errors

In May 1988 Eric Petajan and colleagues at AT&T Bell Laboratories, with Michael Brooke of the University of Bath, showed at CHI '88 a system that lipreads from video of the mouth and combines the result with an AT&T ASR1000 acoustic recogniser. On the English alphabet the combination cut its errors by a quarter to two thirds.

A camera took the face; special hardware at 60 frames a second produced binary images, and the mouth was found by tracking the nostrils. Word templates were vector-quantized with a codebook of just under 256 mouth images, 60 bytes of visual data a second, and compared by dynamic time warping. Four speakers, isolated words, tuned to each speaker; the vocabularies were the alphabet and the digits zero to nine. Twenty-nine heuristic rules did the combining: for one speaker the ASR1000's letter errors fell by nearly half with quantization and nearly two thirds without, for another by about a quarter and a third. The 1984 dissertation is described by its abstract in the University of Illinois repository: isolated words, one speaker, a camera and a minicomputer with custom video hardware; a separate commercial device recognises the sound, and combining the visual and acoustic candidates gives an accuracy that "greatly exceeds" the acoustic one. The abstract gives no figures, and the full text is open only to the university. In a 1996 overview Petajan and Graf write that Petajan completed the first speechreading system in 1984: four parameters of a binary mouth image found by tracking the nostrils, a vocabulary of digits, letters and a few words. Being first is the author's own claim, and the record has no second source for it; the Globecom 1984 and CVPR '85 papers were not read. No percentages from the 1988 tables are given: in the available copy they do not read reliably.