Adaptive mixtures of local experts

In the February 1991 issue of Neural Computation (volume 3, issue 1, pages 79-87) Robert Jacobs and Michael Jordan of MIT and Steven Nowlan and Geoffrey Hinton of the University of Toronto described a system of several expert networks and a gating network that decides, for each training case, which expert takes it. On a task of telling apart four vowels from 75 speakers, mixtures of 4 or 8 simple linear experts reached the error criterion in about half the epochs that backpropagation networks needed, at the same 90 per cent on test.

The paper presents the procedure either as a modular version of a multilayer supervised network or as an associative version of competitive learning. The data are the first two formants of the vowels [i], [I], [a] and [A] in an hVd context from 75 speakers (men, women and children) in Peterson and Barney 1952; training on the first 50 speakers, test on the remaining 25. Table 1, averaged over 25 simulations: all four systems reach 88 per cent on training and 90 per cent on test; the average number of epochs to criterion is 1,124 for 4 experts, 1,083 for 8, 2,209 for backpropagation with 6 hidden units and 2,435 with 12. In one run with 4 experts one expert made no contribution at all, its mixing proportion effectively zero. The 1991 gate mixes the experts' outputs in proportions; computing only a few experts per example is what the 2017 paper added. Read from a scan on Hinton's Toronto page, page by page as images, because the scan has no text layer. What the record does not claim: a gain in accuracy (on test all four systems are equal), or the month of March, which was expected before reading.