A Theory of Adaptive Pattern Classifiers
Amari described training a classifier by stochastic gradient descent and proved convergence even for non-separable distributions, provided the mean risk has a single minimum.
Why it matters
Gradient descent on noisy estimates acquired a theory just as the field was turning away from neural networks.
The paper generalises the rule to multicategory classifiers, to piecewise-linear discriminant functions (the maximum of several linear ones) and to machines with fixed nonlinear features; it has no hidden layers trained by gradient. Without the single-minimum condition the weights converge only to one of the local minima. It appeared two years before Perceptrons and had almost no effect on the English-language discussion of the time. Amari later developed information geometry and the natural gradient.