Reverse-mode automatic differentiation
Seppo Linnainmaa described how to compute the derivatives of a composite function in one backward pass over the computation graph.
Why it matters
The algorithm that makes training deep networks possible appeared in a problem about accumulated rounding error.
Linnainmaa wanted to estimate how the rounding errors of individual operations affect the final result. The solution is to expand the accumulated error as a Taylor series and traverse the graph in reverse. The cost of that pass is proportional to the forward computation regardless of the number of parameters, and that property is what makes training large models feasible.