Theano
The Montreal group released a compiler for mathematical expressions that computes derivatives itself and moves work onto the graphics card itself.
Why it matters
Automatic differentiation became infrastructure: a researcher described a model instead of writing gradients by hand.
Before Theano everyone derived and coded their own derivatives, and an error in the gradient was a common reason a network failed to train. Theano built a symbolic graph, differentiated it and compiled it for the available hardware. That scheme is the basis of TensorFlow. Development stopped in 2017, when more broadly supported systems appeared.