Rectified linear units
Nair and Hinton showed that rectified linear units, a simple nonlinearity that zeroes negative values, give better features than binary stochastic units in restricted Boltzmann machines.
Why it matters
A simple and cheap substitution that later became the basis of deep networks: the advantage of the rectifier over the sigmoid and the hyperbolic tangent in deep networks without pre-training was shown by Glorot, Bordes and Bengio in 2011.
The paper compares rectified units with binary stochastic units in restricted Boltzmann machines, on NORB object recognition and Labeled Faces in the Wild face verification, and does not compare them with a sigmoid in networks trained by backpropagation; it says nothing about training speed. It shows that such units preserve information about relative intensities as the signal travels through layers of feature detectors. The argument about the vanishing gradient belongs to the later paper by Glorot, Bordes and Bengio (2011): in a rectifier network the gradient flows well along the active paths, without the vanishing effect of sigmoid or tanh units, and computation is cheaper because no exponential is needed.