Research · June 2009

Deep networks on graphics processors

Raina, Madhavan and Ng showed that a model with a hundred million parameters trains on graphics processors in a day instead of several weeks.

Why it matters

The computational obstacle that kept deep networks small disappeared, and that opened the road to 2012.

The speedup was roughly seventyfold against a central processor. The authors state plainly that model scale is from now on limited by the hardware budget rather than by the algorithm. This is the link between CUDA in 2007 and AlexNet in 2012: first it was shown to be possible, then it was shown to win.

Event record

Event date

June 2009

Timeline date

Event date

Verification

Sources gathered automatically · September 17, 2026

Lines

ID

evt-0213

Presented at ICML in June 2009.

Sources

primary Large-scale deep unsupervised learning using graphics processors
ICML 2009 · Published June 2009
archive Large-scale Deep Unsupervised Learning using Graphics Processors (full text)
Semantic Scholar · Published June 2009

Related events

Builds on CUDA becomes generally available
The computation runs on graphics processors through CUDA.
Large-scale deep unsupervised learning using graphics processors
Extends Deep belief networks
The same deep belief networks, two orders of magnitude larger.
Large-scale deep unsupervised learning using graphics processors
Related AlexNet at ImageNet
The recipe AlexNet was trained by three years later.

Records that link to this one

Related The Google network that found cats
The same scaling recipe, on thousands of machines instead of graphics processors.
Related TensorFlow
A tool that makes distributed training on accelerators available outside large companies.
Related Theano
An expression compiler that moves the computation onto a graphics processor by itself.
Related Baidu's Institute of Deep Learning
The bet is on the same recipe: deep networks on graphics processors.
Related NVIDIA tensor cores
The same line of accelerating training, taken to dedicated matrix units.
Related NVIDIA becomes the most valuable company in the world
The company whose gaming cards accidentally became the tool for training networks ended up at the top of the market.