Three eras of compute: 21, 6 and 10 months
On 11 February 2022 Epoch AI assembled the training compute of 123 milestone systems going back to 1952 and cut the history into three eras. Before 2010 the amount doubled every 21.3 months; from 2010 every 5.7 months; and from September 2015 a separate series of the largest models doubles every 9.9 months, starting one to two orders of magnitude above the earlier trend.
Why it matters
Until then the growth rate of compute was quoted as a single number for the whole period, and different papers disagreed by a factor of three. Splitting it into three series explained the disagreement: different authors were measuring different eras with one instrument. This partition became the thing any later estimate of future power demand is set against.
Confidence intervals: 17.0 to 29.3 months before 2010 across 19 systems, 4.3 to 9.0 across 72 systems for 2010 to 2022, and 7.7 to 17.1 across the 16 largest models. The figures differ between parts of the paper, which is worth knowing in advance. The abstract says roughly 20 months for the pre-2010 rate, the table says 21.3, the conclusion says 18. Figure 1 is captioned with 118 systems, the abstract says 123. The abstract puts the gap opened by large models at 10 to 100 times, the conclusion at two to three orders of magnitude. What the record does not claim. There is nothing about energy here: the word energy occurs once in the paper, in a bibliography entry. This measures demand for computation, not for power, and Moore's law is named as a yardstick for the rate, not as a measure of efficiency per watt.