Training priced in kilowatt-hours
On 5 June 2019 the University of Massachusetts Amherst put the energy and emissions of training language models into physical units. A full architecture search for a transformer was estimated at 274,120 hours on eight P100 accelerators, 656,347 kWh and 626,155 pounds of CO2 equivalent - roughly five times what a car emits across its whole life including fuel.
Why it matters
Until then the cost of computation was measured in hours and dollars, and the electricity and the emissions were not counted at all. The paper supplied a method: poll the accelerator driver, multiply by the data centre's efficiency coefficient and by the average carbon intensity of the grid. After it, energy could be demanded as a number inside the publication itself, and the later work in this line cites this calculation by name.
The data centre efficiency coefficient used is 1.58, the 2018 global average; the conversion to emissions runs at 0.954 pounds of CO2 equivalent per kilowatt-hour, the United States grid average. The paper's comparison table: one passenger flying New York to San Francisco, 1,984 pounds; a year of an American life, 36,156; a car over its whole life including fuel, 126,000. The transformer itself costs 192 pounds, the same transformer with architecture search 626,155. Cloud rental for such a search is put at 942,973 to 3,201,722 dollars. What the record does not claim. The 274,120 hours are not a run the authors performed but a conversion onto eight P100s of the 979 million training steps that the authors of the Evolved Transformer reported; the authors metered smaller models themselves. The paper counts in pounds, and 284 metric tonnes is a conversion, not its figure.