NVIDIA A100
A new accelerator generation gave four times the training performance and the ability to split one device between workloads.
Why it matters
The hardware the GPT-3 generation of models was trained on, and the unit frontier research has been costed in ever since.
The number of accelerators became the usual measure of a project's scale: models are described not only by parameters but by thousands of devices and months of training. That entrenched the inequality of access the Microsoft and OpenAI record describes. In late 2022 export of these accelerators to China was restricted, and hardware became a matter of foreign policy.