NVIDIA A100
NVIDIA announced a new accelerator generation for which it claimed up to 20 times the AI performance of its predecessors, and the ability to split one device into as many as seven GPUs.
Why it matters
The hardware the GPT-3 generation of models was trained on, and the unit frontier research has been costed in ever since.
NVIDIA's own figures depend on the workload. The press release of 14 May 2020 says up to 20x over its predecessors; that best case is TF32 with structural sparsity. For mixed-precision FP16/FP32 deep learning its developer article gives 2.5x the performance of V100, 5x with sparsity. The number of accelerators became the usual measure of a project's scale: models are described not only by parameters but by thousands of devices and months of training. That entrenched the inequality of access the Microsoft and OpenAI record describes. In late 2022 export of these accelerators to China was restricted, and hardware became a matter of foreign policy.