NVIDIA tensor cores
NVIDIA added units to the Volta architecture for multiplying small matrices at reduced precision, the exact operation training consists of.
Why it matters
The graphics processor stopped being repurposed hardware and became specialised, while keeping its generality and its ecosystem.
Before Volta, networks ran on units meant for graphics. Tensor cores multiply four-by-four matrices in one clock at mixed precision. The gain was several times on the same workloads. The answer to the TPU was quick and set the shape of the rivalry since: specialised chips against general accelerators with a large ecosystem.