22  Hardware for Training and Inference

The reader knows one GPU from Section 13.2 and Section 13.1: a roofline, a memory hierarchy, a prefill that is compute-bound and a decode that is not. A released model is trained on thousands of chips and served from racks, and the chips, the kernels that run on them, the links between them and the power they draw decide what a token costs.