29  Efficient Inference

The reader has a compressed model from Chapter 11 and, from Section 11.3 and Section 13.2, a bound on how fast one GPU can decode it; the plain generation loop reaches a small fraction of that bound. This chapter takes one request, or a handful, from the eager loop to the bound and then past its per-token form: the kernels that read the weights and the cache at full bandwidth, the reduction orders that make outputs reproducible, the verification trick that checks several tokens per weight read, and the cache compressions that a trained model tolerates. It ends on a laptop, where the same bound holds with a tenth of the bandwidth.