11  Transformers

Chapter 10 studied attention the way one studies an organ in isolation; this chapter builds the organism. A transformer wraps the attention layer in a block: attention moves information between positions, a position-wise feed-forward network transforms it in place, and residual connections with normalization keep the signal usable when blocks are stacked dozens or hundreds deep. That block, stacked and wired in one of three ways, is the entire architecture. It has carried machine translation, then language modeling, then images, audio, and protein structure, with so little structural change since 2017 that the differences between today’s frontier models fit in a small table — one this chapter ends by writing down and instantiating.

Our method follows from that observation. Early in the chapter we build a single configurable GPT class: normalization type and placement, activation, positional scheme, and the attention and feed-forward modules are all constructor arguments. Every subsequent section changes one component, re-runs the model, and measures the difference, so that by the close of the chapter the recipe table’s rows are not descriptions but constructor calls. Every training run here finishes in minutes on one GPU; where the small scale genuinely changes the story — and it sometimes does — the prose says so, and stated cost anchors connect our runs to what production training actually spends.

The sections, in order. Section 11.1 dissects the block: where normalization goes (we measure signal propagation through 32 randomly initialized blocks and watch the post-norm arrangement starve its own attention gradients), why RMSNorm and QK-norm displaced plain LayerNorm, and what gated feed-forward networks buy, ending with the configurable block. Section 11.2 assembles the blocks into a language model, trains the modern configuration on real text, samples from it, and then loads the published GPT-2 weights into the same class — the 2019 model is one setting of the flags, and our hand-built code generates with it. Section 11.3 turns to generation, where the economics invert: decoding is bound by memory traffic, not arithmetic, and the key–value cache, grouped-query attention, low-rank cache compression, and sliding windows with attention sinks are the escalating responses; we implement and measure each. Section 11.4 steps back to the taxonomy — encoder-only, decoder-only, encoder–decoder as three wirings of the same block — and treats cross-attention fully, including its most general form: a small learned latent array that reads an arbitrarily long input, the idea behind the Perceiver family and the input adapters of current multimodal models. Section 11.5 feeds the architecture images by cutting them into patch tokens, and lets the result teach the lesson: at our scale a matched convolutional network wins, because the transformer must learn the locality a convolution assumes. Section 11.6 decouples parameters from computation with mixture-of-experts layers, where the interesting problem is not the idea but keeping the router from collapsing — we train the failure and both standard repairs. Section 11.7 closes with the arithmetic of scale: counting FLOPs, a miniature scaling study whose bend reproduces the data-starvation effect behind compute-optimal training, and the modern recipe table.

The history is short enough to tell in full. The 2017 transformer was an encoder–decoder for translation, normalized after each sublayer, with sinusoidal positions and a plain ReLU network. What survived is the block; nearly every choice around it changed, and the changes concentrate on three axes — stability at depth (pre-norm, RMSNorm, QK-norm), the memory bill of generation (grouped queries, cache compression, windows), and capacity (experts). Convergent evolution across independent labs has made new open models look more alike each year, which is what makes the architecture teachable as one object with options rather than a zoo.

What this chapter is not. Tokenization was settled in Chapter 8 and optimizers in Chapter 9; both are used here without comment. Pretraining corpora, instruction tuning, and everything downstream of the base model belong to the Language Models part, as does BERT, whose encoder this chapter’s taxonomy locates but does not train. Vision applications beyond the ViT itself are the Image Models part’s. Kernels, parallelism, quantization, and serving systems belong to the Computational Performance chapter; here they appear only as black-box fused-attention calls, whose kernels are that chapter’s subject, and one closing sentence on speculative decoding. And the state-space alternative to attention has its own chapter, Chapter 12, which picks up exactly where Section 11.3’s cache-relief map leaves off.

Resources and Further Reading

Grouped by the chapter’s arc: building the model, the architecture record, and the arithmetic of scale. All free unless noted.

Build-alongs

The architecture record

The arithmetic of scale