11  Transformers

Chapter 10 developed attention as a mechanism for exchanging information between positions. A transformer places attention inside a block with a position-wise feed-forward network, residual connections, and normalization. Stacking these blocks yields models for machine translation, language modeling, images, audio, and protein structure. Although these applications differ, most current transformer models can be described by a small set of architectural choices.

We therefore begin by building a configurable GPT class: normalization type and placement, activation, positional scheme, and the attention and feed-forward modules are all constructor arguments. Subsequent sections vary one component at a time and measure the resulting difference. At the end of the chapter, the configurations in a table of current models can be expressed as constructor arguments. The training runs finish in minutes on a single GPU. We state when conclusions depend on this small scale and compare the computational cost with that of production training.

We first develop the transformer block and compare normalization placement, RMSNorm, QK-norm, and gated feed-forward networks. We then assemble a GPT, train it on a small text corpus, and load the published GPT-2 weights into the same implementation. The section on generation derives the KV cache and compares grouped-query attention, low-rank compression, and sliding windows with attention sinks.

The remaining sections examine encoder, decoder, and encoder–decoder architectures; apply an encoder to image patches; and replace dense feed-forward networks with mixture-of-experts layers. The chapter concludes by deriving parameter and FLOP counts, conducting a small scaling study, and comparing the configurations of several current models.

The original 2017 Transformer was an encoder–decoder for translation, with post-sublayer normalization, sinusoidal positions, and ReLU feed-forward networks. The model configurations tabulated at the end of this chapter, reported from 2023 through 2025, retain the residual attention–FFN block but vary normalization, position encoding, cache layout, and expert routing. This shared interface motivates the configurable implementation used here; it is not a claim that all transformer families have converged on one design.

This chapter assumes the tokenization methods of Chapter 8 and the optimizers of Chapter 9. It covers base-model architecture, not corpus construction, instruction tuning, or downstream adaptation. Vision applications beyond ViT appear in Chapter 20; kernels, parallelism, quantization, and serving are deferred to Chapter 13. State-space alternatives to a growing key–value cache are developed in Chapter 12.

Resources and Further Reading

The resources are grouped into model construction, architecture, and scaling. All are freely available unless noted.

Build-alongs

• Let’s build GPT: from scratch, in code — Andrej Karpathy (2023) — the video counterpart of Section 11.2: a character-level GPT assembled and trained in real time; nanoGPT and build-nanogpt are its repository forms, and nanochat extends the same discipline to a full chat system with stated dollar costs.
• The Annotated Transformer — Harvard NLP (2018, refreshed 2022) — the original encoder–decoder implemented line by line against the paper; its executable format closely matches the construction in Section 11.4.
• Build a Large Language Model (From Scratch) — Sebastian Raschka (2024) — a book-length version of Section 11.1 through Section 11.3, with bonus notebooks for GQA, sliding windows, and from-scratch ports of current open models.
• Stanford CS336: Language Modeling from Scratch — a course whose first assignment builds BPE, RMSNorm, RoPE, SwiGLU, causal attention, and the training loop from primitives; lectures are available on YouTube.
• CMU Advanced NLP, minLlama assignment — build a Llama-style decoder and load real pretrained weights into it, the same payoff as Section 11.2’s GPT-2 cell at larger scale.

The architecture record

• The Big LLM Architecture Comparison — Sebastian Raschka (2025, maintained) — seventeen current models on the same few axes; the living version of Section 11.7’s recipe table, updated as new models ship.
• On Layer Normalization in the Transformer Architecture — Xiong et al. (2020) — the pre-norm/post-norm analysis behind Section 11.1’s signal-propagation experiment, and the paper that explains why warmup exists.
• GLU Variants Improve Transformer — Shazeer (2020) — the four-page note whose matched-parameter sweep Section 11.1 reproduces in miniature.
• GQA: Training Generalized Multi-Query Transformer Models — Ainslie et al. (2023) — grouped-query attention as Section 11.3 implements it, including the uptraining recipe that converted existing checkpoints.
• An Image is Worth 16x16 Words — Dosovitskiy et al. (2021) — the ViT paper behind Section 11.5; our small-scale experiment examines the same scale-versus-inductive-bias question in a different regime.
• Switch Transformers — Fedus et al. (2021) and DeepSeek-V3 — DeepSeek-AI (2024) — the two poles of Section 11.6: top-1 routing with an auxiliary balancing loss, and fine-grained experts balanced without one.

The arithmetic of scale

• Transformer Inference Arithmetic — kipply (2022) — derives per-token inference FLOPs, cache storage, and bandwidth limits and compares the estimates with a real system.
• Transformer Math 101 — EleutherAI (2023) — the training-side companion: where 6ND comes from and what it predicts, the accounting Section 11.7 verifies against a profiler.
• Training Compute-Optimal Large Language Models — Hoffmann et al. (2022) — Chinchilla: the tokens-per-parameter result whose small-scale shadow is the bend in Section 11.7’s miniature study.
• The Ultra-Scale Playbook — Hugging Face (2025) — a guide to the parallelism and memory engineering required beyond one GPU, topics covered in the Computational Performance chapter.