10.7 Further Reading
The sections of this chapter teach one representative of each idea: one soft lookup, one scoring function, one way to split attention into heads, a few ways to encode position, one exact kernel and two ways to avoid the quadratic cost, and one circuit traced through a small model. This section points to topics they had no room for: other readings of the lookup and its normalization, what trained heads do and how many are needed, further positional schemes and their limits, older and newer ways to reduce the cost, and the tools that analyze circuits in large models. Each topic states the problem it addresses and what it changes relative to the chapter. Implementations, tutorials, and courses for the material that the chapter does teach are listed in its introduction, Chapter 10. Nothing else in the book depends on this section.
10.7.1 The Lookup and Its Normalization
10.7.1.1 Attention as Associative Memory
Section 10.1 read attention as a smoothed lookup in a table of key-value pairs, with a bandwidth that decides between retrieving one entry and averaging many. Ramsauer et al. (2021) read the same computation as memory retrieval: softmax attention with one query is one update step of a modern Hopfield network whose stored patterns are the keys. The network’s energy function makes the trade-off precise. At a large inverse temperature the update converges to a single stored pattern; at a small one it converges to an average of many, which is the role that the bandwidth plays in Section 10.1 and the factor \(1/\sqrt{d}\) in Section 10.2. They also show that such a network can store a number of patterns exponential in the dimension.
10.7.1.2 Bandwidths That Follow the Data
The Nadaraya–Watson estimator of Section 10.1 uses one bandwidth everywhere, which is too narrow where the data are sparse and too wide where they are dense. Adaptive kernel estimators let the bandwidth follow the local density of the data (Silverman 1986), a hand-built analogue of a temperature that depends on the query. The same lookup also works as a representation without any training: ASIF (Norelli et al. 2022) describes an image or a caption by its similarities to a fixed set of image–caption anchor pairs, which lets two separately trained encoders exchange information without a joint model.
10.7.1.3 Monotone Alignment
Bahdanau’s attention of Section 10.2 was not the first differentiable alignment. Graves’s handwriting synthesis (Graves 2013) aligned each pen stroke with a position in the text through a soft window that could only move forward, a constraint inherited from the left-to-right hidden Markov models of speech recognition (Rabiner and Juang 1993). Dropping monotonicity is what lets attention handle reordering between languages, as in the alignment of “black cat” with “chat noir”.
10.7.1.4 Weights That Need Not Sum to One
The softmax forces every row of weights to sum to one, so a query that finds nothing relevant must still put its weight somewhere. Trained models often put it on the first token, the attention sink that Section 10.3 and Section 10.6 mention and Section 11.3.4.2 measures. Several lines of work ask why. Sun et al. (2024) find that sinks coincide with a few activations in the residual stream that are orders of magnitude larger than the rest and act as fixed biases. Barbero, Arroyo, et al. (2025) argue that sinks are useful: they let a deep model with a long context avoid mixing every token into every other. Two designs remove the need for them. Sigmoid attention (Ramapuram et al. 2025) replaces the softmax by an elementwise sigmoid with a bias of \(-\log n\), so that a query can give little weight to every key. It trains stably once that bias is in place and the attention output is kept small early in training, and its fused kernel is about a sixth faster than FlashAttention-2 on an H100. Gated attention (Qiu et al. 2025), which Section 11.3.4.2 describes, keeps the softmax and multiplies each head’s output by a sigmoid gate computed from the layer input.
10.7.2 Heads
10.7.2.1 Redundant Heads
Section 10.3 proved that one head cannot return two values and that two heads can, and the trained heads of its experiment nearly duplicated each other. Trained models at scale show the same redundancy. Michel et al. (2019) removed heads of trained translation and BERT models at test time and found that a large fraction can go with little loss, and that many layers work nearly as well with a single head. Voita et al. (2019) identified the few heads that carry the effect, among them heads that attend to the previous token, to rare words, or along a syntactic relation, and pruned most of the others with a learned gate at a small cost in quality.
10.7.2.3 Subtracting Two Weightings
Differential attention (Ye et al. 2025) computes two softmax weightings from two sets of queries and keys and subtracts a learned multiple of the second from the first, so that weight that both place on irrelevant tokens cancels. The authors report that it matches a standard transformer with about two thirds of the parameters or training tokens. No widely used open model has adopted it so far, so it is a candidate rather than a standard.
10.7.3 Positions
10.7.3.1 Learned Relative Positions
Before RoPE, relative position entered the score as trained vectors, one per clipped offset, added to the keys inside the dot product (Shaw et al. 2018). The Music Transformer (Huang et al. 2018) made this affordable for long sequences by computing the relative logits for all offsets at once and then shifting them into place, instead of storing a vector for every pair of positions. RoPE in Section 10.4 reaches scores that depend only on the offset with no learned parameters at all.
10.7.3.2 RoPE in Trained Models
Released models differ less in whether they use RoPE than in its base. Llama 2 used \(10^4\), Llama 2 Long and Llama 3 use \(5 \times 10^5\) (Xiong et al. 2024), and the global layers of Gemma 3 use \(10^6\) (Gemma Team 2025). A larger base slows every pair except the fastest, and so moves more pairs into the slow group of Section 10.4.4.4. Barbero, Vitvitskyi, et al. (2025) looked at which pairs a trained model uses. Gemma 7B uses the fastest pairs for heads that attend by position, such as previous-token heads, and the slowest pairs as nearly position-free channels for content. These two roles are the trained counterpart of the fast and slow pairs of Section 10.4. Removing the lowest frequencies altogether, a variant they call p-RoPE, helped in their experiments.
10.7.3.3 Per-Pair Rescaling
YaRN, in Section 10.4.4.4, leaves the fast pairs alone, interpolates the slow ones, and ramps between the two groups with a hand-set rule. LongRoPE (Ding et al. 2024) replaces the rule by an evolutionary search for one rescale factor per pair and leaves the first tokens unscaled. With at most a thousand fine-tuning steps at training lengths up to 256k it extends the context to 2048k tokens.
10.7.3.4 Length Generalization of ALiBi and NoPE
Section 10.4.4.3 found that ALiBi and NoPE keep their perplexity beyond the training length. Chi et al. (2023) explain the first. ALiBi’s steepest slopes make most heads attend within a window near the training length, so the extra tokens are effectively never read; the exercise on where extrapolation fails measures this. NoPE has its own limit. Haviv et al. (2022) showed that causal language models without positional encoding are competitive in perplexity and learn absolute position from the causal mask. Wang et al. (2024) found that their length generalization also breaks, because attention spreads over too many tokens, and restored it with a softmax temperature per head. Llama 4 (Meta AI 2025) applies the same fix at inference time to the position-free global layers that it interleaves with RoPE layers.
10.7.3.5 Perplexity and the Use of Long Contexts
A model whose perplexity stays flat at four times its training length need not use the extra tokens. Liu et al. (2024) placed the answer to a question at different positions of a long context and found that models retrieve it far better from the beginning or the end than from the middle. How to train models that use long contexts, by continued pretraining on long documents with a raised base and by schedules that grow the length, belongs to the Language Models part of the book.
10.7.4 Cost
10.7.4.1 Approximating the Softmax Kernel
Before fused exact kernels, several methods approximated attention to reach linear cost. Performer (Choromanski et al. 2021) replaces \(\exp(\mathbf{q}^\top\mathbf{k})\) by an inner product of positive random features whose expectation is the exponential, so that the associativity of Section 10.5 applies. Linformer (Wang et al. 2020) projects the keys and values along the sequence axis to a fixed length, and Reformer (Kitaev et al. 2020) hashes similar queries and keys into shared buckets and attends within each bucket. cosFormer (Qin et al. 2022) uses ReLU features with a cosine reweighting that favors nearby positions. None of them is exact, and in practice they were displaced by exact attention in fused kernels, by windows, and by the gated recurrences below. Tay et al. (2020) catalogue dozens more.
10.7.4.2 Global Tokens and Strided Patterns
The window of Section 10.5.3.2 needs many layers before distant tokens meet. Longformer (Beltagy et al. 2020) adds to the window a few positions chosen for the task, such as a classification token, that attend to every token and are attended by every token, so that the whole document reaches them in one layer. The Sparse Transformer (Child et al. 2019) combines a local pattern with a strided one so that two layers connect any pair of positions. Both patterns are fixed in advance, like the window.
10.7.4.3 Windows in Released Models
Mistral 7B (Jiang et al. 2023) used a window of 4,096 tokens in all 32 layers, with a rolling cache that holds only the last \(w\) keys and values. Its successor, Mistral-7B-Instruct-v0.2 (Mistral AI 2023), dropped the window for a full context of 32k tokens and a RoPE base of \(10^6\). Gemma 2 (Gemma Team 2024) alternates windowed and global layers one to one and reports almost unchanged perplexity when its windows shrink from 4,096 to 1,024 tokens at inference. Gemma 3 (Gemma Team 2025) uses five windowed layers of 1,024 tokens per global layer, the ratio whose cache saving Section 10.5 computes, and finds little change in perplexity even at seven to one.
10.7.4.4 Sparsity Learned from Content
A window decides by position which keys a query sees. Three recent designs decide by content, and each is built to run as a fused kernel over blocks. Natively sparse attention (Yuan et al. 2025) scores compressed blocks of keys, keeps the highest-scoring blocks for each query, and adds a sliding window, all trained end to end. MoBA (Lu et al. 2025) routes each query to the key blocks whose mean key it matches best, a mixture-of-experts gate over blocks. DeepSeek Sparse Attention (DeepSeek-AI et al. 2025) scores all past tokens with a small separate indexer and runs attention over the top-scoring ones only.
10.7.4.5 FlashAttention-3 and FlexAttention in Detail
Section 10.5 describes FlashAttention-3 (Shah et al. 2024) as the same algorithm with a new schedule. On H100 GPUs, some warps issue asynchronous tile loads while others run the matrix products. Two groups of warps also take turns, so that the exponentials of one group, which run on a much slower unit, overlap the matrix product of the other. Its FP8 mode quantizes each block separately. It first multiplies queries and keys by a random orthogonal matrix that spreads outlying coordinates, which lowers the error of FP8 attention by a factor of 2.6 against a single scale per tensor. FlexAttention (Dong et al. 2025) generates kernels for masks and score modifications written as Python functions; on the variants that FlashAttention-2 supports, its kernels run at 0.68 to 1.43 times the speed of the handwritten one. Kernels for decoding, which split the cache across thread blocks and read it from pages, are treated with the serving systems of the Language Models part.
10.7.4.6 Gated Recurrences and Hybrids
Section 10.5 ended with the observation that plain linear attention lost to softmax attention, and that two changes made it competitive. Gated linear attention (Yang et al. 2024) adds a decay that depends on the input, and Gated DeltaNet (Yang et al. 2025) replaces the additive write by a delta rule that overwrites the value stored under a key; both train with chunkwise kernels like the chunked form of the chapter. The models that use them keep a few softmax layers for recall: Qwen3-Next (Qwen Team 2025) places one gated-attention layer after every three Gated DeltaNet layers, Kimi Linear (Kimi Team 2025) mixes three of its recurrent layers with one latent-attention layer, and MiniMax-01 (MiniMax 2025) uses one softmax layer per seven linear ones. Mamba-2 (Dao and Gu 2024) shows that a selective state space model with a scalar decay computes the same output as a masked attention whose mask holds products of the decays, so the chunked form with a decay is one of its algorithms. Chapter 12 teaches these recurrences.
10.7.5 Circuits in Large Models
10.7.5.1 Patching and Automated Circuit Discovery
The ablation of Section 10.6.3.3 shows that a component is needed, not what it carries or along which path. Activation patching copies an activation from a run on a clean input into a run on a corrupted one and measures how much of the clean behavior returns (Meng et al. 2022). Path patching restricts the copy to one path between components, which is how Wang et al. (2023) traced the circuit that GPT-2 small uses to complete “When Mary and John went to the store, John gave a drink to” with “Mary”. ACDC (Conmy et al. 2023) automates the search over edges. Attribution patching (Nanda 2023) replaces each patch by a first-order estimate from gradients, so that one backward pass scores every component, and AtP* (Kramár et al. 2024) corrects the cases where that estimate misses an important component. Zhang and Nanda (2024) and Heimersheim and Nanda (2024) study how the conclusions depend on the choice of corrupted input and on the value that replaces an ablated component. The Language Models part of the book treats these methods on large models.
10.7.5.2 Superposition and Dictionary Features
The circuit algebra of Section 10.6 assumes that each head reads and writes identifiable directions of the residual stream. In large models the features outnumber the dimensions and share directions, a regime that Elhage et al. (2022) call superposition and reproduce in small models. Sparse dictionaries learned on the activations recover features one at a time, and attribution graphs built from them trace which features cause which across layers (Ameisen et al. 2025). These methods, too, belong to the Language Models part.
10.7.5.3 Induction Heads at Scale
Olsson et al. (2022) found the induction heads of Section 10.6 in large models as well. Heads identified on repeated random tokens also copy translated fragments and support few-shot completion, and the authors give six lines of evidence that induction heads carry most of in-context learning: causal in small attention-only models, correlational in large ones.
10.7.5.4 In-Context Learning as an Algorithm
Section 10.6 showed that one head without a softmax takes one gradient step on a regression problem stored in its context. Garg et al. (2022) trained transformers to regress in context on whole function classes, such as linear functions, sparse linear functions, and decision trees, and found errors close to those of the best algorithm for each class. Akyürek et al. (2023) asked which algorithm a trained model implements and found a progression with depth and width, from one gradient step towards ridge regression.