Scaling Laws and the Modern Recipe

Dive into Deep Learning · §11.7

Scaling laws and the modern recipe
count parameters and FLOPs · a scaling study, bend included · seven model families, one argument list

What scale buys

Boundary: §9’s Scaling Up transfers hyperparameters across size (muP); this section asks what size itself buys

  • Our GPT and a frontier model differ by a number: parameters, tokens, FLOPs.
  • Learn to count → verify against the machine → measure what the count buys → read the 2023–2025 reports as configurations of our class.

The parameter census

Blocks scale as 12Ld^2 (4d^2 attention + 8d^2 FFN); embeddings as Vd — the embedding dominates when V \gtrsim 12Ld:

embedding   0.01M (  0%),  blocks   4.72M = 12.01 L d^2,  total 4.73M
embedding  39.38M ( 32%),  blocks  85.06M = 12.02 L d^2,  total 124.44M
  • GPT-2’s “124M” = an 85M transformer + a 39M lookup table.
  • Scaling laws count non-embedding parameters N: the embedding tracks the tokenizer, not the per-token computation.

Six FLOPs per parameter and token

C \approx \underbrace{2ND}_{\textrm{forward}} + \underbrace{4ND}_{\textrm{backward}} = 6ND

  • Forward: each matmul parameter = one multiply–add per token.
  • Backward: two matmuls per layer — grad w.r.t. input (chain rule) and w.r.t. weights (learning).
  • Rounded away: attention scores, n/(6d) — ~8% at n=128, d=256; parity only near n = 6d.

Decode-time arithmetic (why a generated token costs 2N) was §11.3’s — the KV cache turns it into a memory bill.

Check it against the machine

2ND analytic: 7.743e+10 FLOPs, XLA forward: 8.480e+10 (ratio 1.10)
6ND analytic: 2.323e+11 FLOPs, XLA step:    2.547e+11 (ratio 1.10)
backward / forward = 2.00
  • PyTorch profiler: agrees to a fraction of a percent — it and the formula ignore the same term (the fused kernel’s score work).
  • XLA static analysis: ~10% above 6ND (it counts scores + softmax); backward:forward = 2.00, the 4-to-2 of the derivation.
  • Achieved TFLOP/s: an order of magnitude under peak — small models underfeed big GPUs.

A scaling study on one GPU

Design: hold the diet fixed, move only the model.

  • Corpus: The Time Machine + PTB text = 5.1M characters (the novel alone saturates everything past 10^5 parameters).
  • Five sizes, widths 96→384 with depths 3→8: 0.33M → 14.2M params.
  • Identical diet: 16.4M tokens ≈ 3 passes; dropout 0; single seed (seed noise < 0.01 nat, checked).
  • Learning rate ∝ 1/width (§9’s transfer rule) — a frozen rate would handicap one end of the family.

The sweep

d= 96, 3 blocks:  0.33M parameters, train 1.21, validation 1.21
d=128, 3 blocks:  0.59M parameters, train 1.17, validation 1.17
d=192, 4 blocks:  1.77M parameters, train 1.11, validation 1.12
d=256, 6 blocks:  4.72M parameters, train 1.06, validation 1.08
d=384, 8 blocks: 14.16M parameters, train 1.04, validation 1.07

Reading the bend

  • Small sizes: roughly a straight line — the power-law regime that makes scaling predictable (Kaplan et al., 2020).
  • The largest departs: train keeps falling on trend, validation gains a fraction of the trend; the gap widens to several hundredths.
  • Chinchilla (Hoffmann et al., 2022): ~20 tokens/param at optimum → this corpus feeds ~0.26M params. Data must scale with parameters.

No fitted exponents: five points and one seed would dress noise in decimals. The shape — line, then bend — is the finding.

The published form of the law

L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \qquad \textrm{(Hoffmann et al., 2022)}

  • E: the entropy floor — why raw loss cannot stay on a straight line forever.
  • A/N^{\alpha}: capacity — the term our five sizes traversed.
  • B/D^{\beta}: data — with D fixed, an effective floor: our bend.
  • Under C \approx 6ND, the optimum sets \alpha A N^{-\alpha} = \beta B D^{-\beta} → N^* \propto C^{\beta/(\alpha+\beta)}, D^* \propto C^{\alpha/(\alpha+\beta)}; fitted \alpha \approx 0.34, \beta \approx 0.28 put both near C^{1/2}: grow them together, ~20 tokens per parameter.

The modern recipe (2023–2025)

model attention + cache norm FFN / experts
Mistral 7B GQA 32:8, window 4096 RMS pre SwiGLU, dense
Llama 3 GQA 32:8 RMS pre SwiGLU, dense
Qwen3 GQA 64:8 dense, 64:4 MoE RMS pre + QK SwiGLU, dense & MoE 128/8
DeepSeek-V3 MLA: 512-d latent RMS pre SwiGLU, MoE 256+1/8
Gemma 3 GQA, local:global 5:1 RMS pre+post + QK GeGLU, dense
GPT-OSS GQA 64:8, window 128 alt., sinks RMS pre SwiGLU, MoE 128/4

All: RoPE positions, no dropout — at trillion-token scale the corpus outweighs the parameters.

Recipes as constructor calls

GPT-2 (2019) 5.26M  pos='learned', norm='layer', act='gelu', pre_norm=True, bias=True, kv_heads=8
Mistral-7B   4.40M  pos='rope', norm='rms', act='swiglu', pre_norm=True, bias=False, kv_heads=2
Llama-3      4.40M  pos='rope', norm='rms', act='swiglu', pre_norm=True, bias=False, kv_heads=2
Qwen3        4.40M  pos='rope', norm='rms', act='swiglu', pre_norm=True, bias=False, kv_heads=2
  • The three modern rows print the same argument list.
  • What the flags don’t reach: window widths, RoPE bases, expert counts; MLA = §11.3’s low-rank cache, MoE swaps in via ffn_factory (§11.6).
  • Convergent evolution: three selection pressures — stability, cache, capacity per FLOP — one body plan: the 2017 block.

Recap

  • Cost = 6ND; a profiler and a compiler both endorse the formula.
  • On a fixed diet, loss falls with size along a rough line until the corpus saturates the model — the bend is the Chinchilla lesson.
  • The 2026 recipe: GQA/MLA on a pre-norm RMSNorm block, gated FFN, RoPE, no dropout, MoE for capacity — our GPT class with different flags.
  • Next frontiers: linear-attention hybrids (ch. 13), long context as engineering, and the data/post-training story (Language Models part).