Scaling Laws and Current Transformer Configurations

Dive into Deep Learning · §11.7

Scaling laws and current transformer configurations
parameter and FLOP counts · a scaling experiment · seven model families

Performance as a function of scale

§9’s Scaling Up transfers hyperparameters across model sizes; this section studies performance as size changes.

  • Our GPT and a frontier model differ by a number: parameters, tokens, FLOPs.
  • Count parameters and FLOPs, verify the estimate empirically, and express 2023–2025 model reports as configurations of our class.

The parameter census

Blocks scale as 12Ld^2 (4d^2 attention + 8d^2 FFN); embeddings as Vd — the embedding dominates when V \gtrsim 12Ld:

embedding   0.01M (  0%),  blocks   4.72M = 12.01 L d^2,  total 4.73M
embedding  39.38M ( 32%),  blocks  85.06M = 12.02 L d^2,  total 124.44M
  • GPT-2’s “124M” = an 85M transformer + a 39M lookup table.
  • Scaling laws count non-embedding parameters N: the embedding tracks the tokenizer, not the per-token computation.

Six FLOPs per parameter and token

C \approx \underbrace{2ND}_{\textrm{forward}} + \underbrace{4ND}_{\textrm{backward}} = 6ND

  • Forward: each matmul parameter = one multiply–add per token.
  • Backward: two matmuls per layer — grad w.r.t. input (chain rule) and w.r.t. weights (learning).
  • Rounded away: attention scores, n/(6d) — ~8% at n=128, d=256; parity only near n = 6d.

Decode-time arithmetic (why a generated token costs 2N) was §11.3’s — the KV cache makes memory traffic the main constraint.

Check it against the machine

6ND analytic: 2.324e+11 FLOPs
profiler:     2.326e+11 FLOPs (ratio 1.001)
21 ms per step: 11.0 TFLOP/s achieved
  • PyTorch profiler: agrees to a fraction of a percent — it and the formula ignore the same term (the fused kernel’s score work).
  • XLA static analysis: ~10% above 6ND (it counts scores + softmax); backward:forward = 2.00, the 4-to-2 of the derivation.
  • Achieved TFLOP/s: an order of magnitude under peak — small models underfeed big GPUs.

A scaling study on one GPU

The experiment holds the training data fixed and varies only the model.

  • Corpus: The Time Machine + PTB text = 5.1M characters (the novel alone stops improving validation loss substantially past 10^5 parameters).
  • Five sizes, widths 96→384 with depths 3→8: 0.33M → 14.2M params.
  • Identical data exposure: 16.4M tokens, approximately three passes; dropout 0; one seed, so no uncertainty estimate.
  • Learning rate ∝ 1/width (§9’s transfer rule) — a frozen rate would handicap one end of the family.

The sweep

d= 96, 3 blocks:  0.33M parameters, train 1.21, validation 1.20
d=128, 3 blocks:  0.59M parameters, train 1.17, validation 1.17
d=192, 4 blocks:  1.77M parameters, train 1.11, validation 1.12
d=256, 6 blocks:  4.72M parameters, train 1.06, validation 1.09
d=384, 8 blocks: 14.16M parameters, train 1.03, validation 1.06

Fixed-Data Scaling in the Teaching Experiment

  • Smaller sizes lie roughly on a straight log–log segment; the largest has a wider train–validation gap and a smaller validation improvement.
  • The result is consistent with a fixed-data limitation but does not exclude model-specific optimization or tuning effects.
  • Five sizes and one seed cannot estimate a scaling exponent or a compute-optimal token–parameter ratio.

The published law below comes from a separate multi-model, multi-data study.

The published form of the law

L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \qquad \textrm{(Hoffmann et al., 2022)}

  • E: the entropy floor — why raw loss cannot stay on a straight line forever.
  • A/N^{\alpha}: capacity — the term our five sizes traversed.
  • B/D^{\beta}: data — with D fixed, an effective floor: our bend.
  • Under C \approx 6ND, the optimum sets \alpha A N^{-\alpha} = \beta B D^{-\beta} → N^* \propto C^{\beta/(\alpha+\beta)}, D^* \propto C^{\alpha/(\alpha+\beta)}; fitted \alpha \approx 0.34, \beta \approx 0.28 put both near C^{1/2}. The reported allocation of roughly 20 tokens per parameter is specific to that study’s protocol.

The modern recipe (2023–2025)

model attention + cache norm FFN / experts
Mistral 7B GQA 32:8, window 4096 RMS pre SwiGLU, dense
Llama 3 GQA 32:8 RMS pre SwiGLU, dense
Qwen3 GQA 64:8 dense, 64:4 MoE RMS pre + QK SwiGLU, dense & MoE 128/8
DeepSeek-V3 MLA: 512-d latent RMS pre SwiGLU, MoE 256+1/8
Gemma 3 GQA, local:global 5:1 RMS pre+post + QK GeGLU, dense
GPT-OSS GQA 64:8, window 128 alt., sinks RMS pre SwiGLU, MoE 128/4

All: RoPE positions, no dropout — at trillion-token scale the corpus outweighs the parameters.

Recipes as constructor calls

GPT-2 (2019) 5.26M  pos='learned', norm='layer', act='gelu', pre_norm=True, bias=True, kv_heads=8
Mistral-7B   4.40M  pos='rope', norm='rms', act='swiglu', pre_norm=True, bias=False, kv_heads=2
Llama-3      4.40M  pos='rope', norm='rms', act='swiglu', pre_norm=True, bias=False, kv_heads=2
Qwen3        4.40M  pos='rope', norm='rms', act='swiglu', pre_norm=True, bias=False, kv_heads=2
  • The three modern rows print the same argument list.
  • What the flags don’t reach: window widths, RoPE bases, expert counts; MLA = §11.3’s low-rank cache, MoE swaps in via ffn_factory (§11.6).
  • The configurations retain the 2017 block while addressing stability, cache size, and capacity per FLOP.

Recap

  • 6ND estimates parameter-linked matmul FLOPs; attention-score, softmax, normalization, and optimizer work are additional terms.
  • With a fixed number of training tokens, loss initially falls with model size, then improves more slowly as the data term begins to limit performance.
  • The reported 2023–2025 configurations combine these components in different ways; the table is a dated comparison, not a universal recipe.
  • Next frontiers: linear-attention hybrids (ch. 13), long context as engineering, and the data/post-training story (Language Models part).