The transformation preserves direction and caps the global norm. Introduced for RNNs in ch. 8, it also appears in the two reports that disclose clipping; both use threshold 1.
class Clipped:"""Clip the global gradient norm before every optimizer step."""def__init__(self, optimizer, params, max_norm=1.0):self.optimizer = optimizerself.params =list(params)self.max_norm = max_normself.norms = []def step(self): norm = nn.utils.clip_grad_norm_(self.params, self.max_norm)self.norms.append(float(norm))self.optimizer.step()def zero_grad(self):self.optimizer.zero_grad()
Preventing numerical overflow with clipping
§9.6’s knife edge: SGD’s best lr sat one grid point below a NaN. Rerun the divergent point, with and without the guard:
Unclipped: a large step reaches a region with larger gradients, momentum compounds the growth, and the run overflows.
Clipped at the same learning rate: final loss is in the tuned runs’ range.
Frequency of clipping
The instrumented run: median gradient norm ~0.3, threshold 1.0 — clipping changed the update on 6 of 2,000 steps.
Language-model gradient noise is heavy-tailed (Zhang et al., 2020); the guard exists for the tail.
If clipping fires on most steps, it continuously changes the update rule; tune \eta and \theta together.
For a persistent gradient, Adam’s coordinate update is on the order of \eta (with transients up to about 3×), so clipping is no substitute for lowering a too-large \eta — it guards \mathbf{m}, \mathbf{v} from one huge gradient lingering 1/(1-\beta_2) steps.
Additional stability methods
Clipping is one item. The rest aims at attention logits and the softmax:
z-loss (PaLM): penalize \log^2 Z of the softmax normalizer.
QK-norm (OLMo 2): normalize q, k right before their dot product.
QK-clip (MuonClip): cap the largest attention logit — 15.5T tokens, zero spikes (§9.9).
When prevention failed, PaLM rewound to an earlier checkpoint and skipped the implicated batches. Replaying those batches from a different state did not reproduce the spike, indicating an interaction between state and data. The OPT logbook documents two months of restarts and learning-rate changes.
Weight averaging
Weight averaging reduces endpoint variability without changing the learning rate: \bar{\mathbf{x}}_t = \alpha \bar{\mathbf{x}}_{t-1} + (1-\alpha)\mathbf{x}_t, the chapter’s leaky average, now on the weights (SWA; Izmailov et al., 2018).
Window must fill first — the early gap is lag on stale iterates, not a bias to correct; warm up the decay or start late.
Afterward, this EMA run is modestly above the live-weight curve and less sensitive to the stopping epoch.
Averaging: where it matters
Constant rate plus EMA reaches the range of the decayed schedules in §9.8, though decay changes future iterates whereas averaging changes the evaluated parameters. With prior decay, one run shows no clear added gain.
LLMs: checkpoint averaging (LAWA); Llama 3 shipped an average of its annealing checkpoints. Model soups: average fine-tuned models.
Diffusion models (:numref:chap_diffusion) commonly use EMA; Karras et al. (2024) reconstruct multiple EMA windows after training to tune the window.
How to tune
The Tuning Playbook’s vocabulary
Scientific hyperparameters define the question. Nuisance hyperparameters are retuned per arm. Fixed settings limit the claim.
This chapter, named: optimizer scientific, lr nuisance (four-point grid per contestant), all else fixed and stated.
Schmidt et al. (2021): several optimizers at defaults ≈ one optimizer heavily tuned. Untuned comparisons confound optimizer choice with tuning effort.
With few runs, prioritize the peak learning rate; with tens, add weight decay and schedule parameters. Log the configuration, seed, changes, and diverged runs so the comparison is reproducible.
Topics covered elsewhere
SAM: flat minima at 2× gradient cost — wins concentrate in vision and fine-tuning.
Variance reduction: strong finite-sum theory, but no consistent improvement reported for deep networks → ch. 25.
LARS/LAMB: in the cited benchmark, retuned momentum/AdamW matched them at the studied batch sizes.
Systems: sharding state, data parallelism, overlap → ch. 11 and the training-systems appendix.
Recap: three decisions
Direction: a norm — gradient, sign, or orthogonalized (§9.2, §9.6, §9.9).
Step scale over time: warmup, cosine, and WSD (§9.8); clipping is a separate guard on unusually large gradients.
Each row of the table is one reported coordination of these choices. When a run misbehaves, ask which decision is failing. Optimizers change; the decomposition has been stable for decades.
