Gradient-Based Optimization
Dive into Deep Learning · §25.1
What moves every neural network
why -\nabla f works, how fast it converges, and what breaks it.
The foundations under the optimizer zoo
Motivation
Before momentum, Adam, and learning-rate schedules, three questions have closed-form answers on quadratics:
- Why does a negative-gradient step make progress?
- How fast does it converge?
- What throttles it?
One number runs through all three: the condition number \kappa = \lambda_{\max}/\lambda_{\min}, read off the Hessian.
Develop the theory where it is exact (quadratics), then add momentum and noise to make it practical.
01
Directions & smoothness
where to step, and how far to trust it
Steepest descent is a theorem
Directions
A small step \eta\,\mathbf{d} changes f by \eta\,\nabla f^\top\mathbf{d} to first order, so \mathbf{d} is a descent direction when \nabla f^\top\mathbf{d} < 0. By Cauchy–Schwarz the steepest unit direction is unique:
\min_{\|\mathbf{d}\|=1}\nabla f^\top\mathbf{d} = -\|\nabla f\|, \qquad \mathbf{d}^\star = -\frac{\nabla f}{\|\nabla f\|}.
Descent fills an entire half-space (everything within 90^\circ of -\nabla f); the gradient is the greediest choice, not the only one.
Every negative slope really descends
Directions
On f(x,y)=\tfrac12(x^2+10y^2): compare the first-order slope with the actual decrease, then scan 3600 directions for the most negative slope.
steepest -grad/|grad|: slope = -10.1980, f(x + eta d) - f(x) = -0.101498
diagonal -(1,1)/sqrt(2): slope = -8.4853, f(x + eta d) - f(x) = -0.084578
axis (-1,0): slope = -2.0000, f(x + eta d) - f(x) = -0.019950
uphill +(1,1)/sqrt(2): slope = +8.4853, f(x + eta d) - f(x) = +0.085128
best of 3600 sampled unit directions: [-0.1959 -0.9806] vs -grad/|grad| = [-0.1961 -0.9806]The brute-force winner lands on -\nabla f/\|\nabla f\|, to grid resolution.
L-smoothness erects a ceiling
Smoothness
The slope goes stale as we move. Bound how fast: f is L-smooth when \|\nabla f(\mathbf{x})-\nabla f(\mathbf{y})\|\le L\|\mathbf{x}-\mathbf{y}\|, i.e. every Hessian eigenvalue lies in [-L,L]. That traps the graph under a quadratic ceiling and gives the descent lemma:
f(\mathbf{x}_{k+1}) \le f(\mathbf{x}_k) - \eta\big(1 - \tfrac{L\eta}{2}\big)\|\nabla f(\mathbf{x}_k)\|^2.
Gain \eta\|\nabla f\|^2 grows linearly; the curvature tax \tfrac{L}{2}\eta^2\|\nabla f\|^2 grows quadratically. Progress for 0<\eta<2/L, best at \eta=1/L.
The one guarantee deep nets keep
Smoothness
Telescoping the lemma at \eta=1/L, with f merely bounded below by f^\star:
\min_{0\le k<K}\|\nabla f(\mathbf{x}_k)\|^2 \;\le\; \frac{2L\,(f(\mathbf{x}_0)-f^\star)}{K}.
No convexity needed. Gradient descent reliably finds points where the gradient is small at rate O(1/K). It does not promise a minimum, only stationarity. This theorem applies verbatim to training a network.
Backtracking: a step near 1/L without knowing L
Smoothness
The local curvature varies, so no fixed step fits everywhere. Start optimistic and halve \eta until the Armijo sufficient-decrease condition holds, here on the quartic f(x)=\tfrac14 x^4:
fixed eta = 0.3 from x0 = 3: |x_6| = 6.52e+103 (diverged)
accepted steps: [0.03125 0.0625 0.125 0.25 0.5 1. 1. 1. ] ...
backtracking from x0 = 3: f(x_25) = 1.09e-04 (monotone descent)Fixed \eta=0.3 diverges from x_0=3; backtracking accepts 0.031, then doubles as the valley flattens. A line search is curvature estimation by trial.
02
The condition number
the quadratic model, solved exactly
Gradient descent decouples into modes
Quadratics
Near a minimum every smooth f is a quadratic \tfrac12\mathbf{x}^\top A\mathbf{x}. In the eigenbasis of A the iteration \mathbf{x}_{k+1}=(I-\eta A)\mathbf{x}_k splits into n independent geometric recursions, one per curvature eigenvalue:
c_i^{(k)} = (1-\eta\lambda_i)^k\,c_i^{(0)}.
- Stability needs |1-\eta\lambda_i|<1 for all modes: \eta<2/L.
- Speed is set by the slowest: \rho(\eta)=\max_i|1-\eta\lambda_i|.
The 2/L the descent lemma offered as sufficient is now necessary: past it the stiffest mode oscillates with growing amplitude.
The step-size sweep, in six rows
Quadratics
Sweep \eta across the ceiling on A=\mathrm{diag}(1,10), so L=10 and the ceiling is \eta=0.2:
eta |1-eta*1| |1-eta*10| spectral radius |x_60|
0.020 0.980 0.800 0.980 3.0e-01
0.100 0.900 0.000 0.900 1.8e-03
0.182 0.818 0.818 0.818 8.3e-06
0.190 0.810 0.900 0.900 1.8e-03
0.200 0.800 1.000 1.000 1.0e+00
0.210 0.790 1.100 1.100 3.0e+02Tiny \eta is stable but glacial; \eta=2/11 balances the extreme modes and wins (\rho=0.818); at \eta=0.2 the stiff mode bounces forever; one tick past, it explodes.
The best step levels the tent
Quadratics
Each step size draws a tent |1-\eta\lambda| with vertex at \lambda = 1/\eta; the rate is the taller endpoint over [\lambda_{\min}, \lambda_{\max}]. The best tent levels its endpoints:
\eta^\star = \frac{2}{\lambda_{\min}+\lambda_{\max}}, \qquad \rho(\eta^\star) = \frac{\kappa-1}{\kappa+1}.
Lowering either endpoint would raise the other.
With \eta^\star the iteration contracts by 0.818182, to six digits, at every step: on a quadratic the law is an exact identity.
The valley picture
Quadratics
The optimal step contracts by exactly (\kappa-1)/(\kappa+1), so the cost of gradient descent is linear in \kappa.
Stability chains the step to the steep mode while the flat mode barely moves: the zig-zag is the visible cost of a large \kappa.
Real networks train at the ceiling
Quadratics
The classical advice: measure L, pick \eta < 2/L. Measured on a real (tiny) network, the causality runs backwards: same init, two step sizes, sharpness \lambda_{\max}(\nabla^2 f) tracked by finite differences:
eta = 0.40 (2/eta = 5.0) eta = 0.25 (2/eta = 8.0)
k loss sharpness loss sharpness
0 0.5097 3.32 0.5097 3.32
4000 0.0124 4.94 0.0104 8.02
8000 0.0008 4.97 0.0021 7.96
12000 0.0007 5.00 0.0007 7.97
16000 0.0005 5.00 0.0004 7.97
20000 0.0004 5.00 0.0003 7.97Training raises the sharpness to the ceiling 2/\eta (5.00 for \eta=0.4; within half a percent of 8 for \eta=0.25), then it hovers there while the loss keeps falling, non-monotonically. The edge of stability: you pick \eta, the curvature adapts to your choice.
Convexity upgrades stationarity to optimality
Quadratics
The \kappa-law generalizes from quadratics to convex functions. Each hypothesis upgrades what converges and how fast:
smooth only \;\min_k\|\nabla f\|^2 = O(1/k) (stationarity)
+ convex \;f-f^\star = O(1/k) (global values)
+ \mu-strongly convex \;f(\mathbf{x}_k)-f^\star \le (1-\tfrac{1}{\kappa})^k\,(f(\mathbf{x}_0)-f^\star)
linear, O(\kappa\log\tfrac1\varepsilon) steps
Stated here; proved with convexity in the next section.
03
Momentum, noise, curvature
making it practical at scale
Momentum is a damped oscillator
Acceleration
Give the iterate a velocity with memory, \mathbf{v}_{k+1}=\beta\mathbf{v}_k-\eta\nabla f. In each mode this is a mass-spring-damper and \beta is the damping knob: too little over-damps (crawls), too much under-damps (rings), critical damping returns fastest.
Inertia turns \kappa into \sqrt\kappa
Acceleration
Tuned heavy ball contracts every mode at (\sqrt{\kappa}-1)/(\sqrt{\kappa}+1); Nesterov’s look-ahead makes \sqrt{\kappa} a theorem beyond quadratics, and these rates are optimal for first-order methods. Race all three to 10^{-6}:
kappa GD heavy ball Nesterov GD/kappa HB/sqrt(kappa)
10 69 27 43 6.9 8.5
100 691 93 155 6.9 9.3
1000 6908 315 508 6.9 10.0GD’s count is linear in \kappa (6{,}908 at \kappa=1000); heavy ball grows like \sqrt{\kappa} (315, a 22\times speedup). At one gradient per step, the speedup adds no per-step cost.
Minibatch noise: unbiased, variance \propto 1/b
Stochastic gradients
A training loss averages over data, so the exact gradient costs a full pass. A random minibatch estimate is unbiased with variance that falls like 1/b:
\mathbb{E}[\hat{\mathbf{g}}_b] = \nabla f, \qquad \mathbb{E}\big[\|\hat{\mathbf{g}}_b - \nabla f\|^2\big] = \frac{\mathrm{tr}\,\Sigma}{b}.
Noise energy falls like 1/b, so amplitude falls like 1/\sqrt{b}: 100\times the compute reduces the noise only 10\times, which is why huge batches show diminishing returns.
Fixed-step SGD rattles in a noise ball
Stochastic gradients
A contraction plus constant noise injection has a nonzero fixed point: SGD descends like GD, then rattles inside a ball of squared radius
\mathbb{E}[x_\infty^2] \approx \frac{\eta\,\sigma^2}{2\lambda}.
Halving \eta halves the ball, which is exactly why schedules decay \eta.
Decay reaches the optimum; fixed steps plateau
Stochastic gradients
Robbins–Monro decay (\sum\eta_k=\infty,\ \sum\eta_k^2<\infty) quenches the noise while still travelling far, on the logistic toy:
fixed eta = 0.8: mean gap, last 1000 steps = 2.34e-02
fixed eta = 0.4: mean gap, last 1000 steps = 1.08e-02
decay 0.8/(1+k/100): mean gap, last 1000 steps = 5.67e-04Fixed \eta stalls at a floor; 1/k grinds on: noise moves the rate class from geometric to polynomial.
Why not Newton?
Curvature as information
Newton minimizes the local quadratic exactly, \mathbf{x}_{k+1}=\mathbf{x}_k-(\nabla^2 f)^{-1}\nabla f: affine-invariant, immune to \kappa, with digits doubling per step.
Newton on the quadratic: |x_1| = 0.0e+00 after one step
GD with the optimal step contracts by 0.9802 per step; it needs 691 steps for |x_k| <= 1e-6
Newton iteration 1: |grad| = 2.8e-01
Newton iteration 2: |grad| = 3.2e-02
Newton iteration 3: |grad| = 1.1e-03
Newton iteration 4: |grad| = 1.2e-06
Newton iteration 5: |grad| = 1.6e-12
Newton iteration 6: |grad| = 3.1e-17
agrees with the SGD section optimum: TrueThe catch is arithmetic: O(d^2) memory and O(d^3) time per step. At d=10^9, eight exabytes before the first FLOP. Deep learning keeps the idea: L-BFGS (low-rank), Adam (diagonal), fed by minibatch gradients.
Recap
Wrap-up
- Steepest descent is Cauchy–Schwarz; descent fills a half-space.
- L-smoothness gives the descent lemma and O(1/K) stationarity, the one guarantee with no convexity.
- On quadratics GD decouples per mode: stability \eta<2/L, cost linear in \kappa.
- Convexity upgrades to global optimality; momentum turns \kappa\to\sqrt{\kappa}, optimal for first-order.
- SGD is unbiased with 1/b variance and a noise ball; decay reaches the optimum.
- Newton ignores \kappa but costs O(d^3), hence its diagonal and low-rank stand-ins (Adam, L-BFGS).
Every learning-rate recipe in the main book is an elaboration of this ledger: gain versus curvature tax, signal versus noise.