Dive into Deep Learning · §25.3
The line between “gradient descent provably works” and “we hope”
convex sets · three lenses · Jensen · global rates · what deep nets break.
Motivation
For a convex problem the guarantees are global:
Convexity also names the easy pieces of deep learning (the softmax loss, the regularizers, projections, the SVM dual), and tells you exactly what stacking nonlinear layers forfeits.
All code here is a few lines of plain NumPy: no framework, no GPU.
01
Convex sets
the segment between any two points stays inside
Convex sets
A set C is convex if every segment between two of its points never leaves it: \theta\mathbf{x} + (1 - \theta)\mathbf{y} \in C for all \mathbf{x}, \mathbf{y} \in C, \theta \in [0, 1].
Left stays inside; the crescent’s chord passes outside. Right: the simplex and a half-space.
Convex sets
Intersections preserve convexity (a point in every C_i is a point in their meet). Unions do not: [0,1]\cup[2,3] has a chord that escapes.
This one fact certifies the catalog:
The catalog: half-spaces, norm balls, the simplex, the PSD cone.
Affine maps and convex hulls round it out.
02
Three lenses on a convex function
chord, tangent, Hessian, and the kink repair
Three lenses
Two pictures of one property: the chord joining two points lies above the graph; the tangent at any point lies below it. A single gradient at \mathbf{x} thus certifies every \mathbf{y}, however far.
Three lenses
f(\theta\mathbf{x}+(1-\theta)\mathbf{y}) \le \theta f(\mathbf{x}) + (1-\theta) f(\mathbf{y})
is equivalent, for smooth f, to the tangent under-estimator, and to
\nabla^2 f(\mathbf{x}) \succeq 0 \quad \textrm{everywhere.}
Pick the cheapest lens: the chord needs no derivatives, the first-order one feeds every proof, the Hessian is the usual test for smooth losses.
Strong convexity adds a floor, \nabla^2 f \succeq \mu I: a bowl with guaranteed curvature, and \kappa = L/\mu is the condition number.
Three lenses
\ell_1 and the hinge have kinks, so no gradient exists there. A subgradient \mathbf{g} keeps the under-estimate anyway:
f(\mathbf{y}) \ge f(\mathbf{x}) + \mathbf{g}^\top(\mathbf{y}-\mathbf{x}).
At a corner the slopes fan out: \partial|x|(0) = [-1, 1], and the zero-slope member is an optimality certificate: \mathbf{x}^\star minimizes f iff \mathbf{0} \in \partial f(\mathbf{x}^\star).
Subgradients carry Jensen, local-equals-global, and descent through ReLU-style kinks.
Three lenses
On the least-squares loss in two weights: a thousand random chords, a thousand random tangents, and the Hessian’s eigenvalues.
worst chord violation over 1000 trials: -2.46e-03
worst tangent violation over 1000 trials: -4.95e-02
Hessian eigenvalues: [ 6.273 18.6557]Both worst “violations” are negative, and X^\top X has positive eigenvalues. Sampling can only ever refute convexity; the Hessian is the one that proves it.
03
Jensen’s inequality
the chord, lifted to expectations
Jensen
The chord inequality, with weights read as a probability distribution and then as an expectation:
f(\mathbb{E}[X]) \;\le\; \mathbb{E}[f(X)].
Proof in two lines: take a subgradient at \boldsymbol{\mu}=\mathbb{E}[X] (the supporting-hyperplane granted fact supplies one), so f(X) \ge f(\boldsymbol{\mu}) + \mathbf{g}^\top(X-\boldsymbol{\mu}) pointwise; take expectations, the linear term has mean zero.
The graph bends up, so spreading X out can only push \mathbb{E}[f(X)] above f(\mathbb{E}[X]). For concave f the inequality flips.
Jensen
Apply Jensen to the right convex (or concave) function and out fall the staples of the probabilistic chapters:
E[exp(X)] = 1.6466 vs exp(E[X]) = 0.9998 (theory: sqrt(e) = 1.6487 vs 1)
AM >= GM in all trials: True, smallest AM/GM ratio = 1.0002
min KL(p||q) = 0.1331 >= 0, max |KL(p||p)| = 0.0e+00A Jensen gap of \sqrt{e} vs 1 that no sampling closes (it is geometry, not noise), AM \ge GM in every draw, and KL nonnegative throughout.
04
Why it matters
local equals global, and a global rate
Global guarantees
If a better point existed anywhere, the chord toward it would already descend inside your neighborhood. So “no local improvement” means “no improvement anywhere,” and stationary \Rightarrow global.
Global guarantees
The same gradient-descent loop and step size, from 500 random starts, on a convex bowl and on a tilted double well:
convex bowl: all 500 runs end at x = 0.500000 (spread 6.7e-16)
double well: 271 runs -> x = -1.0575 (g = -0.5148)
229 runs -> x = 0.9304 (g = 0.4833)The bowl collapses all 500 runs onto one point (spread at machine epsilon). The double well splits them across two basins. Convexity is a property of the objective, and it alone decides whether the start matters.
Global guarantees
Chaining the descent lemma through the first-order lens upgrades “eventually flat” to “the optimum, this fast”:
f(\mathbf{x}_k) - f^\star \le \frac{L\,\|\mathbf{x}_0 - \mathbf{x}^\star\|^2}{2k}
and with strong convexity it sharpens to a geometric rate
f(\mathbf{x}_k) - f^\star \le \bigl(1 - \tfrac{\mu}{L}\bigr)^{k}\bigl(f(\mathbf{x}_0) - f^\star\bigr).
Both bounds are dimension-free: a million parameters cost no more iterations than two. This is why first-order methods scale.
Global guarantees
Gradient descent at \eta = 1/L on the least-squares toy, with \mu and L read off the eigenvalues of X^\top X:
mu = 6.2730, L = 18.6557: the theorem promises contraction <= 1 - mu/L = 0.6637
measured per-step contraction of f - f*: [0.4406 0.4406 0.4406 0.4406]
worst step: 0.4406 (the bound holds at every step)The bound holds at every step, and the measured 0.4406 is no accident: the slow mode’s distance contracts by exactly 1-\mu/L, and the value gap, quadratic in distance, contracts by its square, (1-\mu/L)^2 = 0.4406. Valid for the class; loose by a square on quadratics.
05
Recognizing convexity
a calculus, and log-sum-exp
Recognizing convexity
Most convexity proofs are assembled from parts. Four operations preserve convexity:
nonnegative sums · affine pre-composition · pointwise max · monotone convex composition
So in one line each: the hinge is a max of affines; \ell_1 a sum of such maxes; logistic is the convex \log(1+e^t) after an affine map; ridge-anything is strongly convex. The one missing rule, non-monotone inner maps, is where deep networks exit the theory.
Recognizing convexity
The function behind every softmax cross-entropy:
\nabla\,\mathrm{lse} = \mathrm{softmax} = \mathbf{s}, \qquad \nabla^2\mathrm{lse} = \mathrm{diag}(\mathbf{s}) - \mathbf{s}\mathbf{s}^\top.
That Hessian is the covariance of a one-hot draw from \mathbf{s}, so \mathbf{v}^\top\nabla^2\mathrm{lse}\,\mathbf{v} = \mathrm{Var}(v_I) \ge 0: PSD, hence \mathrm{lse} is convex.
The one zero eigenvalue (direction \mathbf{1}) is the shift invariance behind the stable softmax. Its conjugate is negative entropy.
Recognizing convexity
Eigenvalues of the analytic Hessian, the covariance identity by Monte Carlo, and the predicted flat direction:
eigenvalues of H: [-0. 0.008103 0.052497 0.063313 0.099101 0.284324]
max |H - covariance of one-hot samples| = 0.0006
max |H @ 1| = 1.8e-16 (flat shift direction)Every eigenvalue is nonnegative down to one numerical zero, and 200k one-hot draws reproduce the analytic Hessian: it really is a covariance you can sample from.
Recognizing convexity
\mathrm{prox}_f(\mathbf{z}) moves toward lower f but pays quadratically for straying from \mathbf{z}: a projection, generalized from sets to functions. For \lambda|x| the subgradient criterion solves it in closed form: soft-thresholding,
\mathrm{prox}_{\lambda|\cdot|}(z) = \mathrm{sign}(z)\,\max(|z| - \lambda,\, 0),
and alternating a gradient step on the smooth part with a prox on the kink is ISTA, which keeps gradient descent’s O(1/k) rate:
max |prox - brute-force argmin| = 8.9e-16
w after 20 ISTA steps: [ 1.964 0. 0. -1.462 0. 0. 0. 0.963]
exact zeros: 5 of 8 coordinatesFive of eight coordinates are exactly zero, snapped there by the \max, which no smooth gradient step can do. This is how \ell_1 sparsifies where \ell_2 only shrinks.
06
Reality check
deep nets are non-convex, and what survives
Reality check
Take f(a,b) = (ab-1)^2, a two-weight linear model. Its minima form the hyperbola \{ab=1\}, a non-convex set, but minimizer sets of convex functions are convex. So f cannot be convex.
Real networks inherit this: permuting hidden units leaves the function unchanged, scattering equivalent minima everywhere. Deep architectures rule out convexity structurally.
The minimizers (1,1) and (-1,-1) average to the origin, where f = 1 > 0: not a minimum.
Reality check
The linear-rate proof never used convexity in its second half, only
\tfrac12\|\nabla f\|^2 \ge \mu\,(f - f^\star) \qquad \textrm{(Polyak--Łojasiewicz).}
PL says the gradient is small only where the value is near-optimal, so flat spots sit only at the bottom: linear convergence with no convexity.
PL constant on [-5, 5]: mu = 0.1755
min f'' = -4.00 (f is not convex)
successive gap ratios: [0.6293 0.2907 0.25 0.25 0.25 ]The gap contracts by a constant factor on x^2 + 3\sin^2 x, whose Hessian dips to -4. One current account of why huge overparameterized nets train.
Reality check
When many minima exist, gradient descent chooses systematically: it has an implicit bias:
Which minimum an optimizer prefers is part of what a trained model is.
Wrap-up
Convex theory is the idealization the working optimizer approximates, and its instruments (descent lemma, PL, rates) we carry into non-convex territory.