Dive into Deep Learning · §26.3
Convexity and global optimization guarantees
convex sets · equivalent characterizations · Jensen’s inequality · convergence rates.
Motivation
For a convex problem the guarantees are global:
Convexity applies directly to several subproblems in deep learning (the softmax loss, the regularizers, projections, the SVM dual), while nonlinear multilayer parameterizations generally lose these global guarantees.
All code here is a few lines of plain NumPy: no framework, no GPU.
01
Convex sets
the segment between any two points stays inside
Convex sets
A set C is convex if every segment between two of its points never leaves it: \theta\mathbf{x} + (1 - \theta)\mathbf{y} \in C for all \mathbf{x}, \mathbf{y} \in C, \theta \in [0, 1].
Left stays inside; the crescent’s chord passes outside. Right: the simplex and a half-space.
Convex sets
Intersections preserve convexity (a point in every C_i is a point in their meet). Unions do not: [0,1]\cup[2,3] has a chord that escapes.
This one fact certifies the catalog:
The catalog: half-spaces, norm balls, the simplex, the PSD cone.
Affine maps and convex hulls round it out.
02
Equivalent characterizations of convexity
chord, tangent, Hessian, and subgradients at nondifferentiable points
Convex functions
Two equivalent geometric characterizations: the chord joining two points lies above the graph; the tangent at any point lies below it. A single gradient at \mathbf{x} thus certifies every \mathbf{y}, however far.
Convex functions
f(\theta\mathbf{x}+(1-\theta)\mathbf{y}) \le \theta f(\mathbf{x}) + (1-\theta) f(\mathbf{y})
is equivalent, for smooth f, to the tangent under-estimator, and to
\nabla^2 f(\mathbf{x}) \succeq 0 \quad \textrm{everywhere.}
Use the characterization suited to the available structure: chords require no derivatives, first-order inequalities support optimization proofs, and the Hessian tests smooth losses.
Strong convexity adds a curvature lower bound, \nabla^2 f \succeq \mu I: positive minimum curvature, and \kappa = L/\mu is the condition number.
Convex functions
\ell_1 and the hinge have kinks, so no gradient exists there. A subgradient \mathbf{g} keeps the under-estimate anyway:
f(\mathbf{y}) \ge f(\mathbf{x}) + \mathbf{g}^\top(\mathbf{y}-\mathbf{x}).
At a corner the subdifferential contains multiple slopes: \partial|x|(0) = [-1, 1], and the zero-slope member is an optimality certificate: \mathbf{x}^\star minimizes f iff \mathbf{0} \in \partial f(\mathbf{x}^\star).
Subgradients extend Jensen’s inequality and local-global optimality arguments to convex nondifferentiable functions.
Convex functions
On the least-squares loss in two weights: a thousand random chords, a thousand random tangents, and the Hessian’s eigenvalues.
worst chord violation over 1000 trials: -2.46e-03
worst tangent violation over 1000 trials: -4.95e-02
Hessian eigenvalues: [ 6.273 18.6557]Both maximum residuals are negative, and X^\top X has positive eigenvalues. Sampling can only ever refute convexity; the constant PSD Hessian proves convexity in this example.
03
Jensen’s inequality
the chord, lifted to expectations
Jensen
The chord inequality, with weights read as a probability distribution and then as an expectation:
f(\mathbb{E}[X]) \;\le\; \mathbb{E}[f(X)].
Proof: take a subgradient at \boldsymbol{\mu}=\mathbb{E}[X] (the supporting-hyperplane granted fact supplies one), so f(X) \ge f(\boldsymbol{\mu}) + \mathbf{g}^\top(X-\boldsymbol{\mu}) pointwise; take expectations, the linear term has mean zero.
Dispersion of X can increase \mathbb{E}[f(X)] relative to f(\mathbb{E}[X]). For concave f the inequality flips.
Jensen
Applying Jensen’s inequality to suitable convex or concave functions gives:
E[exp(X)] = 1.6466 vs exp(E[X]) = 0.9998 (theory: sqrt(e) = 1.6487 vs 1)
AM >= GM in all trials: True, smallest AM/GM ratio = 1.0002
min KL(p||q) = 0.1331 >= 0, max |KL(p||p)| = 0.0e+00The population Jensen gap is \sqrt{e} versus 1; sampling affects only its estimation error, AM \ge GM in every draw, and KL nonnegative throughout.
04
Why it matters
local equals global, and a global rate
Global guarantees
If a better point existed anywhere, the chord toward it would contain points with lower values in every local neighborhood. Therefore local optimality implies global optimality, and stationary \Rightarrow global.
Global guarantees
The same gradient-descent loop and step size are applied from 500 random starts, on a convex bowl and on a tilted double well:
convex bowl: all 500 runs end at x = 0.500000 (spread 6.7e-16)
double well: 271 runs -> x = -1.0575 (g = -0.5148)
229 runs -> x = 0.9304 (g = 0.4833)For this stable step, all 500 quadratic runs converge to one point, while the double-well runs converge to two distinct local minima. Convexity rules out nonglobal local minima; convergence of the update still requires its smoothness and step-size conditions.
Global guarantees
Combining the descent lemma with the first-order convexity condition gives a global objective-value rate:
f(\mathbf{x}_k) - f^\star \le \frac{L\,\|\mathbf{x}_0 - \mathbf{x}^\star\|^2}{2k}
and with strong convexity it sharpens to a geometric rate
f(\mathbf{x}_k) - f^\star \le \bigl(1 - \tfrac{\mu}{L}\bigr)^{k}\bigl(f(\mathbf{x}_0) - f^\star\bigr).
The displayed bounds contain no explicit dimension factor. Dimension can still affect L, \mu, gradient cost, and the validity of the assumptions.
Global guarantees
Gradient descent at \eta = 1/L on the least-squares toy, with \mu and L read off the eigenvalues of X^\top X:
mu = 6.2730, L = 18.6557: the theorem promises contraction <= 1 - mu/L = 0.6637
measured per-step contraction of f - f*: [0.4406 0.4406 0.4406 0.4406]
worst step: 0.4406 (the bound holds at every step)The bound holds at every step, and the measured 0.4406 follows from the quadratic recurrence: the slow mode’s distance contracts by exactly 1-\mu/L, and the value gap, quadratic in distance, contracts by its square, (1-\mu/L)^2 = 0.4406. The class-wide bound is valid but differs by a square on this quadratic.
05
Recognizing convexity
a calculus, and log-sum-exp
Recognizing convexity
Many convexity proofs use closure rules. Four operations preserve convexity:
nonnegative sums · affine pre-composition · pointwise max · monotone convex composition
The hinge is a maximum of affine functions; \ell_1 is a sum of such maxes; logistic is the convex \log(1+e^t) after an affine map; adding ridge regularization to a convex loss gives strong convexity. Non-monotone nonlinear inner maps do not satisfy this rule, so standard deep network parameterizations are generally nonconvex.
Recognizing convexity
The function behind every softmax cross-entropy:
\nabla\,\mathrm{lse} = \mathrm{softmax} = \mathbf{s}, \qquad \nabla^2\mathrm{lse} = \mathrm{diag}(\mathbf{s}) - \mathbf{s}\mathbf{s}^\top.
That Hessian is the covariance of a one-hot draw from \mathbf{s}, so \mathbf{v}^\top\nabla^2\mathrm{lse}\,\mathbf{v} = \mathrm{Var}(v_I) \ge 0: PSD, hence \mathrm{lse} is convex.
The one zero eigenvalue (direction \mathbf{1}) is the shift invariance behind the stable softmax. Its conjugate is negative entropy.
Recognizing convexity
Eigenvalues of the analytic Hessian, the covariance identity by Monte Carlo, and the predicted flat direction:
eigenvalues of H: [-0. 0.008103 0.052497 0.063313 0.099101 0.284324]
max |H - covariance of one-hot samples| = 0.0006
max |H @ 1| = 1.8e-16 (flat shift direction)Every eigenvalue is nonnegative down to one numerical zero, and 200k one-hot draws reproduce the analytic Hessian: the empirical covariance agrees with the analytic Hessian.
Recognizing convexity
\mathrm{prox}_f(\mathbf{z}) balances lower values of f against a quadratic penalty for moving away from \mathbf{z}: a projection, generalized from sets to functions. For \lambda|x| the subgradient criterion solves it in closed form: soft-thresholding,
\mathrm{prox}_{\lambda|\cdot|}(z) = \mathrm{sign}(z)\,\max(|z| - \lambda,\, 0),
and alternating a gradient step on the smooth part with a proximal update on the nonsmooth part is ISTA, which keeps gradient descent’s O(1/k) rate:
max |prox - brute-force argmin| = 8.9e-16
w after 20 ISTA steps: [ 1.964 0. 0. -1.462 0. 0. 0. 0.963]
exact zeros: 5 of 8 coordinatesFive of eight coordinates are exactly zero because of the \max thresholding operation; an ordinary smooth gradient step does not produce this result. This is how \ell_1 sparsifies where \ell_2 only shrinks.
06
Nonconvex models
structural nonconvexity and rates under the PL condition
Nonconvex models
Take f(a,b) = (ab-1)^2, a two-weight linear model. Its minima form the hyperbola \{ab=1\}, a non-convex set, but minimizer sets of convex functions are convex. So f cannot be convex.
In standard hidden-layer parameterizations, permuting units leaves the computed function unchanged and produces distinct equivalent parameters. Their averages need not remain minima, so the parameter-space loss is generally nonconvex.
The minimizers (1,1) and (-1,-1) average to the origin, where f = 1 > 0: not a minimum.
Nonconvex models
The linear-rate proof never used convexity in its second half, only
\tfrac12\|\nabla f\|^2 \ge \mu\,(f - f^\star) \qquad \textrm{(Polyak--Łojasiewicz).}
PL says the gradient is small only where the value is near-optimal, so stationary points attain the global minimum value, giving linear convergence with no convexity.
PL constant on [-5, 5]: mu = 0.1755
min f'' = -4.00 (f is not convex)
successive gap ratios: [0.6293 0.2907 0.25 0.25 0.25 ]The gap contracts by a constant factor on x^2 + 3\sin^2 x, whose Hessian dips to -4. Local PL analyses apply in specified overparameterized regimes, not to neural networks in general.
Nonconvex models
When many minima exist, gradient descent can select a systematic solution; this is its implicit bias:
The selected minimum can affect the resulting model and its generalization.
Wrap-up
Convex theory supplies global comparison results; the descent lemma, PL condition, and related rates also support analyses of specified nonconvex regimes.