Constrained Optimization and Duality

Dive into Deep Learning · §26.4

Conditions and algorithms for constrained optimization
Lagrange multipliers · KKT · projections · duality.

First-order conditions under constraints

Motivation

At an unconstrained minimum the gradient vanishes. At a constrained minimum it need not, but there is no feasible descent direction: no move that both stays feasible and lowers f.

Equality constraints give Lagrange multipliers, \nabla f = -\nu\,\nabla g.
Inequality constraints give the KKT conditions.
Convex feasible sets make it an algorithm: projected gradient descent.

An optimal multiplier measures sensitivity to relaxation of its constraint.

Lagrange multipliers

equality constraints and tangency

Feasible Stationarity Implies Tangency

Lagrange

If a level set of f crosses the constraint, \nabla f has a component along the feasible direction, which gives local descent and lowers f. At a regular constrained optimum, the curves are tangent, so their normals are parallel:

\nabla f(\mathbf{x}^\star) + \nu^\star\,\nabla g(\mathbf{x}^\star) = \mathbf{0}.

Valid wherever \nabla g(\mathbf{x}^\star) \neq \mathbf{0}, the constraint qualification.

The Lagrangian Combines the Conditions

Lagrange

Parallel gradients and feasibility become joint stationarity of one function with an additional variable:

\mathcal{L}(\mathbf{x}, \nu) = f(\mathbf{x}) + \nu\, g(\mathbf{x}), \qquad \nabla_{\mathbf{x}}\mathcal{L} = \mathbf{0},\;\; \partial_\nu \mathcal{L} = g = 0.

Closest point on a hyperplane \mathbf{a}^\top\mathbf{x} = b: stationarity places \mathbf{x} along the normal \mathbf{a}, giving \mathbf{x}^\star = \tfrac{b}{\|\mathbf{a}\|^2}\,\mathbf{a}.

Already a shadow price: \partial p^\star/\partial b = -\nu^\star. The multiplier tracks how the optimum moves when the constraint moves.

The KKT conditions

inequalities and the active set

Active and Inactive Inequality Constraints

KKT

With g_i(\mathbf{x}) \le 0, each constraint is inactive (g_i < 0, locally irrelevant) or active (g_i = 0, contributing like an equality). At the optimum -\nabla f is a nonnegative combination of the active outward normals, in their cone.

An inequality multiplier satisfies \lambda_i \ge 0, so active constraint normals enter stationarity with nonnegative coefficients.

The four KKT conditions

KKT

Stationarity: \nabla f + \sum_i \lambda_i \nabla g_i + \sum_j \nu_j \nabla h_j = \mathbf{0}

Primal feas.: g_i \le 0,\;\; h_j = 0

Dual feas.: \lambda_i \ge 0

Comp. slackness: \lambda_i\, g_i = 0

Complementary slackness finds the active set: for each i, either g_i = 0 (active, \lambda_i free) or \lambda_i = 0 (slack, priced at zero). The pattern of zero multipliers identifies which constraints contribute to stationarity.

Convex f, g_i and affine h_j: a KKT point is a global minimum, by a direct convexity argument. Thus KKT is sufficient in this setting.

Projections

projection-based constrained updates

Projected Gradient Descent

Projections

For nonempty closed convex C, \Pi_C(\mathbf{y}) is the unique nearest feasible point and is nonexpansive (\|\Pi_C\mathbf{x} - \Pi_C\mathbf{y}\| \le \|\mathbf{x} - \mathbf{y}\|), which supports projected-gradient analysis:

\mathbf{x}_{t+1} = \Pi_C\!\left(\mathbf{x}_t - \eta\,\nabla f(\mathbf{x}_t)\right).

Its fixed points are constrained stationary points. They are global optima when f is convex; otherwise no global conclusion follows.

Non-negativity and max-norm parameter constraints give PGD steps. Gradient norm clipping projects the gradient, but is not generally PGD for a parameter constraint.

Simplex projection = sort, shift, clip

Projections

Projecting onto \{\mathbf{x} \ge 0,\, \sum_i x_i = 1\} is a QP the KKT conditions solve in closed form up to one threshold \tau:

x_i^\star = \max(y_i - \tau,\, 0), \qquad \textstyle\sum_i \max(y_i - \tau,\, 0) = 1.

A common shift followed by clipping gives sparsemax: genuinely sparse attention weights, with complementary slackness determining the zeros.

x* = [0.0676 0.     0.5823 0.0467 0.     0.3034]  sum = 1.000000
KKT residuals: stationarity 5.6e-17 | comp. slack 0.0e+00 | dual feas. 0.0e+00
f(x*) = 0.158962 <= best random feasible 0.162139

Two coordinates are exactly zero, corresponding to active sign constraints; every KKT residual is at machine precision.

Duality

lower bounds and sensitivity

The Dual Function Gives Lower Bounds

Duality

Minimizing the Lagrangian over \mathbf{x} defines a function of the multipliers:

g(\boldsymbol{\lambda}, \boldsymbol{\nu}) = \inf_{\mathbf{x}}\, \mathcal{L}(\mathbf{x}, \boldsymbol{\lambda}, \boldsymbol{\nu}).

Always concave (an infimum of affine functions), even for a non-convex primal.
Weak duality: d^\star \le p^\star always. Every dual point certifies a lower bound.

Strong duality and shadow prices

Duality

Slater: convex problem + one strictly feasible point \Rightarrow d^\star = p^\star, the gap closes.

Maximizing the concave dual function is a convex optimization problem, and under strong duality its optimum equals the primal optimum.

Shadow price:

\lambda_i^\star = -\frac{\partial p^\star}{\partial u_i}.

Relax constraint i by a unit and the optimum improves by \lambda_i^\star.

Slack constraints have zero shadow price; binding constraints can have a nonzero price.

Strong duality is a saddle point

Duality

The primal value is \inf_{\mathbf{x}} \sup_{\boldsymbol{\lambda} \succeq 0} \mathcal{L}: a violated constraint makes the supremum +\infty. The dual reverses the optimization order. Weak duality gives \sup\inf \le \inf\sup, and strong duality makes the two values equal:

\mathcal{L}(\mathbf{x}^\star, \boldsymbol{\lambda}) \;\le\; \mathcal{L}(\mathbf{x}^\star, \boldsymbol{\lambda}^\star) \;\le\; \mathcal{L}(\mathbf{x}, \boldsymbol{\lambda}^\star).

This is the formulation used in minimax training: GANs and adversarial training are \min_{\boldsymbol{\theta}}\max_{\boldsymbol{\phi}} problems, and a saddle point makes the min–max and max–min values equal. Primal–dual methods descend in \mathbf{x} and ascend in \boldsymbol{\lambda}; their convergence requires additional assumptions.

Penalty and Norm-Constrained Forms

Duality at work

The penalty and constraint forms of \ell_2 regularization are linked by the Lagrangian:

\underbrace{\min_{\mathbf{w}}\, L(\mathbf{w}) + \lambda\|\mathbf{w}\|^2}_{\text{penalty}} \;\Longleftrightarrow\; \underbrace{\min_{\mathbf{w}}\, L(\mathbf{w}) \;\text{s.t.}\; \|\mathbf{w}\|^2 \le r^2}_{\text{constraint}}

Same stationarity equation, \lambda playing the multiplier: \lambda = -\partial p^\star/\partial(r^2) prices the weight budget. Verified on ridge regression, where the theorem is exact:

 lambda    r = |w_pen|   |w_con - w_pen|   recovered multiplier
   0.10       0.4460         9.23e-17             0.1000
   1.00       0.4235         9.64e-17             1.0000
  10.00       0.2840         5.55e-17            10.0000

The same coefficient appears as weight decay, the multiplier of a norm constraint, and the precision of a Gaussian prior.

Worked: the SVM dual

Duality at work

Eliminate (\mathbf{w}, b) from the max-margin Lagrangian and the dual is a concave QP over the orthant that can be solved by projected gradient ascent:

\max_{\boldsymbol{\alpha} \succeq 0}\;\; \mathbf{1}^\top \boldsymbol{\alpha} - \tfrac12 \boldsymbol{\alpha}^\top Q\, \boldsymbol{\alpha}, \qquad \boldsymbol{\alpha} \leftarrow \max(\mathbf{0},\, \boldsymbol{\alpha} + \eta\,(\mathbf{1} - Q\boldsymbol{\alpha})).

alpha* = [0.1309 0.0732 0.     0.     0.     0.2442 0.     0.3885]
w* = [0.5714 0.5714]  b* = -0.4286  separates data: True
margins y_i (w^T x_i + b) = [1.     1.     2.1429 2.1429 1.2857 1.     2.4286 1.    ]
KKT: comp. slack 3.0e-16 | worst primal feas. 7.8e-16
primal = 0.418367, dual = 0.418367, gap = 4.4e-16

The four nonzero \alpha_i are exactly the points at margin 1, the support vectors. The primal and dual values agree to 10^{-16}.

Worked: water-filling

Duality at work

Allocate power P across noisy channels. KKT gives a common level w for every channel receiving power; complementary slackness assigns zero power to sufficiently noisy channels:

p_i^\star = \max(w - n_i,\, 0), \qquad \textstyle\sum_i \max(w - n_i, 0) = P.

The multiplier \mu = 1/w equals the marginal value of power; the run below confirms it by finite differences, 0.697674 both ways.

An Explicit Duality Gap

Duality at work

Weak duality always holds; strong duality can fail. Minimize the concave f_0 = -x^2 on [0,1] s.t. x \le \tfrac12:

p^\star = -\tfrac14, \qquad d^\star = -\tfrac12.

A strictly feasible point exists, yet the gap is real: Slater certifies strong duality only for convex problems.

p* = -0.2500,  d* = -0.5000 at lambda = 1.00,  gap = 0.2500
weak duality g(lambda) <= p* everywhere: True

Constraints replace zero gradients by feasible stationarity

Wrap-up

A constrained optimum has no feasible descent direction, made precise by Lagrange and KKT; complementary slackness finds the active set.
Projections onto convex sets are nonexpansive; projected GD is gradient descent followed by projection back to feasibility.

The dual is always concave and always a lower bound; Slater closes the gap; multipliers are shadow prices.
The SVM dual and water-filling provide computable examples; non-convexity can leave a positive gap.

Many deep-learning subproblems, including projections, clipped updates, and last-layer fits, are convex optimization problems.