Divergences and Distances Between Distributions

Dive into Deep Learning · §27.2

Choose your divergence, inherit its failure modes
f-divergences, optimal transport, and the objective map.

Which notion of closeness?

Motivation

Every generative model minimizes “distance from model to data.” The choice of divergence fixes the objective, the gradients, and the failure modes. Three families:

f-divergences: a convex f of the ratio p/q,
integral probability metrics: the largest mean gap,
optimal transport: the cheapest way to move mass.

The f-divergence family

one convex generator, many divergences

One template, one proof

The f-divergence

D_f(P\|Q) = \mathbb{E}_{x\sim Q}\bigl[f(p/q)\bigr] for convex f with f(1)=0.

Proof that D_f \ge 0: \mathbb{E}_Q[f(p/q)] \ge f(\mathbb{E}_Q[p/q]) = f(1) = 0. Jensen, once, for the whole family. \blacksquare

Convexity of f is exactly what buys non-negativity.

One more Jensen buys data processing: pushing P and Q through any channel K only loses distinguishability, D_f(PK\,\|\,QK) \le D_f(P\|Q), for every f at once.

A gallery of generators

The gallery

One curve f per divergence:

KL: u\log u · reverse KL: -\log u
Pearson \chi^2: (u-1)^2
Hellinger: (\sqrt u-1)^2
total variation: \tfrac12|u-1|
Jensen–Shannon (symmetric)
\alpha-family: a dial from reverse to forward KL; its log-transform is Rényi’s D_\alpha

Near P=Q all agree, \propto f''(1)\,\chi^2; they differ only far apart.

The gallery, numerically

The numbers

Each generator gives a different number on the same pair; KL is asymmetric, JS symmetric and capped below \log 2:

KL           D_f(P||Q) = 0.3961   D_f(Q||P) = 0.3653
reverse KL   D_f(P||Q) = 0.3653   D_f(Q||P) = 0.3961
chi^2        D_f(P||Q) = 1.0133   D_f(Q||P) = 0.8000
Hellinger^2  D_f(P||Q) = 0.1862   D_f(Q||P) = 0.1862
TV           D_f(P||Q) = 0.4000   D_f(Q||P) = 0.4000
JS           D_f(P||Q) = 0.0911   D_f(Q||P) = 0.0911
JS via the mixture formula: 0.0911

Duality: divergence as a game

the Fenchel conjugate, the f-GAN, and mode geometry

A divergence is a game against a critic

Variational form

A convex f is the envelope of its tangents, so for any critic T:

D_f(P\|Q) \ge \mathbb{E}_P[T] - \mathbb{E}_Q[f^*(T)],

expectations only, no densities. The ordinary GAN is the Jensen–Shannon case.

Tight only at the right critic

The f-GAN

For the \chi^2 generator the bound hits the true divergence exactly at the optimal critic T^\star = f'(p/q), and any other critic falls short:

exact chi^2(P||Q)           = 1.0133
bound at the optimal critic = 1.0133
bound at a perturbed critic (scale 0.5): 0.9678
bound at a perturbed critic (scale 1.0): 0.3661
bound at a perturbed critic (scale 2.0): 0.9600

An undertrained critic biases the estimate low; the adversary’s job is to make the bound tight.

Forward vs. reverse KL

Mode geometry

Same target, two optima: forward KL (maximum likelihood) must cover every mode; reverse KL (variational inference) hugs one, paying \log(1/0.7) nats for the mass it drops:

forward KL             mu = -0.800, sigma = 1.929, KL = 0.558 nats
reverse KL             mu = -1.998, sigma = 0.603, KL = 0.356 nats
reverse KL, 2nd start  mu = +1.995, sigma = 0.607, KL = 1.202 nats

Metrics: when densities fail

total variation, MMD, and optimal transport

Total variation and Pinsker

The strongest test

\mathrm{TV}(P,Q) = \sup_A|P(A)-Q(A)| = \tfrac12\|p-q\|_1. Hand a tester one sample: the best test succeeds with probability \tfrac12(1+\mathrm{TV}); in the cryptographers’ convention, an advantage of exactly \mathrm{TV}.

Pinsker: \mathrm{TV} \le \sqrt{D_{\mathrm{KL}}/2}, so a small KL silences every test at once.

Pinsker’s constant is sharp

A stress test

Across ten thousand random pairs the ratio \mathrm{TV}/\sqrt{D_{\mathrm{KL}}/2} never exceeds 1, and near-fair coins push it to 0.999999:

max TV / sqrt(KL/2) over 10,000 random pairs: 0.9926
coins 1/2 vs 1/2+0.1: TV / sqrt(KL/2) = 0.989881
coins 1/2 vs 1/2+0.01: TV / sqrt(KL/2) = 0.999900
coins 1/2 vs 1/2+0.001: TV / sqrt(KL/2) = 0.999999

The factor \tfrac12 cannot be improved.

IPMs and the kernel trick

Sample-only

An integral probability metric is \sup_{f\in\mathcal F}\mathbb{E}_P[f]-\mathbb{E}_Q[f]. Over an RKHS ball it becomes MMD, a closed-form kernel expectation with no critic and no densities:

MMD^2, same distribution : +0.00054
MMD^2, mean shifted 0.5  : +0.05890

A half-\sigma shift jumps \mathrm{MMD}^2 by two orders of magnitude.

Optimal transport and Wasserstein

Moving mass

On disjoint supports every f-divergence is constant (zero gradient) while W_1 still moves smoothly. That is the WGAN.

Dual (Kantorovich–Rubinstein): the 1-Lipschitz critics. In one dimension, W_1 = \int|F_P - F_Q|.

One integral replaces a linear program

Wasserstein-1

The CDF formula and a 36-variable transport LP agree to ten digits:

W1 via the CDF formula : 1.7000000000
W1 via the primal LP   : 1.7000000000

Beyond one dimension no such formula exists, and the LP has n^2 variables.

Sinkhorn and entropic regularization

Entropic OT

Sinkhorn’s row/column rescalings solve the -\varepsilon H(\gamma)-regularized plan on a GPU. At \varepsilon=1 the blurred plan costs 1.77 vs 1.70; shrinking \varepsilon sharpens it onto the LP’s never-crossing staircase:

epsilon = 1.00: entropic cost = 1.7700   (unregularized LP: 1.7000)
epsilon = 0.10: entropic cost = 1.7000   (unregularized LP: 1.7000)
epsilon = 0.02: entropic cost = 1.7000   (unregularized LP: 1.7000)

Scores and the objective map

the normalizer-free divergence, and the unifying table

The score never sees the normalizer

Fisher divergence

s_P(\mathbf x) = \nabla_{\mathbf x}\log p drops the intractable Z entirely, since \nabla\log Z = 0. The Fisher divergence compares score fields, the basis of score matching and diffusion.

Normalizer-blindness, numerically

The point

Rescaling the density leaves the score unchanged to floating-point dust, and the Gaussian Fisher divergence matches its closed form:

mixture score, max |analytic - numerical|: 3.52e-04
score change from rescaling p by 2.7    : 1.82e-12
Fisher divergence N(0,1)||N(1,1): quadrature 0.500000, closed form 0.500000

This is why score matching works where density estimation cannot.

Stein’s identity and goodness of fit

KSD

For any smooth f, \mathbb{E}_P[f' + f\,s_P] = 0: a fingerprint of P that needs only its score. Violating it certifies the sample is not from P:

rng = np.random.default_rng(7)
z = rng.standard_normal(1_000_000)
# Stein operator for P = N(0,1):  (A_P f)(x) = f'(x) - x f(x)
for name, f, fprime in [('x^3', lambda t: t ** 3, lambda t: 3 * t ** 2),
                        ('sin x', np.sin, np.cos)]:
    val = (fprime(z) - z * f(z)).mean()
    print(f"E[ f'(Z) - Z f(Z) ] for f(x) = {name}: {val:+.4f}")

E[ f'(Z) - Z f(Z) ] for f(x) = x^3: +0.0020
E[ f'(Z) - Z f(Z) ] for f(x) = sin x: +0.0005

The kernel Stein discrepancy turns this into a test; its descent direction is SVGD.

One map for every objective

The capstone

MLE/flows → forward KL · VAE → reverse KL · GAN → JS · f-GAN → any f · WGAN → W_1 · MMD-GAN → MMD · diffusion → Fisher · SVGD → KSD.

Choose the divergence; you have chosen the objective, and inherited its failure modes.

Recap

Wrap-up

f-divergence = \mathbb{E}_Q[f(p/q)]; Jensen gives D_f \ge 0 and its DPI.
The \alpha/Rényi family sweeps reverse \to forward KL.
Fenchel duality turns any f into a critic game (the f-GAN); GAN is the JS case.
Forward KL covers modes; reverse KL hugs one.

TV is the best single-sample advantage; Pinsker bounds it by \sqrt{\mathrm{KL}/2}.
W_1 remains informative on disjoint supports (WGAN), though not necessarily smooth; Sinkhorn approaches the unregularized transport problem as regularization vanishes.
The score drops Z; the Fisher/Stein row powers diffusion and SVGD.

Next: mutual information, a divergence from independence, put to work in representation learning.