Entropy, Cross-Entropy, and KL Divergence

Dive into Deep Learning · §27.1

The number your loss prints is a code length
entropy, cross-entropy, and KL divergence.

Three numbers in every loss

Motivation

The scalar your training loop reports is a code length in nats (Shannon, 1948). It splits into three pieces:

entropy H(P): the irreducible floor,
cross-entropy \mathrm{CE}(P,Q): what the model pays,
KL D_{\mathrm{KL}}(P\|Q): the waste you train away.

Surprise and entropy

self-information, Shannon’s H, the uniform maximum

Rare events are surprising

Self-information

I(x) = -\log p(x) is the only measure that is zero for a certain event, decreasing in p, and additive over independent events; additivity is exactly what forces a logarithm.

The fair coin carries \ln 2 \approx 0.693 nats; a one-in-a-million event, \approx 13.8.

Entropy averages the surprise

Entropy

H(P) = \mathbb{E}_P[-\log p] = -\sum_x p(x)\log p(x), with the convention 0\log 0 = 0; it is the expected length of the best possible code for P:

import numpy as onp
def entropy(p):
    """Entropy of a probability vector, in nats."""
    p = onp.asarray(p, dtype=float)
    ent = -p * onp.log(p)
    # `nansum` encodes the convention 0 log 0 = 0
    return float(onp.nansum(ent))

entropy(onp.array([0.1, 0.5, 0.1, 0.3]))

1.1682824501765625

Less than \ln 4 \approx 1.386 nats: this distribution is not uniform.

Uncertainty peaks at the uniform law

The maximum

On k outcomes, 0 \le H(P) \le \log k, with the maximum exactly at the uniform distribution.

Proof. H = \mathbb{E}[\log\tfrac{1}{p}] \le \log\mathbb{E}[\tfrac1p] = \log m \le \log k, where m counts the outcomes with p_i>0, by Jensen on the strictly concave \log. Equality forces 1/p constant and m=k: the uniform law, exactly. \blacksquare

Cross-entropy and KL

the gap you train away: Gibbs’ inequality

KL divergence: the price of the wrong model

Relative entropy

D_{\mathrm{KL}}(P\|Q) = \mathbb{E}_{x\sim P}\bigl[\log p(x)/q(x)\bigr]: the extra nats from coding P’s data with Q’s code. Asymmetric, and +\infty where q(x)=0 but p(x)>0:

import numpy as onp
def kl_divergence(p, q):
    """KL(P || Q) for two probability vectors, in nats."""
    p, q = onp.asarray(p, dtype=float), onp.asarray(q, dtype=float)
    kl = p * onp.log(p / q)
    return float(onp.nansum(kl))

Gibbs’ inequality

The one fact

D_{\mathrm{KL}}(P\|Q) \ge 0, with equality iff P=Q.

Proof. D_{\mathrm{KL}} = \mathbb{E}_P[-\log\tfrac{q}{p}] \ge -\log\mathbb{E}_P[\tfrac{q}{p}] = -\log Q(\mathrm{supp}\,P) \ge 0, by Jensen on the strictly convex -\log; the sum collects at most all of Q’s mass, never more. Equality needs q/p constant and Q(\mathrm{supp}\,P)=1, i.e. P=Q. \blacksquare

Every bound in this chapter is a corollary.

Cross-entropy = floor + waste

The decomposition

\mathrm{CE}(P,Q) = -\mathbb{E}_P[\log q] = H(P) + D_{\mathrm{KL}}(P\|Q), so Gibbs gives \mathrm{CE} \ge H(P), and minimizing CE in Q is minimizing KL:

import numpy as onp
p = onp.array([0.6, 0.3, 0.1])
q = onp.array([0.2, 0.5, 0.3])

ce_pq = float(-onp.sum(p * onp.log(q)))
print(f'KL(P||Q) = {kl_divergence(p, q):.4f} nats, '
      f'KL(Q||P) = {kl_divergence(q, p):.4f} nats')
print(f'CE(P, Q) - H(P) = {ce_pq - float(entropy(p)):.4f} nats')

KL(P||Q) = 0.3961 nats, KL(Q||P) = 0.3653 nats
CE(P, Q) - H(P) = 0.3961 nats

The forward KL and \mathrm{CE}-H agree to the digit; KL is asymmetric.

Gaussian KL, in closed form

A check

D_{\mathrm{KL}}\!\bigl(\mathcal N(\mu_1,\sigma_1^2)\,\|\,\mathcal N(\mu_2,\sigma_2^2)\bigr) = \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2} - \frac12 matches Monte Carlo:

closed_form = gaussian_kl(0.0, 1.0, 1.0, 1.0)
monte_carlo = mc_kl(0.0, 1.0, 1.0, 1.0)

closed_form, monte_carlo

(0.5, np.float64(0.49812769302542803))

Swapping the arguments changes the number, 0.318 vs 0.807 nats:

kl_pq = gaussian_kl(0.0, 1.0, 0.0, 2.0)
kl_qp = gaussian_kl(0.0, 2.0, 0.0, 1.0)

kl_pq, kl_qp

(0.3181471805599453, 0.8068528194400546)

Cross-entropy is the classification loss

Maximum likelihood

A one-hot target collapses CE to -\log\hat y_{\text{true}}, the model’s surprise at the correct class, averaged over the batch:

import numpy as onp
labels = onp.array([0, 2])
preds = onp.array([[0.3, 0.6, 0.1], [0.2, 0.3, 0.5]])

cross_entropy(preds, labels)

0.9485599924429406

This is exactly the negative log-likelihood.

The same number the built-in loss prints

One principle

Hand-rolled cross-entropy equals the built-in loss to the digit:

preds_jax = jnp.asarray(preds)
labels_jax = jnp.asarray(labels)
logits = jnp.log(preds_jax)
optax.softmax_cross_entropy_with_integer_labels(
    logits, labels_jax).mean()

Array(0.94856, dtype=float32)

Minimizing CE = maximum likelihood = KL-projection of the empirical distribution onto the model family. One loss, three readings.

Truthful reporting is the unique optimum

Proper scoring rules

A scoring rule S(\mathbf q, y) is strictly proper if reporting the truth \mathbf p is the unique minimizer of the expected penalty. The log score S = -\log q_y qualifies: its expected penalty is exactly

\mathrm{CE}(\mathbf p,\mathbf q) = H(\mathbf p) + D_{\mathrm{KL}}(\mathbf p\|\mathbf q),

which Gibbs minimizes uniquely at \mathbf q = \mathbf p.

At population risk, strict propriety makes the true conditional distribution the unique optimum. Finite data, a restricted model class, or distribution shift can still leave a classifier miscalibrated. The Brier score \sum_j (q_j - \mathbf 1[j{=}y])^2 is also strictly proper, with the same optimum and a bounded penalty on confident errors.

The coding view

Kraft, Shannon, arithmetic coding: why KL is extra bits

Prefix codes and the Kraft inequality

Codes

A prefix-free binary code with lengths \ell_i exists iff \sum_i 2^{-\ell_i} \le 1: each codeword claims a dyadic interval, and prefix-free means the intervals are disjoint.

Shannon picks \ell_i = \lceil\log_2 \tfrac{1}{p_i}\rceil.

Shannon’s code: KL is the extra bits

The payoff

The matched code spends H_2(P) \le \mathbb{E}[\ell] < H_2(P)+1; the wrong code Q overspends by exactly D_{\mathrm{KL}}:

Shannon lengths [1, 2, 3, 3], Kraft sum = 1.0
E[l] = 1.75 bits = H(P) = 1.75 bits: no waste
uniform codebook: 2.0 bits = H + KL = 2.0 bits

H is the floor, \mathrm{CE} is the bill, and D_{\mathrm{KL}} is the wasted bits.

Arithmetic coding pays the ceiling once

Beyond symbol codes

Shannon’s \lceil\log_2(1/p)\rceil rounds up to a whole bit on every symbol: a p=0.9 symbol carries 0.15 bits yet costs 1. Arithmetic coding runs the Kraft picture in reverse: narrow one interval per message, to width w = \prod_i q(x_i \mid x_{<i}), and name it in \lceil\log_2(1/w)\rceil + 1 bits:

final width = 3.874205e-02, prod of probs = 3.874205e-02
total surprisal = 4.69 bits
arithmetic code: 6 bits, symbol-by-symbol Shannon code: 13 bits

4.69 bits of surprisal: the arithmetic code spends 6; the symbol-by-symbol code, 13.

A language model is a compressor

Prediction = compression

Nothing required i.i.d.: any next-symbol model q(\cdot\mid x_{<i}) drives the coder. With a shared tokenizer, model, and finite-precision coder, a document’s total surprisal costs fewer than two extra bits; file framing and model distribution are separate costs. An LLM plus an arithmetic coder out-compresses gzip (Delétang et al., 2023).

The floor is the source’s entropy rate H_{\mathrm{rate}} = \lim_n H(X_n \mid X_{<n}): about 1 bit/character for English (Shannon, 1951), far below the 4.75 of random letters.

The scaling laws, read in this lens: held-out CE falls as a power law in compute, so the approach to the entropy-rate floor is empirically predictable.

Perplexity: an effective branching factor

Language models

\mathrm{PPL} = \exp(\mathrm{CE}): the inverse geometric-mean probability, the number of equally-likely choices the model is as confused among:

q_tok = [0.2, 0.1, 0.4, 0.25, 0.05]   # model prob. of each observed token
nll = [-math.log(q_i) for q_i in q_tok]
mean_nll = sum(nll) / len(nll)

print(f'per-token NLL (nats): {[round(v, 3) for v in nll]}')
print(f'mean NLL = {mean_nll:.4f} nats, perplexity = {math.exp(mean_nll):.2f}')

per-token NLL (nats): [1.609, 2.303, 0.916, 1.386, 2.996]
mean NLL = 1.8421 nats, perplexity = 6.31

Base-free: a single bad token (p=0.05) costs as much as several good ones.

Three corollaries of Gibbs

label smoothing, distillation, one principle

Label smoothing softens the target

Calibration

A smoothed target (1-\epsilon)\,\mathbf e_y + \tfrac{\epsilon}{k}\mathbf 1 makes the CE optimum a finite logit gap (Gibbs), so the model can no longer chase infinite confidence:

optimal logit gap = ln(91) = 4.5109
CE at q* = H(p_eps) = 0.5003 nats
CE at the overconfident q = 0.9004 nats

The loss floors at H(\mathbf p^\epsilon) > 0, not zero.

Distillation matches a soft teacher

Transfer

The student minimizes T^2\,D_{\mathrm{KL}}(\text{teacher}_T\,\|\,\text{student}_T); the T^2 cancels the 1/T^2 gradient shrinkage at temperature T, keeping the update scale-matched. Autograd confirms the closed form:

T= 1.0  |autograd - closed| = 6.0e-08  T^2 grad = [-0.462, 0.441, 0.022]
T= 2.0  |autograd - closed| = 1.5e-08  T^2 grad = [-0.672, 0.549, 0.123]
T= 5.0  |autograd - closed| = 3.7e-09  T^2 grad = [-0.741, 0.478, 0.263]
T=10.0  |autograd - closed| = 2.8e-09  T^2 grad = [-0.719, 0.413, 0.306]

One principle, many losses

Synthesis

Maximum likelihood = cross-entropy minimization = KL-projection of the data onto the model. Label smoothing and distillation are both just Gibbs with a softened target.

Pick the target distribution; Gibbs picks the optimum and guarantees the floor.

Recap

Wrap-up

Self-information -\log p; entropy H averages it and tops out at \log k (uniform).
KL \ge 0 (Gibbs, one line of Jensen); everything else follows from it.
\mathrm{CE} = H + \mathrm{KL}; minimizing it is maximum likelihood.

Kraft + Shannon: KL is the extra bits of a wrong code.
Arithmetic coding sits on the surprisal floor: prediction is compression.
Perplexity = \exp(\mathrm{CE}), a base-free branching factor.
Label smoothing, distillation, proper scoring: corollaries of Gibbs.

Next: the wider family of divergences and distances that define modern generative objectives.