Entropy, Cross-Entropy, and KL Divergence

Dive into Deep Learning · §28.1

The reported loss is a code length
entropy, cross-entropy, and KL divergence.

Three numbers in every loss

Motivation

The scalar reported by a training loop is a code length in nats (Shannon, 1948). It splits into three pieces:

entropy H(P): the irreducible expected code length,
cross-entropy \mathrm{CE}(P,Q): the expected length under the model,
KL D_{\mathrm{KL}}(P\|Q): their difference.

Surprise and entropy

self-information, Shannon’s H, the uniform maximum

Rare events are surprising

Self-information

I(x) = -\log p(x) is the only measure that is zero for a certain event, decreasing in p, and additive over independent events; additivity is exactly what forces a logarithm.

The fair coin carries \ln 2 \approx 0.693 nats; a one-in-a-million event, \approx 13.8.

Entropy averages the surprise

Entropy

H(P) = \mathbb{E}_P[-\log p] = -\sum_x p(x)\log p(x), with the convention 0\log 0 = 0; it is the expected length of the best possible code for P:

import numpy as onp
def entropy(p):
    """Entropy of a probability vector, in nats."""
    p = onp.asarray(p, dtype=float)
    if p.ndim != 1 or not onp.all(onp.isfinite(p)):
        raise ValueError("p must be a finite one-dimensional probability vector")
    if onp.any(p < 0):
        raise ValueError("probabilities must be nonnegative")
    if not onp.isclose(p.sum(), 1.0):
        raise ValueError("probabilities must sum to one")
    positive = p > 0
    return float(-onp.dot(p[positive], onp.log(p[positive])))

entropy(onp.array([0.1, 0.5, 0.1, 0.3]))

1.1682824501765625

Less than \ln 4 \approx 1.386 nats: this distribution is not uniform.

Uncertainty peaks at the uniform law

The maximum

On k outcomes, 0 \le H(P) \le \log k, with the maximum exactly at the uniform distribution.

Proof. H = \mathbb{E}[\log\tfrac{1}{p}] \le \log\mathbb{E}[\tfrac1p] = \log m \le \log k, where m counts the outcomes with p_i>0, by Jensen on the strictly concave \log. Equality forces 1/p constant and m=k: the uniform law, exactly. \blacksquare

Cross-entropy and KL

the trainable gap: Gibbs’ inequality

KL divergence as excess code length

Relative entropy

D_{\mathrm{KL}}(P\|Q) = \mathbb{E}_{x\sim P}\bigl[\log p(x)/q(x)\bigr]: the extra nats from coding P’s data with Q’s code. Asymmetric, and +\infty where q(x)=0 but p(x)>0:

import numpy as onp
def kl_divergence(p, q):
    """KL(P || Q) for two probability vectors, in nats."""
    p, q = onp.asarray(p, dtype=float), onp.asarray(q, dtype=float)
    kl = p * onp.log(p / q)
    return float(onp.nansum(kl))

Gibbs’ inequality

The one fact

D_{\mathrm{KL}}(P\|Q) \ge 0, with equality iff P=Q.

Proof. D_{\mathrm{KL}} = \mathbb{E}_P[-\log\tfrac{q}{p}] \ge -\log\mathbb{E}_P[\tfrac{q}{p}] = -\log Q(\mathrm{supp}\,P) \ge 0, by Jensen on the strictly convex -\log; the sum collects at most all of Q’s mass, never more. Equality needs q/p constant and Q(\mathrm{supp}\,P)=1, i.e. P=Q. \blacksquare

This inequality supports the results developed below.

Cross-entropy = entropy + KL

The decomposition

\mathrm{CE}(P,Q) = -\mathbb{E}_P[\log q] = H(P) + D_{\mathrm{KL}}(P\|Q), so Gibbs gives \mathrm{CE} \ge H(P), and minimizing CE in Q is minimizing KL:

import numpy as onp
p = onp.array([0.6, 0.3, 0.1])
q = onp.array([0.2, 0.5, 0.3])

ce_pq = float(-onp.sum(p * onp.log(q)))
print(f'KL(P||Q) = {kl_divergence(p, q):.4f} nats, '
      f'KL(Q||P) = {kl_divergence(q, p):.4f} nats')
print(f'CE(P, Q) - H(P) = {ce_pq - float(entropy(p)):.4f} nats')

KL(P||Q) = 0.3961 nats, KL(Q||P) = 0.3653 nats
CE(P, Q) - H(P) = 0.3961 nats

The forward KL equals \mathrm{CE}-H numerically; KL is asymmetric.

Gaussian KL, in closed form

A check

D_{\mathrm{KL}}\!\bigl(\mathcal N(\mu_1,\sigma_1^2)\,\|\,\mathcal N(\mu_2,\sigma_2^2)\bigr) = \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2} - \frac12 matches Monte Carlo:

closed_form = gaussian_kl(0.0, 1.0, 1.0, 1.0)
monte_carlo = mc_kl(0.0, 1.0, 1.0, 1.0)

closed_form, monte_carlo

(0.5, np.float64(0.4996905134959491))

Swapping the arguments changes the number, 0.318 vs 0.807 nats:

kl_pq = gaussian_kl(0.0, 1.0, 0.0, 2.0)
kl_qp = gaussian_kl(0.0, 2.0, 0.0, 1.0)

kl_pq, kl_qp

(0.3181471805599453, 0.8068528194400546)

Cross-entropy is the classification loss

Maximum likelihood

A one-hot target collapses CE to -\log\hat y_{\text{true}}, the model’s surprise at the correct class, averaged over the batch:

import numpy as onp
labels = onp.array([0, 2])
preds = onp.array([[0.3, 0.6, 0.1], [0.2, 0.3, 0.5]])

cross_entropy(preds, labels)

0.9485599924429406

This is exactly the negative log-likelihood.

The same number the built-in loss prints

Core identity

The direct cross-entropy calculation equals the built-in loss:

preds_jax = jnp.asarray(preds)
labels_jax = jnp.asarray(labels)
logits = jnp.log(preds_jax)
optax.softmax_cross_entropy_with_integer_labels(
    logits, labels_jax).mean()

Array(0.94856, dtype=float32)

Minimizing CE = maximum likelihood = KL-projection of the empirical distribution onto the model family. These are three interpretations of the same objective.

Truthful reporting is the unique optimum

Proper scoring rules

A scoring rule S(\mathbf q, y) is strictly proper if reporting the truth \mathbf p is the unique minimizer of the expected penalty. The log score S = -\log q_y qualifies: its expected penalty is exactly

\mathrm{CE}(\mathbf p,\mathbf q) = H(\mathbf p) + D_{\mathrm{KL}}(\mathbf p\|\mathbf q),

which Gibbs minimizes uniquely at \mathbf q = \mathbf p.

At population risk, strict propriety makes the true conditional distribution the unique optimum. Finite data, a restricted model class, or distribution shift can still leave a classifier miscalibrated. The Brier score \sum_j (q_j - \mathbf 1[j{=}y])^2 is also strictly proper, with the same optimum and a bounded penalty on confident errors.

The coding view

Kraft, Shannon, arithmetic coding: why KL is extra bits

Prefix codes and the Kraft inequality

Codes

A prefix-free binary code with lengths \ell_i exists iff \sum_i 2^{-\ell_i} \le 1: each codeword claims a dyadic interval, and prefix-free means the intervals are disjoint.

Shannon picks \ell_i = \lceil\log_2 \tfrac{1}{p_i}\rceil.

Shannon’s code: KL is the extra bits

Coding interpretation

The matched code spends H_2(P) \le \mathbb{E}[\ell] < H_2(P)+1; a code based on Q has excess expected length D_{\mathrm{KL}}:

Shannon lengths [1, 2, 3, 3], Kraft sum = 1.0
E[l] = 1.75 bits = H(P) = 1.75 bits: no waste
uniform codebook: 2.0 bits = H + KL = 2.0 bits

H is the minimum expected code length, cross-entropy is the length obtained under Q, and D_{\mathrm{KL}} is their difference.

Arithmetic Coding and Rounding Overhead

Beyond symbol codes

Shannon’s \lceil\log_2(1/p)\rceil rounds up to a whole bit on every symbol: a p=0.9 symbol carries 0.15 bits yet costs 1. Arithmetic coding runs the Kraft picture in reverse: narrow one interval per message, to width w = \prod_i q(x_i \mid x_{<i}), and name it in \lceil\log_2(1/w)\rceil + 1 bits:

final width = 3.874205e-02, prod of probs = 3.874205e-02
total surprisal = 4.69 bits
arithmetic code: 6 bits, symbol-by-symbol Shannon code: 13 bits

4.69 bits of surprisal: the arithmetic code spends 6; the symbol-by-symbol code, 13.

A language model is a compressor

Prediction = compression

Nothing required i.i.d.: any next-symbol model q(\cdot\mid x_{<i}) drives the coder. With a shared tokenizer, model, and finite-precision coder, a document’s total surprisal costs fewer than two extra bits; file framing and model distribution are separate costs. Delétang et al. (2023) demonstrate this connection using language models as compressors.

The floor is the source’s entropy rate H_{\mathrm{rate}} = \lim_n H(X_n \mid X_{<n}): about 1 bit/character for English (Shannon, 1951), far below the 4.75 of random letters.

The scaling laws, read in this lens: held-out CE falls as a power law in compute, so the approach to the entropy-rate floor is empirically predictable.

Perplexity: an effective branching factor

Language models

\mathrm{PPL} = \exp(\mathrm{CE}): the inverse geometric-mean probability, the number of equally-likely choices the model is as confused among:

q_tok = [0.2, 0.1, 0.4, 0.25, 0.05]   # model prob. of each observed token
nll = [-math.log(q_i) for q_i in q_tok]
mean_nll = sum(nll) / len(nll)

print(f'per-token NLL (nats): {[round(v, 3) for v in nll]}')
print(f'mean NLL = {mean_nll:.4f} nats, perplexity = {math.exp(mean_nll):.2f}')

per-token NLL (nats): [1.609, 2.303, 0.916, 1.386, 2.996]
mean NLL = 1.8421 nats, perplexity = 6.31

Base-free: a single bad token (p=0.05) costs as much as several good ones.

Three corollaries of Gibbs

label smoothing, distillation, and related objectives

Label smoothing softens the target

Calibration

A smoothed target (1-\epsilon)\,\mathbf e_y + \tfrac{\epsilon}{k}\mathbf 1 makes the CE optimum a finite logit gap (Gibbs), so the model can no longer chase infinite confidence:

optimal logit gap = ln(91) = 4.5109
CE at q* = H(p_eps) = 0.5003 nats
CE at the overconfident q = 0.9004 nats

The loss floors at H(\mathbf p^\epsilon) > 0, not zero.

Distillation matches a soft teacher

Transfer

The student minimizes T^2\,D_{\mathrm{KL}}(\text{teacher}_T\,\|\,\text{student}_T); the T^2 cancels the 1/T^2 gradient shrinkage at temperature T, keeping the update scale-matched. Autograd confirms the closed form:

T= 1.0  |autograd - closed| = 6.0e-08  T^2 grad = [-0.462, 0.441, 0.022]
T= 2.0  |autograd - closed| = 1.9e-09  T^2 grad = [-0.672, 0.549, 0.123]
T= 5.0  |autograd - closed| = 0.0e+00  T^2 grad = [-0.741, 0.478, 0.263]
T=10.0  |autograd - closed| = 3.5e-09  T^2 grad = [-0.719, 0.413, 0.306]

Relations among training objectives

Synthesis

Maximum likelihood = cross-entropy minimization = KL-projection of the data onto the model. Label smoothing and distillation follow from the same Gibbs inequality after replacing a hard target with a softened distribution.

Once the target distribution is specified, Gibbs’ inequality identifies both the optimum and the irreducible loss.

Information Measures Link Prediction and Compression

Wrap-up

Self-information -\log p; entropy H averages it and tops out at \log k (uniform).
KL \ge 0 follows from Jensen’s inequality and supports the subsequent results.
\mathrm{CE} = H + \mathrm{KL}; minimizing it is maximum likelihood.

Kraft + Shannon: KL is the excess expected length of a mismatched code.
Arithmetic coding approaches the surprisal bound: prediction is compression.
Perplexity = \exp(\mathrm{CE}), a base-free branching factor.
Label smoothing, distillation, proper scoring: corollaries of Gibbs.

Next: the wider family of divergences and distances that define modern generative objectives.