27  Information Theory and Divergences

Information theory is the language of losses. This chapter builds entropy, cross-entropy, and the Kullback–Leibler divergence, shows that minimizing cross-entropy is maximum likelihood, and then broadens to the wider family of divergences and distances (f-divergences, optimal transport, integral probability metrics) that define modern generative objectives, and to mutual information and the contrastive objectives of representation learning.

Resources and Further Reading

A short, curated reading list for information theory and divergences as they appear in machine and deep learning: entropy and cross-entropy, the KL and broader \(f\)-divergences, mutual information, optimal transport, and the information bottleneck.

Books

• Elements of Information Theory — Cover & Thomas — The standard graduate reference; entropy, relative entropy, mutual information, and their chain rules, all developed from first principles.
• Information Theory, Inference, and Learning Algorithms — David MacKay — Free to read online; the rare book that treats information theory and machine learning as one subject, with a strong Bayesian and coding-theoretic flavor.
• Information Theory: From Coding to Learning — Polyanskiy & Wu — A modern graduate text deliberately oriented toward statistics and learning; the freely available MIT 6.441 lecture notes are its precursor.
• Computational Optimal Transport — Peyré & Cuturi — The standard reference on OT for data science; Wasserstein distances, entropic regularization, and Sinkhorn, with the full text on arXiv.
• The Minimum Description Length Principle — Peter Grünwald — The definitive account of learning as data compression: two-part codes, universal coding, and why shorter descriptions generalize; the core ideas are also in his freely available tutorial.

Courses and lecture notes

• MIT 6.441 Information Theory — Polyanskiy & Wu — Free OCW lecture notes (single PDF or per-chapter) covering information measures, hypothesis testing, and the bridge to statistical learning.
• Stanford EE376A Information Theory — Tsachy Weissman — An undergraduate-to-graduate course with public, student-scribed lecture notes; intuition-first, with applications throughout.

Tutorials, blogs, and surveys

• Visual Information Theory — Christopher Olah — The clearest visual introduction to entropy, cross-entropy, KL divergence, and mutual information; ideal for building intuition before the formalism.
• From GAN to WGAN — Lilian Weng — Walks through KL, Jensen–Shannon, and Wasserstein distances and exactly why a generative objective might prefer one over another.

Foundational papers

• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization — Nowozin et al. — Shows that any \(f\)-divergence yields a GAN-style objective via its variational (Fenchel) lower bound, unifying many adversarial losses.
• Wasserstein GAN — Arjovsky et al. — Replaces JS divergence with the Earth-Mover (Wasserstein-1) distance, giving smoother gradients and meaningful loss curves when supports do not overlap.
• Language Modeling Is Compression — Delétang et al. — Drives an arithmetic coder with an LLM’s next-token probabilities and obtains a state-of-the-art general-purpose compressor, making “cross-entropy is a code length” literal at scale.
• Representation Learning with Contrastive Predictive Coding — van den Oord et al. — Introduces the InfoNCE loss, framing contrastive self-supervision as maximizing a tractable lower bound on mutual information.
• The Information Bottleneck Method — Tishby, Pereira & Bialek — The original formulation: compress \(X\) while preserving information about \(Y\), generalizing rate–distortion theory.
• Deep Learning and the Information Bottleneck Principle — Tishby & Zaslavsky — Recasts deep networks as a sequence of information bottlenecks, an influential (and debated) lens on representation and generalization.