17  Diffusion Models

A generative model is asked for two things: a probability, or a density, for data like those it was trained on, such as an image of a coat, and new samples that could have come from the same source, such as coats nobody photographed. Four established families of generative models pay for these in one of two ways. Likelihood-based models keep their density normalized by construction: they factorize it into conditionals, as the language models of Section 8.3 do, average normalized components over latent variables and train through a tractable lower bound on the log-likelihood (Section 27.3.5), or compose invertible maps whose volume change is tractable (Section 29.1.5). Each of these choices constrains the architecture. Generative adversarial networks (Chapter 16) leave the generator’s architecture unconstrained and give up the density. They are trained through a two-player game whose dynamics need not converge and whose samples can collapse onto a few modes.

An energy-based model keeps the density and removes the architectural constraint, and pays a third price. It writes the density as \(p_{\boldsymbol{\theta}}(\mathbf{x}) = \exp(-E_{\boldsymbol{\theta}}(\mathbf{x})) / Z(\boldsymbol{\theta})\), where the energy \(E_{\boldsymbol{\theta}}\) is any network for which the normalizing constant \(Z(\boldsymbol{\theta})\), the integral of \(\exp(-E_{\boldsymbol{\theta}})\) over the whole input space, is finite. The network is free, but every likelihood evaluation involves this integral, which for a general network has no closed form and is expensive to approximate reliably in high dimension, and every gradient step involves expectations under the distribution it normalizes. This chapter shows how to train and sample from models of this kind, and from their descendants, with methods that never compute it.

The device is the score, the gradient of the log-density with respect to the input. Differentiating \(\log p_{\boldsymbol{\theta}}\) removes \(Z(\boldsymbol{\theta})\), which does not depend on \(\mathbf{x}\). A model of the score can be fitted with objectives computable from data alone, and approximate samples can be drawn from a score by a noisy gradient ascent called Langevin dynamics. Both steps become unreliable where the data are sparse: the score is poorly determined there, and the sampler crosses such regions slowly. The repair, estimating the score of progressively noisier copies of the data and then denoising from the noisiest level to the cleanest, is the idea behind diffusion models.

The sections follow this dependency. Section 17.1 states what adversarial training leaves unsolved and introduces energy-based models together with the normalization problem. Section 17.2 derives the maximum-likelihood gradient of an energy-based model and the Markov chain Monte Carlo methods that estimate it, which exposes the cost of drawing samples from the model during training. Section 17.3 replaces the likelihood with the Fisher divergence and derives score matching objectives that need no samples from the model. Section 17.4 obtains the score of noise-perturbed data from a denoising regression and applies it to images. Section 17.5 turns a score into a sampler and identifies where a score estimated at a single noise level fails. Section 17.6 estimates scores at many noise levels and anneals the sampler through them. Section 17.7 defines the forward noising process and the learned reverse process of a denoising diffusion probabilistic model, derives its variational objective, and shows that its terms are weighted noise-prediction regressions, the same regression as before, whose unweighted version is the loss used in practice. Section 17.8 conditions the U-Net of Section 17.6 on the class, trains it on Fashion-MNIST, and adds classifier-free guidance and, for comparison, classifier guidance.

Three sections then extend the model along the directions in which the field has moved. Sampling from the model of Section 17.8 runs the reverse chain through all of its thousand steps; Section 17.9 shows that the trained noise predictor also defines a deterministic sampler that skips most of them and, for a well-trained predictor, reaches comparable samples in a few dozen. Section 17.10 recasts the whole construction as learning a velocity field along a prescribed path from noise to data and shows, through an exact dictionary between scores and velocities on Gaussian paths, that diffusion is one choice of path among many. Section 17.11 carries the framework to categorical data, where noise means masking or resampling tokens, and derives the masked-diffusion objective used by recent diffusion language models.

Section 17.2 through Section 17.7 share one running example, a two-dimensional mixture of three Gaussians with unequal weights, and the three closing sections return to it, the last one in a quantized form. Its density, its score, and the density and score of every noise-perturbed version of it are available in closed form, so each learned quantity can be compared with the truth rather than judged by eye. Images enter in Section 17.4 through Fashion-MNIST (Section 3.2) and return in Section 17.6 and in every section from Section 17.8 on. The \(28 \times 28\) images keep each experiment within about half an hour on a laptop processor, and most within minutes. The running times quoted in the chapter were measured on a ten-core Apple M5 and vary with the hardware.

Table 17.1 lists the models of the chapter, the quantity each network represents, the objective that fits it, and the sampler that turns it into data. The rows are ordered as the chapter is. Through the fifth row each removes a limitation of the one above it; the last three extend the fifth in different directions.

Table 17.1: The models of this chapter. Each row names what the network represents, the objective that trains it, and the procedure that produces samples.
model the network represents training objective sampler where
energy-based model an energy \(E_{\boldsymbol{\theta}}(\mathbf{x})\); the density is proportional to \(\exp(-E_{\boldsymbol{\theta}})\) maximum likelihood, whose gradient lowers the energy at the data and raises it at samples from the model Markov chain Monte Carlo, also needed inside training Section 17.1, Section 17.2
score model the score \(\mathbf{s}_{\boldsymbol{\theta}}(\mathbf{x}) \approx \nabla_{\mathbf{x}} \log p(\mathbf{x})\) Fisher divergence, made computable by score matching or sliced score matching Langevin dynamics Section 17.3, Section 17.5
denoising score model the score of the data after Gaussian perturbation at one noise level denoising score matching, a regression on the added noise Langevin dynamics at that noise level, which samples the perturbed data Section 17.4, Section 17.5
noise-conditional score network the scores at a ladder of noise levels denoising score matching, weighted by the squared noise level and summed over the ladder annealed Langevin dynamics Section 17.6
denoising diffusion probabilistic model the noise \(\boldsymbol{\epsilon}\) in \(\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\, \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t}\, \boldsymbol{\epsilon}\), the accumulated noise of the first \(t\) steps a variational bound whose terms are weighted noise-prediction regressions; trained with the unweighted simple loss ancestral sampling of the learned reverse chain Section 17.7, Section 17.8
denoising diffusion implicit model the same noise predictor, without retraining the same simple loss a deterministic update on a sparse subset of the steps, with a knob that interpolates back to the stochastic chain Section 17.9
flow matching a velocity field along a path from noise to data regression onto the conditional velocity of the path; on Gaussian paths the velocity is an affine function of the score, so diffusion is one such path numerical integration of an ordinary differential equation Section 17.10
discrete (masked) diffusion the distribution of each clean token given a partially masked sequence a variational bound that reduces to a weighted cross-entropy on masked tokens iterative unmasking Section 17.11

The chapter assumes the probability and maximum-likelihood material of Section 27.3, including the Kullback–Leibler divergence and the evidence lower bound, and it recalls the Markov chain material it needs. Results from Chapter 16 are referenced but not required. Proofs already given in the mathematics appendix are cited rather than repeated: Hyvärinen’s identity, Vincent’s theorem, the stationarity of Langevin dynamics, and the correspondence between the diffusion chain and a stochastic differential equation all appear in Section 29.4, and Section 29.3 develops the continuous-time view on which that section builds. Readers who want the differential-equation perspective behind Section 17.10 will find it in these two sections.

The order of presentation of Section 17.1 through Section 17.8 follows lectures 10 through 13 of the Deep Generative Models course of Kuleshov (2023), whose slides credit figures and content adapted from Yang Song, Stefano Ermon, and Lilian Weng. Each section cites the course where it follows that exposition and cites the original papers for its results; the three closing sections go beyond those lectures and cite the original papers directly. The monograph of Lai et al. (2025) develops the same material from the variational, score-based, and flow-based viewpoints at greater length.

Resources and Further Reading

The chapter derives each objective from the one before it and verifies the derivations on a distribution with a closed-form score. The following resources provide the original expositions, longer treatments, and reference implementations.