16  Generative Adversarial Networks

A generative model can be used as a sampler: draw a latent variable and pass it through a network to obtain an image, a waveform, or a row of a table. The models in this chapter provide this sampling operation but no tractable density for their outputs. They can use flexible architectures and generate a sample in one forward pass, but they cannot be trained by maximum likelihood. As Section 27.3.2.1 showed, maximum likelihood requires a model density with which to evaluate a Kullback–Leibler divergence.

Adversarial training replaces likelihood evaluation with a learned comparison. A second network distinguishes generated samples from real ones, and the generator is trained to make that distinction difficult. Three design choices determine the resulting objective: the comparison loss, the class of functions available to the critic, and whether the critic scores individual samples or pairs. These choices determine which divergence the game evaluates at its optimum and whether the generator receives a useful gradient when its samples are far from the data.

Diffusion and flow models, developed in Chapter 17, have largely replaced stand-alone GANs at the frontier of image synthesis. Adversarial objectives nevertheless remain important in one-step image generators, the tokenizers used by latent generative models, and neural vocoders. This chapter therefore treats both the classical theory and the regularization techniques that make modern adversarial training substantially more reliable.

The sections follow the logical dependencies of the analysis. Section 16.1 analyzes one game exactly: the original logistic-loss objective, its optimal discriminator, and the divergence it evaluates. Section 16.2 varies the design choices and maps the resulting space of objectives, including the ones whose gradients survive when supports separate. Section 16.3 changes the critic’s input from one sample to a real–generated pair and computes the value of the resulting relativistic objective in closed form. Section 16.4 then explains why gradient descent can fail even when the objective has the correct optimum, and which regularizers restore convergence. Section 16.5 applies these results to image generation and evaluates the trained models. Section 16.6 supplies a class, caption, or image to both networks and derives the projection discriminator from the conditional density ratio. Section 16.7 examines the roles that adversarial losses retain in current systems and reconnects the central density-ratio identity to likelihood-based models.

Table Table 16.1 summarizes the chapter’s analytical results. It lists each adversarial game, the critic’s input, and the value attained by the optimal critic. This last quantity is what the generator minimizes. Here \(p\) denotes the data distribution, \(q\) the generator distribution, and \(\mathrm{JS}\) the Jensen–Shannon divergence; Section 16.1 defines all three.

Table 16.1: The games of this chapter. Each row gives an objective, what its critic scores, and the value of the game at the optimal critic.
objective the critic scores value at the optimal critic where
log loss, the original GAN one sample’s realness logit \(2\,\mathrm{JS}(p, q) - 2 \log 2\), attained at the logit \(\log(p/q)\) Section 16.1
non-saturating generator loss the same critic; the generator maximizes its samples’ scores the same minimizer \(q = p\) under an optimal critic; confidently rejected samples receive large weights when the critic retains a nonzero score gradient Section 16.1
proper classification losses and \(f\)-GAN one sample, under other classification losses an \(f\)-divergence: an average of a convex function of the ratio \(p/q\) Section 16.2
integral probability metrics: MMD, Wasserstein one sample, scored by a critic from a constrained class the largest expectation gap the class can certify; geometry-sensitive classes remain informative as supports separate, while a bounded kernel loses sensitivity beyond its length scale Section 16.2
relativistic pairing, RpGAN a real–generated pair, rewarded for ranking the real member higher \(\mathrm{JS}(p \otimes q,\, q \otimes p)\): the information a pair carries about which member is real Section 16.3
zero-centered gradient penalties \(R_1\), \(R_2\) any of the above, with a penalty on the critic’s input gradients near equilibrium, a squared linearized Wasserstein-2 distance and dynamics that converge for every positive penalty weight Section 16.4
the conditional game a sample together with the condition it claims to satisfy the average over conditions, \(E_c\big[2\,\mathrm{JS}(p(\cdot \mid c), q(\cdot \mid c))\big] - 2\log 2\); shared log-linear class posteriors motivate the projection discriminator Section 16.6

Section 16.5 and Section 16.7 add no further games. The first assembles the penalized relativistic loss into an image generator and evaluates it. The second identifies the same objectives in current one-step generators, tokenizers, and vocoders.

Readers who want the working recipe first can read Section 16.1, then the recipe and experiment of Section 16.4 and Section 16.5, returning to Section 16.2 and Section 16.3 for the theory that explains why the recipe looks the way it does. Otherwise the sections build in order, each on the results of the last.

Resources and Further Reading

The chapter derives its objectives in closed form. The following resources provide historical context, interactive demonstrations, and full-scale implementations.