Adversarial Losses Beyond GANs
Dive into Deep Learning · §16.7
Adversarial losses beyond GANs
the capacity argument · distillation · tokenizers and vocoders · three responses to instability · likelihood models as discriminators
A Pointwise Loss Predicts the Conditional Mean
For any predictor f, with \mu(x) = E[y \mid x]:
E\big[(y - f(x))^2\big] = E\big[(y - \mu(x))^2\big] + E\big[(\mu(x) - f(x))^2\big]
- Squared error grades every f by its distance to the conditional mean.
- If p(y \mid x) has several valid answers, \mu averages them — and the average of sharp answers is blurry.
- A critic instead asks whether the prediction resembles samples from the data distribution. Its optimum is the log ratio \log(p/q).
Capacity Limits Make the Two Losses Differ
Two bands y = x \pm 1; one deterministic student per loss:
The MSE student runs through the empty corridor at the conditional mean; the adversarial student commits to a band, crossing only where it switches.
An Ablation Separates Regression and Adversarial Terms
ADD trains a one-step distillation student with two losses; ablating them (everything else fixed):
| loss configuration | FID |
|---|---|
| distillation only | 315.6 |
| adversarial only | 20.8 |
| both | 20.6 |
At one step, regression toward the teacher averages; the adversarial term carries essentially all of the sample fidelity.
Distillation Uses the Adversarial Term for Different Purposes
- ADD: stay on the image manifold at one step (frozen DINOv2 critic).
- DMD2: train on real data — escape the teacher’s ceiling (ImageNet-64 FID 1.28, below its teacher).
- LADD: the teacher is the discriminator; the distillation loss is dropped entirely.
- Production: FLUX.1-schnell, 12B parameters, trained by latent adversarial diffusion distillation, 1–4 steps.
One-Step Generation Without a Discriminator
- MeanFlow: one evaluation, ImageNet-256 FID 3.43, from scratch — no teacher, no discriminator.
- sCM: two evaluations, ImageNet-512 FID 1.88 — no adversarial term.
Both replace the critic with a better-constructed regression target. The adversarial term is one working answer to few-step capacity — useful, not necessary.
Tokenizers, Audio, Video
- Tokenizers: VQGAN’s patch critic improves local realism; ViTok measured a ~3× reconstruction-FID gain — then ViTok-v2 removed the loss at 5B parameters: it “prevents stable scaling”.
- Audio: adversarial training is common in deployed low-latency vocoders (HiFi-GAN through BigVGAN); RAF ports the pairing loss of §16.3 to waveforms; flow-matching vocoders are active non-adversarial alternatives.
- Video: adversarial post-training gives real-time 720p generation in a single forward evaluation.
Three Responses to Adversarial Instability
Fixed discrepancies have repeatedly been supplemented by learned components: a learned kernel, selected projections, or a critic used during distillation.
A Likelihood Model Is an Implicit Discriminator
Any two evaluable densities define a log-ratio critic:
D(x) = \log\frac{q_\theta(x)}{q_{\textrm{ref}}(x)}
- DDO fine-tunes q_\theta by playing the log-loss game with this implicit critic — no discriminator network, no two-player loop.
- EDM FID: CIFAR-10 1.79 → 1.30; ImageNet-64 1.58 → 0.97.
- The density-ratio objective remains, without simultaneous optimization of a second network.
Recap
- Squared error → conditional mean; under capacity limits, the mean is off-manifold. A critic instead scores samples through an estimated density ratio.
- Adversarial terms remain useful when capacity is limited and ambiguity is concentrated in local detail: distillation, tokenizers, vocoders, and video.
- It is not necessary: MeanFlow and sCM reach few-step quality with regression targets alone; ViTok-v2 dropped it at scale.
- Three responses address instability: regularize the dynamics, constrain the critic, or remove the game. Each introduces a different limitation.
- DDO: likelihood models are implicit discriminators. :numref:
chap_diffusionbuilds the models whose distillation brings this loss back.