Conditional Generation

Dive into Deep Learning · §16.6

Conditional generation
the game on pairs · how the condition enters · class-conditional CIFAR-10 · measuring condition alignment · translation

The Game on Sample–Condition Pairs

Data supplies sample–condition pairs (x, c); the generator draws c from the same label marginal and attaches a sample: x' = G(z, c). The critic scores the pairs:

V(D) = E_{(x,c) \sim p}[\log \sigma(D(x, c))] + E_{c,\, x' \sim q(\cdot \mid c)}[\log \sigma(-D(x', c))]

This is the log-loss game from :numref:sec_basic_gan on pairs rather than samples. The derivation below assumes that the real and generated pairs share the marginal p(c).

The Optimal Critic, Slice by Slice

The pointwise argument applies to any classifier input space:

D^\star(x, c) = \log \frac{p(c)\, p(x \mid c)}{p(c)\, q(x \mid c)} = \log \frac{p(x \mid c)}{q(x \mid c)}

\max_D V = E_c\big[2\, \mathrm{JS}(p(\cdot \mid c), q(\cdot \mid c))\big] - 2 \log 2

  • An average of per-condition games, weighted by p(c).
  • Minimum iff q(\cdot \mid c) = p(\cdot \mid c) on every slice: stronger than matching the marginal (a label-shifted generator matches the marginal perfectly and fails every slice).
  • Every chapter result — saturation, weights, pairing, penalties — transfers per condition.

How the Condition Enters the Networks

  • Concatenation (Mirza & Osindero, 2014): embed c and concatenate it onto an input pathway. This imposes no explicit interaction form, so the network learns the interaction from the adversarial signal.
  • Modulation: h \mapsto \gamma(c) \cdot \mathrm{norm}(h) + \beta(c): conditional BatchNorm, FiLM, AdaIN; SPADE computes \gamma, \beta spatially from a layout. (R3GAN omits it for minimalism; it helps FID.)
  • Compatibility heads on the critic: the projection head (derived next) or an auxiliary classifier, whose classification reward may favor class-prototypical samples.

Text conditions: cross-attention (GigaGAN). Our experiment: concatenation into G, projection head in D.

Deriving the Projection Discriminator

Bayes’ rule separates label consistency from unconditional realness:

\log \frac{p(x \mid c)}{q(x \mid c)} = \underbrace{\log \frac{p(c \mid x)}{q(c \mid x)}}_{\textrm{label consistency}} + \underbrace{\log \frac{p(x)}{q(x)}}_{\textrm{unconditional realness}}

Model both label posteriors as softmax classifiers over shared features \varphi; the normalizers are c-free, so

D(x, c) = e_c^\top \varphi(x) + \psi(x)

  • One embedding per class, one inner product, one unconditional head.
  • Bayes’ rule gives an identity. The shared log-linear representation of both posteriors is an assumption; learning \varphi makes it more flexible but does not remove it.
  • The standard head at scale: BigGAN, R3GAN’s conditional runs.

Class-Conditional CIFAR-10, the Chapter’s Recipe

32×32, one fewer stage than :numref:sec_dcgan; class embedding concatenated at the 4×4 stage of G; projection head in D. Loss: d2l.rpgan_loss_D/G + both zero-centered penalties. Pairs share their condition: real batch’s labels condition the fakes.

A pilot sweep over \gamma \in \{0.05, 0.5, 5\} remained stable at all three values but showed progressively stronger damping. We use the intermediate value \gamma = 0.5. The appropriate value depends on the dataset and implementation: the sprite experiment uses 10, while R3GAN starts CIFAR-10 at 0.05 and decays it.

Generation on Demand

Each row fixes a requested class, and each column uses a new latent draw. Vehicle classes are usually recognizable from shape; animal classes rely more on palette and background. The variation across columns provides a visual check of within-class diversity.

Measuring Condition Alignment

class         alignment  classifier
airplane           0.73        0.81
automobile         0.73        0.84
bird               0.39        0.49
cat                0.28        0.38
deer               0.80        0.86
dog                0.47        0.58
frog               0.80        0.85
horse              0.69        0.81
ship               0.54        0.64
truck              0.78        0.84
overall            0.62        0.71

Alignment is the fraction of conditioned samples that the classifier assigns to the requested class. The classifier’s accuracy on real images provides a reference, not an upper bound. Because the classifier is evaluated under distribution shift, the comparison remains a proxy. The weakest classes differ between runs and score below their visual impression in the grid.

Alignment Sees the Permutation; the Pooled Distances Do Not

permuted labels: alignment 0.60 -> 0.06, FD 1.56, MMD^2 0.052 (both unchanged)
critic label check: mean D(x, y) - D(x, y+1) on real = 0.13
collapse baseline: alignment 0.80, per-class feature variance 0 (generator 0.188, real 0.218)
  • Shifting the requested labels of the same 1,000 images reduces alignment, while FD and MMD^2 remain unchanged because the images do not change.
  • Repeating one fixed image per class gives alignment near the real-image reference but nearly zero per-class feature variance. The variance detects within-class collapse that alignment misses.

The Trade-off Triangle

Three axes: fidelity, diversity, condition alignment.

  • FD / MMD^2 in the chapter-trained feature space measure the marginal fit; alignment measures agreement with the requested condition.
  • Alignment can be high for a generator that returns one recognizable image per class. This is within-class collapse.
  • The conditional objective penalizes this failure, but alignment does not. Per-class metrics are needed to detect it (Exercises 4–5).

Translation as Conditioning

The condition can be an image:

  • pix2pix (paired): conditional critic on output–input pairs + L1. The input pins the low frequencies; pointwise losses average over the ambiguous texture, so the patch critic carries it (:numref:sec_gan_beyond develops the argument).
  • CycleGAN (unpaired): two marginal critics — each judges only its target collection and never sees the source. The cycle-consistency loss alone carries the relation, without guaranteeing the intended correspondence.

Implementations: :numref:chap_cv.

Recap

  • Conditioning = the same game on pairs; optimal critic = conditional log ratio; value = expected per-slice JS. Analysis inherited, slice by slice.
  • Matching every conditional is strictly stronger than matching the marginal.
  • Bayes’ rule and log-linear class posteriors motivate the projection head.
  • CIFAR-10: the chapter’s loss, conditional, \gamma re-tuned; most grid rows read as their class, and which rows are weakest is a per-run finding, far below their visual impression.
  • Alignment is interpreted against the classifier’s real-image accuracy, but remains an uncalibrated proxy. Per-class feature variance reveals within-class collapse that alignment cannot detect.
  • Paired translation reuses the conditional critic; unpaired translation plays marginal games held together by a cycle loss.