Adversarial Image Generation

Dive into Deep Learning · §16.5

Adversarial image generation
the 2015 recipe · a modern minimal backbone · the loss A/B · measuring sample quality · limited data

The 2015 Recipe Stabilizes the Classic Objective

DCGAN (Radford et al., 2015) made the classic loss train on images through architectural commitments:

  • transposed-conv upsampling in G, strided conv in D;
  • batch normalization in both networks;
  • ReLU + tanh output in G, leaky ReLU in D; Adam with \beta_1 = 0.5.

Together, these choices control activation and optimizer statistics as the two networks change. The objective remains the non-saturating log loss of :numref:sec_basic_gan.

The Historical Baseline on Pokemon Sprites

The samples have plausible palettes and rough sprite-like silhouettes. The architecture made the classic loss practical on images, but normalization does not remove its mode-dropping minima or divergent dynamics.

A Modern Minimal Backbone

R3GAN’s principles at sprite scale:

  • bilinear resampling + 3 \times 3 conv (no strided/transposed conv);
  • leaky ReLU everywhere; no tanh; no normalization anywhere;
  • Adam \beta_1 = 0, lr 2 \cdot 10^{-4}; flip augmentation; weight EMA.

Latent injection: project z linearly to 4 \times 4, concatenate with the learned constant, fuse with a mix conv — a deliberate simplification of R3GAN’s basis layer (z-modulated learned 4 \times 4 feature maps).

One Backbone, Two Losses

Identical backbone, initialization, optimizer, augmentation, EMA, and budget (15,000 steps). Only the loss differs:

  • classic: d2l.update_D / d2l.update_G — the non-saturating log loss of :numref:sec_basic_gan;
  • RpGAN + R_1{+}R_2: d2l.rpgan_loss_D/G + d2l.r1_r2_penalty, \gamma = 10 — the loss of :numref:sec_gan_convergence.

\gamma picked by sweeping powers of ten: 1–100 all stable here, 0.1 collapses. R3GAN tunes \gamma from 0.05 to 150 per dataset — no single value is portable.

The arms differ in the complete loss recipe: pairing and penalties change together. The toy experiment in :numref:sec_gan_convergence and the cited StackedMNIST ablation isolate the individual components.

The Penalized Relativistic Loss Avoids Collapse

  • Classic arm: complete collapse — all 64 latent codes map to the same image. Initialization-dependent: framework-default inits survive this budget; the 2015 recipe’s \mathcal{N}(0, 0.02^2) reaches the mode-dropping minima.
  • Penalized relativistic arm: diverse, creature-shaped sprites, no repeats; critic train–holdout gap \approx 0 — this critic does not separate the sets. Generator memorization is a separate test: the nearest-neighbor check of :numref:subsec_gan_limited_data.

Critic Scores Diverge under the Classic Recipe

  • Classic: D’s loss remains near 0, while its real-image scores grow without bound and oscillate widely.
  • Penalized: both losses remain near equilibrium. The critic’s mean score stays within a narrow band whose location is arbitrary because the pairing objective is shift invariant. The A/B changes the objective and penalties together, so it does not isolate the source of the bounded behavior.

FID Is the Gaussian W2 Closed Form

Fit Gaussians to real and generated features, report the W_2^2 closed form :numref:sec_gan_objectives deferred:

\mathrm{FID} = \|\mu_p - \mu_q\|^2 + \operatorname{tr}\big(\Sigma_p + \Sigma_q - 2(\Sigma_p^{1/2} \Sigma_q \Sigma_p^{1/2})^{1/2}\big)

  • Commuting case: mean shift plus per-axis standard-deviation shifts.
  • Sees only two moments; biased at finite n — the floor row shows it.

KID Is the MMD U-Statistic with Learned Features

run                     FD (CIFAR-CNN)  MMD^2 (CIFAR-CNN)  out-of-range
real vs. real                     0.53               0.03            --
classic                         104.05              26.99         0.000
RpGAN + R1 + R2                   2.50               0.67         0.157

Unbiased MMD^2 estimator, polynomial kernel on learned features — the kernel choice :numref:sec_gan_objectives fixed, reopened. Our cells print FD / MMD^2 (CIFAR-CNN): the same formulas on chapter-trained features, not comparable to published FID/KID. Real-vs-real floor \approx 0 — unbiased, so in principle it can even print negative; both scores rank the penalized run far above the collapsed one, with the floor two orders of magnitude below the gap.

What the Numbers Do Not Settle

  • Feature dependence: scores depend on \phi. Using a CIFAR-trained CNN to score sprites has the same type of distribution mismatch as using an ImageNet-trained Inception network on other image domains.
  • Leakage: FID drops by matching ImageNet class histograms (Kynkäänniemi et al., 2023); R3GAN avoids pretrained discriminators for this reason.
  • Resizing: aliased image resizing shifts FID by method-sized margins (Parmar et al., 2022).
  • Bias: model-dependent at finite n (Chong & Forsyth, 2020).
  • Precision/recall separate fidelity from coverage: a collapsed generator can keep precision while recall \to 0.

On Limited Data the Critic Memorizes First

  • Small real set + capable critic: scores drift from density ratio to set membership; the train–holdout score gap is the overfitting statistic.
  • Fix: augment both real and generated inputs of D, differentiably (Zhao et al., 2020) — augmenting reals alone teaches G the augmented distribution. ADA feedback-controls the strength from an overfitting statistic (Karras et al., 2020).

median distance to nearest training image: generated 1.99, held-out real 0.68

The generator-side check is calibrated with held-out real images. Generated samples lie farther from the training set than held-out sprites do, and even the closest pairs depict different creatures. This test finds no evidence of direct copying.

Scale Changes the Constants, Not the Recipe

R3GAN’s real budgets (quoted):

benchmark compute
StackedMNIST 7 h on 8 L40
CIFAR-10 4 days on 8 L40
FFHQ-256 ~3 weeks on 8 A6000
ImageNet (cond.) ~1 day on 32 H100

At scale: \gamma per dataset (0.05–150), BF16 not FP16, EMA half-life in Mimg, tuned augmentation. Steering the sampler with a class or caption: :numref:sec_gan_conditional.

Recap

  • 2015: architecture stabilizes the classic objective. The modern recipe combines a regularized objective with a simpler backbone.
  • Progressive growing, spectral normalization, attention, and style blocks address distinct architectural and optimization problems. R3GAN’s control shows which stabilizers become unnecessary after changing the objective.
  • Same backbone, same budget: classic loss collapses completely from the 2015-recipe init (framework defaults survive the budget); RpGAN + R_1{+}R_2 at \gamma = 10 trains stably to diverse sprites.
  • FID = Bures–Wasserstein W_2^2; KID = unbiased MMD — the chapter’s two closed forms, in learned feature space.
  • Both metrics strongly prefer the penalized arm; the ordering is more transferable than the numerical values.
  • Every metric inherits its meaning from its feature network.
  • Next: steering the sampler with conditions (:numref:sec_gan_conditional), then where the adversarial loss survives beyond stand-alone GANs (:numref:sec_gan_beyond).