Relativistic Objectives

Dive into Deep Learning · §16.3

Relativistic objectives
the critic ranks pairs · the value is JS of the two orderings · rank weight replaces threshold weight · paper versus code

The Critic Scores a Real–Fake Pair

\Phi(D) = E_{x \sim p,\; x' \sim q}\big[\log \sigma\big(D(x) - D(x')\big)\big]

  • \sigma(D(x) - D(x')): the Bradley–Terry probability that x is preferred to x' — the reward-model likelihood of :numref:sec_regularized.
  • The same scalar realness score as before, entering only through differences.
  • The critic solves a ranking problem, not a classification problem.

Two Symmetries With Consequences

  • Additive shift: \Phi(D + b) = \Phi(D) — the critic is identified only up to a constant, where the log-loss game pinned it exactly.
  • \Phi depends on (p, q) through p \otimes q and is quadratic in the pair. The affine results of :numref:sec_gan_objectives therefore do not apply directly.

The Optimal Critic Is Still the Log Ratio

Because \Phi is concave in D, any stationary point is a global maximum. Its functional derivative is

\frac{\delta \Phi}{\delta D(t)} = p(t)\, E_{x' \sim q}[\sigma(D(x') - D(t))] - q(t)\, E_{x \sim p}[\sigma(D(t) - D(x))]

D^\star = \lambda = \log \frac{p}{q} \quad \text{up to an additive constant}

Ranking estimates the same density ratio as classification. What changes is the value of the game.

Lifting: Play the Log-Loss Game on Pairs

P = p \otimes q, Q = q \otimes p: the two orderings of an independent real–fake pair.

  • The product ratio separates: \log \frac{dP}{dQ}(a_1, a_2) = \lambda(a_1) - \lambda(a_2).
  • The swap symmetry doubles the objective: V_{P,Q}\big(D(a_1) - D(a_2)\big) = 2\,\Phi(D).
  • Differences lose nothing: restricting the pair critic to differences does not lower the supremum.

The Value Is JS Between the Two Orderings

d_{\mathrm{Rp}}(p, q) := \sup_D \Phi + \log 2 = \mathrm{JS}(p \otimes q,\, q \otimes p) = H\big[\tfrac12(p \otimes q + q \otimes p)\big] - H[p] - H[q]

  • Information interpretation: the divergence equals the mutual information between a randomly ordered pair and the position of its real member.
  • The divergence property is :citet:Jolicoeur-Martineau.2020 (general concave payoffs, value not computed); the closed form above is computed here, for the logistic payoff.

Effects and Limits of Pairing

property statement
sandwich \mathrm{JS}(p, q) \leq d_{\mathrm{Rp}} \leq \log 2
near q = p d_{\mathrm{Rp}} = 2\,\mathrm{JS} + O(\epsilon^3): a pair carries twice the information
disjoint supports d_{\mathrm{Rp}} = \log 2: saturates exactly as JS does

Pairing still saturates on disjoint supports, and unregularized RpGAN training need not converge. Section :numref:sec_gan_convergence analyzes the training dynamics.

Rank Weight Replaces Threshold Weight

w_{\mathrm{Rp}}(x') = E_{x \sim p}[\sigma(D(x') - D(x))] \qquad w_{\mathrm{GAN}}(x') = \sigma(D(x'))

  • A threshold weight can remain high when generated mass concentrates beyond one decision boundary without matching the data distribution across modes.
  • Rank: anchored to the real population’s scores, invariant to the critic’s constant.
  • :citet:Sun.Fang.Schwing.2020: n^n - n! mode-dropping local minima for the classical loss; none for the relativistic one.

The Closed Form, Checked Exactly

sup Phi + log 2, by gradient ascent:    0.376889913
JS(p (x) q, q (x) p), computed directly: 0.376889913
entropy form H[(P+Q)/2] - H[p] - H[q]:  0.376889913
JS(p, q) = 0.234050,  log 2 = 0.693147
max |D - lambda|, both centered:        4.4e-16

Gradient ascent using the stationarity condition agrees with the direct product-space JS computation to nine decimal places. The recovered critic equals \lambda up to a constant.

The Two Weights on the Same Atoms

The threshold weight changes around the arbitrary zero level. The rank weight changes across the distribution of real scores and continues to distinguish samples after the threshold weight is nearly saturated.

Saturating Paper, Non-Saturating Code

generator plays weight badly ranked sample
minimize \Phi (paper’s zero-sum Eq. 2) E_{x \sim p}[\sigma(D(x') - D(x))] \to 0: stalls
maximize E[\log \sigma(D(x') - D(x))] (the code) E_{x \sim p}[\sigma(D(x) - D(x'))] \to 1: largest

Both variants have the same fixed point, but the paper’s displayed zero-sum objective gives smaller updates to badly ranked samples early in training.

Recap

  • Pairing objective \Phi: Bradley–Terry ranking of real above fake, shift invariant, a function of p \otimes q only.
  • Optimal critic: the same \lambda, now up to a constant.
  • Value: d_{\mathrm{Rp}} = \mathrm{JS}(p \otimes q, q \otimes p) — the log-loss game lifted to the two orderings of a pair.
  • \mathrm{JS} \leq d_{\mathrm{Rp}} \leq \log 2, locally 2\,\mathrm{JS}, still saturating on disjoint supports.
  • The generator weight is a rank statistic rather than a threshold statistic; the finite-sample mode-dropping basins disappear.
  • The five-atom example verifies the value to nine decimal places. Convergence requires the analysis of :numref:sec_gan_convergence.