PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Generator Conditioning Causally Related to GAN Performance?

Augustus Odena, Jacob Buckman, Catherine Olsson, Tom B. Brown, Christopher Olah, Colin Raffel, Ian Goodfellow

arXiv:1802.08768v2stat.MLcs.LG

TL;DR

GAN generator Jacobians can become ill-conditioned during training, and their average conditioning predicts Inception Score and Frechet Inception Distance. The paper tests Jacobian Clamping as an intervention and reports improved mean scores and reduced inter-run variance, while noting that the technique does not directly penalize condition number.

  • Problem

    The paper asks whether generator Jacobian conditioning relates to GAN performance, given that conditioning can become ill-conditioned during training and GAN quality is assessed with Inception Score and FID.

    Source: §1 · p. 1

  • Method

    The authors propose Jacobian Clamping, a regularization technique that controls generator Jacobian conditioning during GAN training.

    Source: §1 · p. 1, §4 · p. 5

  • Results

    Jacobian Clamping improves mean Inception Score and FID and reduces inter-run variance, while better-conditioned generators tend to perform better on both metrics.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    The results provide evidence of a causal relationship between generator conditioning and GAN quality within the studied GAN training setting.

    Source: §6 · p. 8

    Jacobian Clamping does not directly penalize condition number, because efficiently estimating the smallest eigenvalue is not straightforward.

    Source: §4 · p. 5

Abstract

from arXiv · show

Recent work (Pennington et al, 2017) suggests that controlling the entire distribution of Jacobian singular values is an important design consideration in deep learning. Motivated by this, we study the distribution of singular values of the Jacobian of the generator in Generative Adversarial Networks (GANs). We find that this Jacobian generally becomes ill-conditioned at the beginning of training. Moreover, we find that the average (with z from p(z)) conditioning of the generator is highly predictive of two other ad-hoc metrics for measuring the 'quality' of trained GANs: the Inception Score and the Frechet Inception Distance (FID). We test the hypothesis that this relationship is causal by proposing a 'regularization' technique (called Jacobian Clamping) that softly penalizes the condition number of the generator Jacobian. Jacobian Clamping improves the mean Inception Score and the mean FID for GANs trained on several datasets. It also greatly reduces inter-run variance of the aforementioned scores, addressing (at least partially) one of the main criticisms of GANs.

1. Introduction

The paper studies generator Jacobian conditioning as a way to characterize GAN training and its pathologies. It reports that conditioning tracks established GAN quality metrics and that Jacobian Clamping improves their mean values while reducing run-to-run variance.

1. Introduction

  • GANs have been studied for training pathologies such as mode collapse, where generators produce samples from only a small subset of data modes.The paper takes a geometric rather than probabilistic perspective on these pathologies.Source: §1 · p. 1
  • The generator Jacobian generally becomes ill-conditioned quickly, then falls into a good cluster with stable or declining condition numbers or a bad cluster with continuing growth.The paper studies squared singular values at individual latent-space points.Source: §1 · p. 1
  • Better-conditioned GAN generators tend to score better on the Inception Score and Frechet Inception Distance.The paper identifies a strong correspondence between Jacobian conditioning and both GAN-quality metrics.Source: §1 · p. 1
  • Jacobian Clamping constrains generator Jacobian conditioning and improves mean Inception Score and FID while reducing inter-run variance.The authors present this intervention as evidence that the correspondence is causal.Source: §1 · p. 1

2. Background and Notation

This section introduces GAN components and evaluation measures, then sets up generator geometry through its Jacobian and the associated metric tensor. Conditioning summarizes the spread of that tensor’s eigenvalues.

2. Background and Notation

  • A GAN has two neural networks: the generator maps random noise to samples, while the discriminator distinguishes training samples from generated ones.The discriminator is trained to maximize its cost, and the generator to minimize its objective.Source: §2 · pp. 1–2
  • The footnote identifies the generator objective as the widely used Non-Saturating GAN formulation, while noting that other formulations exist.The described Jacobian Clamping algorithm doubles the batch size.Source: §2 · pp. 1–2
  • The Inception Score evaluates generated samples using a pretrained classifier’s conditional label probabilities and their overall label distribution.The paper also presents Frechet Inception Distance, which models real and generated classifier activations as multivariate Gaussians.Source: §2 · pp. 2–3
  • For a generator G mapping latent space to observation space, its Jacobian J_z records output derivatives with respect to latent coordinates.The paper studies the distribution of J_z’s squared singular values.Source: §2 · p. 3
  • The matrix J_z^T J_z defines a positive-definite metric tensor whose eigenvalues describe how latent-space steps change the generator output.Eigenvectors with larger eigenvalues correspond to directions where small latent steps produce larger output changes.Source: §2 · p. 3
  • The paper summarizes the metric tensor’s spectrum with its condition number, λ_max/λ_min; high values indicate poor conditioning.The tensor’s eigenvalues equal the squared singular values of J_z.Source: §2 · p. 3

3. Analyzing the Local Geometry of GAN Generators

The section examines generator Jacobian conditioning over training and its relationship to GAN quality metrics. It also compares average-Jacobian spectra across GAN and VAE runs and across datasets.

The Metric Tensor Becomes Ill-Conditioned During Training

  • The condition number starts well-conditioned and quickly becomes poor at fixed MNIST latent points, while across 10 runs it either stays high or decreases after rising rapidly.The reported log-condition numbers range from about 12 to 20.Source: Figure 2 · p. 3, §3 · p. 3
  • Across 10 runs, VAE average-Jacobian spectra were nearly indistinguishable, whereas GAN spectra provide a way to quantify differences in training stability.The VAE decoder was less sensitive to its input than the GAN decoder, yet still successfully modeled MNIST.Source: §3 · pp. 3–4, Figure 3 · p. 4

Conditioning is Predictive of Other Quality Metrics

  • Conditioning corresponds strongly with Inception Score and Frechet Inception Distance on MNIST, CIFAR-10, and STL-10.On MNIST, the four runs with the lowest condition numbers also had the four best scores; the correspondence was also strong on CIFAR-10 and STL-10.Source: §3 · p. 4
  • Average condition number is a first-order approximation of generator sensitivity under the Euclidean metric and does not use the classifier underlying the quality scores.The paper uses this measure to study its correspondence with classifier-based Inception Score and Frechet Inception Distance.Source: §3 · p. 4

Conditioning is Related to Missing Modes

  • −0.86 was the correlation between mean log condition number and the sample count in each model’s least-sampled MNIST class.Ill-conditioned models often produced zero samples from that class, while well-conditioned models were close to uniformly distributed.Source: §3 · pp. 4–5

4. Jacobian Clamping

The authors intervene on generator conditioning with Jacobian Clamping to test its relationship to GAN scores. Across three datasets, clamping lowers conditioning and improves average scores, though it does not improve the best-case performance.

4. Jacobian Clamping

  • The method feeds noise and a perturbed copy through the generator, penalizing output-to-input change ratios outside chosen bounds.The intended effect is to encourage generator-Jacobian singular values to remain within [λmin, λmax]; the method does not directly penalize the condition number.Source: §4 · p. 5
  • Across 3 standard datasets, Jacobian Clamping decreased generator condition numbers and improved score quality.The reported effect was to prevent GANs from entering the ill-conditioned cluster; average-case performance improved, but best-case performance did not.Source: §4 · p. 6
  • The authors limit their claim to evidence for a conditioning–score relationship, not state-of-the-art scores; single-run reports can obscure gains comparable to inter-run variance.They also note that clamping does not directly penalize the condition number because efficiently estimating the smallest eigenvalue is difficult.Source: §4 · pp. 5–6
  • Jacobian Clamping keeps MNIST condition numbers from growing early and yields lower score variance than unregularized runs.Its scores are similar to the best scores achieved without clamping.Source: §4 · p. 6
  • On CIFAR-10, 9 of 10 clamped runs entered the good cluster; on STL-10, all clamped runs did.The poorly scoring CIFAR-10 run also had a high condition number, and its quotient was not successfully constrained.Source: §4 · pp. 6–7
  • With one discriminator update, the class-conditional baseline collapsed while Jacobian Clamping avoided collapse; fewer updates more than halved wall-clock time without reducing score.With five updates, clamping performed slightly worse on Inception Score, a small difference the authors say could reflect untuned hyperparameters.Source: Figure 7 · p. 8, §4 · p. 8

5. Related Work

Related work spans GAN scaling and applications, training pathologies, geometric analyses of neural networks, and comparisons between adversarial and maximum-likelihood training. The paper distinguishes its focus on conditioning variation across GAN runs and its intervention with Jacobian Clamping.

5. Related Work

  • Prior GAN work addresses scaling to larger, higher-resolution datasets, applications, training pathologies, theory, and evaluation.The survey is explicitly incomplete.Source: §5 · p. 8
  • With λmin = 0, Jacobian Clamping is vaguely similar to spectral normalization, which approximately penalizes the discriminator’s individual-layer Jacobian largest singular value.Spectral normalization uses the power method.Source: §5 · p. 8
  • Geometric neural-network work includes derivative penalties, manifold interpretations, classifier invariances, Jacobian spectra and norms, and VAE geometry.The cited examples include Contractive Autoencoders and classifier Jacobian spectra.Source: §5 · p. 8
  • Prior invertible-decoder comparisons found faster spectral drop-off under adversarial than maximum-likelihood training; this paper instead studies variation across runs and intervenes with Jacobian Clamping.The paper also notes that ill-conditioning does not always occur under adversarial training.Source: §5 · p. 8

6. Conclusions and Future Work

The paper tracks generator-Jacobian conditioning during GAN training, relates it to two GAN evaluation metrics, and tests conditioning control with Jacobian Clamping. The intervention improves those metrics, supporting evidence for a causal relationship within the studied setting.

6. Conclusions and Future Work

  • The authors report that Jacobian Clamping improves two quantitative measures of GAN performance.They describe the result as evidence of a causal relationship between generator conditioning and model quality.Source: §6 · p. 8
  • Generator Jacobians generally become ill-conditioned everywhere during training.This is the reported training-dynamics finding.Source: §6 · p. 8
  • Generator-Jacobian conditioning strongly corresponds with two quantitative GAN evaluation metrics.The conclusion describes this correspondence as part of its evidence for a causal relationship.Source: §6 · p. 8

A. Why Compute the Condition Number?

The appendix motivates condition number as a compact summary of the Jacobian spectrum, while noting that other conditioning measures may also be useful. For MNIST, the relative largest singular values of well- and poorly behaved runs support focusing on conditioning rather than log-determinant.

A. Why Compute the Condition Number?

  • The condition number summarizes the Jacobian spectrum using the ratio of its maximum to minimum eigenvalue.The appendix notes that selecting this ratio is not self-evidently justified.Source: A. Why Compute the Condition Number? · p. 11
  • For MNIST, well-behaved runs have distinctly lower largest singular values than poorly behaved runs, suggesting conditioning may matter more than determinant.A glance at the spectra figures could otherwise make log-determinant seem reasonable.Source: A. Why Compute the Condition Number? · p. 11
  • Other conditioning measures that emphasize extreme singular values less may also correlate with GAN performance, but the authors retain the condition number for simplicity and alignment with Jacobian Clamping.Source: A. Why Compute the Condition Number? · p. 11

B. Additional Experimental Results

The appendix section contains results included for completeness rather than for following the paper’s narrative.

B. Additional Experimental Results

B.1. Misbehaving Generators can be Well-Conditioned

A deliberately pathological generator trained without a GAN loss is well-conditioned despite severe mode collapse and mode dropping. This counterexample suggests that the conditioning–performance relationship may be a property of GAN training dynamics, not generators in isolation.

B.1. Misbehaving Generators can be Well-Conditioned

  • The generator memorizes 10,000 latent-to-example associations using L2 reconstruction, with half the targets random MNIST digits and half copies of one four; no discriminator is involved.At evaluation, random latent vectors not seen during training produce collapsed and often indistinct outputs, with a severe missing-mode problem confirmed by a pretrained classifier.Source: §B.1 · p. 11, Figure 10 · p. 12, Figure 9 · p. 12
  • Its Classifier Score is 4.95 for memorized z values and 2.22 for non-memorized z values, while its Frechet Distance is 118 and 240, respectively.The larger Frechet Distance and lower Classifier Score on non-memorized inputs accompany its poor out-of-sample behavior.Source: §B.1 · p. 11
  • A well-conditioned generator trained without a GAN loss nevertheless exhibits conspicuous mode collapse and mode dropping, suggesting conditioning–performance correspondence may be specific to GAN training dynamics.Its condition-number trajectory would suggest membership in the good cluster when considered alone.Source: §B.1 · p. 11, Figure 11 · p. 12

Related papers

Loading 1802.08768v2…