PaperlayerView exact revision on arXiv

Source-linked AI summary

Fisher GAN

Youssef Mroueh, Tom Sercu

arXiv:1705.09675v3cs.LGstat.ML

TL;DR

GAN training stability depends in part on the distance used to compare real and generated distributions. Fisher GAN uses a data-dependent second-order-moment constraint within an IPM and an Augmented Lagrangian training algorithm. The authors report stable, fast training and competitive semi-supervised learning results, and relate Fisher IPM to whitened mean matching and χ2 distance.

  • Problem

    GAN training uses different distribution distances, and the choice of distance affects optimization stability.

    Source: §1 · p. 1

  • Method

    Fisher GAN uses an IPM with a data-dependent second-order-moment constraint on the critic and an Augmented Lagrangian algorithm for training.

    Source: §1 · p. 1, §4 · p. 6

  • Results

    The authors report stable training, fast convergence to good sample quality, and competitive semi-supervised learning performance without critic normalization.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Fisher IPM is presented as a scale-invariant GAN objective with whitened mean-feature and χ2-distance interpretations.

    Source: §6 · p. 9

    With a tractable restricted hypothesis class, approximation of the χ2 distance depends on how well the class approximates the optimal critic.

    Source: §3 · p. 5

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) are powerful models for learning complex distributions. Stable training of GANs has been addressed in many recent works which explore different metrics between distributions. In this paper we introduce Fisher GAN which fits within the Integral Probability Metrics (IPM) framework for training GANs. Fisher GAN defines a critic with a data dependent constraint on its second order moments. We show in this paper that Fisher GAN allows for stable and time efficient training that does not compromise the capacity of the critic, and does not need data independent constraints such as weight clipping. We analyze our Fisher IPM theoretically and provide an algorithm based on Augmented Lagrangian for Fisher GAN. We validate our claims on both image sample generation and semi-supervised classification using Fisher GAN.

1 Introduction

Fisher GAN addresses GAN training stability through an Integral Probability Metric with a data-dependent constraint on the critic’s second-order moments. The paper presents this constraint as a route to stable, efficient training without compromising critic capacity, and reports image-generation and semi-supervised learning results.

1 Introduction

  • Earlier GAN objectives include Jensen-Shannon, f-divergence, KL, least-squares, and earth mover distances, with the choice of distance affecting optimization stability.The introduction says WGAN weight clipping can reduce critic capacity and be sensitive to its hyperparameter, while WGAN-GP’s gradient penalty has high computational cost.Source: §1 · p. 1
  • Fisher GAN constrains the critic’s second-order moments within an IPM, aiming for stable, time-efficient training without compromising critic capacity.The authors contrast this data-dependent constraint with WGAN-GP’s gradient penalty and data-independent constraints.Source: §1 · p. 1
  • The paper’s stated contributions include a scaling-invariant Fisher IPM, theoretical analysis linking it to χ2, and an Augmented Lagrangian algorithm for GAN training.The authors also report strong semi-supervised learning results without batch normalization in the critic.Source: §1 · p. 2
  • Table 1 compares Fisher GAN with recent related approaches.The supplied caption does not report the table’s entries.Source: Table 1 · p. 2

2 Learning GANs with Fisher IPM

Fisher IPM replaces an unconstrained mean discrepancy with a scaling-invariant standardized discrepancy controlled by the critic’s second-order moments. With neural-network features, this becomes whitened mean matching, or equivalently a Mahalanobis distance between feature means.

2.1 Fisher IPM in an arbitrary function space: General framework

  • Scaling the critic can make a standard IPM unbounded, so Fisher IPM standardizes the mean discrepancy as a Rayleigh quotient and constrains second-order moments in a data-dependent way.The paper says this construction bounds the distance and critic while maintaining critic capacity.Source: §2.1 · pp. 2–3
  • The Fisher discrepancy trades off separating the distributions’ function means against reducing their pooled second-order moment.Its scaling-invariant distance also has an equivalent constrained form.Source: §2.1 · p. 3

2.2 Fisher IPM with Neural Networks

  • In the neural-network formulation, the critic is linear in an embedding Φω(x), with v selecting a direction in feature space.A positive regularization term γ is added to avoid singular covariances.Source: §2.2 · p. 4
  • For neural-network features, Fisher IPM is a Mahalanobis distance between the distributions’ mean feature embeddings, equivalently a whitened mean discrepancy.The covariance matrix defines the Mahalanobis distance; the paper notes that WGAN-GP’s gradient penalty yields a similar interpretation.Source: §2.2 · p. 4
  • Training with Fisher IPM sets up a min-max game in which the feature space maximizes Mahalanobis separation between real and generated feature means while the generator minimizes it.The generator is trained to make its distribution close to the real-data distribution under the parametric Fisher IPM.Source: §2.1 · p. 3, §2.2 · p. 4

3 Theory

At full function-space capacity, Fisher IPM equals the χ² distance; in a restricted function class, its approximation depends on how well that class represents the optimal critic.

3 Theory

  • On synthetic 2D data, the MLP estimate is very good with enough samples; at smaller sample sizes it bounds χ² from below and converges toward the ground truth.When χ² is small, a larger training sample is needed for a better estimate.Source: Figure 2 · p. 5, §3 · p. 5
  • At full capacity, Fisher IPM equals the χ² distance between distributions and has a corresponding optimal critic.The result holds for any distinct P and Q.Source: §3 · p. 5
  • LSGAN at full capacity also corresponds to a chi-squared divergence, but uses different generator and discriminator objectives rather than optimizing one distribution distance.Source: §3 · p. 5
  • For a restricted class H, dH(P, Q) never exceeds χ²(P, Q), and approximation quality depends on how well H approximates the optimal critic.A richer class that approximates the optimal critic well yields a good approximation of the χ² distance.Source: §3 · pp. 5–6
  • Generalization error combines approximation error with statistical error bounded using data-dependent local Rademacher complexities.The approximation component is described in Theorem 2; the statistical component is bounded in the supplementary material.Source: §3 · p. 6

4 Fisher GAN Algorithm using ALM

Fisher GAN uses an Augmented Lagrangian to impose its data-dependent constraint while alternating critic and generator updates. The training illustrations show the constraint tightening as the multiplier grows and compare sample quality without batch normalization.

4 Fisher GAN Algorithm using ALM

  • The Augmented Lagrangian combines the objective, a Lagrange-multiplier term for the constraint, and a quadratic penalty.The penalty weight is ρ > 0.Source: §4 · p. 6
  • Training alternates: the critic update maximizes over its parameters while minimizing over λ, then the generator minimizes the objective for the current critic.The constraint is imposed when training the critic only.Source: §4 · p. 6
  • The algorithm uses Adam for critic and generator parameters and SGD with learning rate ρ for the Lagrange multiplier.Source: §4 · p. 6
  • Across three benchmark datasets, the constraint becomes tight as the Lagrange multiplier grows slowly during training.Source: Figure 3 · p. 7
  • Without batch normalization, Fisher GAN produces decent samples while WGAN with weight clipping does not; the authors hypothesize implicit whitening as an explanation.WGAN-GP also converges without batch normalization [7], and both models use a reduced learning rate.Source: Figure 4 · p. 7

5 Experiments

Fisher GAN is evaluated on image generation and semi-supervised learning, with reported stable training, faster convergence, and competitive classification performance. Its CIFAR-10 inception-score comparison reports better scores and a speed advantage over WGAN-GP.

Qualitative: Loss stability and sample quality

  • The experiments report stable training with a meaningful loss that decreases during training and correlates with sample quality, alongside fast convergence to good sample quality.Training uses more discriminator updates on LSUN than on CIFAR-10 and CelebA; the reported settings are nc=5 and nc=2, respectively.Source: §5 · p. 7

Quantitative analysis: Inception Score and Speed

  • Fisher GAN reports better CIFAR-10 inception scores and a clear speed advantage over WGAN-GP.Scores are plotted against generator updates and wall-clock time.Source: §5 · p. 8
  • CIFAR-10 ResNet inception-score experiments used layer normalization, which outperformed unnormalized ResNets, without additional hyperparameter tuning for stable training.The caption identifies the architecture and codebase as those from [7].Source: Table 2 · p. 8

Quantitative analysis: SSL

  • For semi-supervised learning, the critic is trained with a cross-entropy classifier on its feature space, while conditional generation also adds a generator cross-entropy term.The experiments cover CIFAR-10, LSUN, and CelebA, with different architectures and resolutions across tasks.Source: §5 · pp. 7–8
  • On CIFAR-10 semi-supervised learning, Fisher GAN is competitive with literature baselines, recovers performance lost by weight clipping, and the K+1 critic performs better across the board.Conditional generation adds no gains with layer normalization or without normalization.Source: §5 · p. 9, Table 3 · p. 9

6 Conclusion

The conclusion presents Fisher GAN as a stable, fast method based on a scale-invariant IPM constrained by the critic’s second-order moments. It interprets the method through whitened mean feature matching and the χ2 distance.

6 Conclusion

  • Fisher GAN is presented as a stable and fast way to train GANs, using a scale-invariant IPM that constrains the critic’s second-order moments.The paper also interprets it as whitened, Mahalanobis mean feature matching and relates it to the χ2 distance.Source: §6 · p. 9

B Chi-squared distance and Pearson Divergence

  • The conclusion connects Fisher GAN to the χ2 distance, while the supplied passages identify a subsection on χ2 distance and Pearson divergence.The available passage for that subsection labels its definitions and relation but does not state the relation itself.Source: §6 · p. 9, B Chi-squared distance and Pearson… · p. 13

A WGAN-GP versus Fisher GAN

  • Fisher IPM and improved WGAN both perform mean matching with different Mahalanobis distances: Fisher uses pooled covariance, while improved WGAN uses the expected metric tensor.The paper identifies a computational advantage for Fisher GAN.Source: A WGAN-GP versus Fisher GAN · p. 12

C Fisher GAN and ϕ-divergence Based GANs

The section contrasts Fisher’s constrained IPM formulation with ϕ-divergence GANs, focusing on their theoretical relation and different training behavior. It argues that the Fisher constraint controls critic variance during training, whereas the ϕ-divergence formulation can be unstable.

C Fisher GAN and ϕ-divergence Based GANs

  • For symmetric χ2, the Fisher criterion is an IPM with a data-dependent constraint, while the ϕ-divergence formulation is a ϕ-divergence; the formulations are equivalent at the optimum but have different properties.The section also states that Fisher’s formulation gives an inequality, whereas ϕ-GAN gives a lower variational bound.Source: C Fisher GAN and ϕ-divergence Based GANs · pp. 13–14
  • The section says Fisher GAN inherits the known stability of IPM-based GAN training when symmetric χ2 is cast as an IPM with a distribution-dependent function class.The stated theoretical benefit is specific to allowing the function class to depend on the distributions.Source: C Fisher GAN and ϕ-divergence Based GANs · p. 14
  • With a neural critic and linear output, the unconstrained ϕ-divergence formulation can have exploding weight updates and unstable behavior, while Fisher’s slowly imposed constraint controls variance and prevents critic blow-up.The Lagrange multiplier grows slowly to enforce the constraint and approach the χ2 distance as training converges.Source: C Fisher GAN and ϕ-divergence Based GANs · p. 14
  • The section notes that exponential activation functions in the cited ϕ-GAN formulation most likely render it unstable for GAN training.The instability is expressed as a likelihood, not a definitive result.Source: C Fisher GAN and ϕ-divergence Based GANs · p. 14

D Proofs

The proofs characterize Fisher IPM through a χ2 distance and its optimal critic, then establish an approximation guarantee for restricted function classes. The guarantee depends on the class containing a sufficiently good approximation to the optimal critic.

Proofs

  • For the L2 function class under the pooled distribution, the proof writes the Fisher IPM Lagrangian and uses convexity and KKT conditions to characterize the optimum and multiplier.The passages identify the optimal critic as fχ and the multiplier as λ*.Source: D Proofs · p. 15
  • The proof defines a χ2 distance between distributions and gives an expression for its optimal critic.The supplied passages do not provide the expression itself.Source: D Proofs · p. 15
  • If H is rich enough to find, for ε ∈ (0, 1), a 1 −ε approximation of fχ in H ∩ SL2(X, P+Q), then dH is a 1 −ε approximation of χ2(P, Q).Source: D Proofs · p. 16

E Theorem 3: Generalization Bounds

Theorem 3 bounds the relative error in estimating the χ² distance through approximation and statistical error. Its statistical component becomes more demanding when the distributions are close, while the local complexity bound can depend on the effective dimension rather than the ambient dimension.

E Theorem 3: Generalization Bounds

  • The stated complexity and estimation rates are typically O(1/√n), with n the effective sample-size quantity defined in Theorem 3.The effective dimension is described as d(γ) << m.Source: E Theorem 3: Generalization Bounds · pp. 17, 20
  • The relative error combines approximation error of the optimal critic with statistical error from finite-sample estimation.The theorem's decomposition identifies these as separate sources of error.Source: E Theorem 3: Generalization Bounds · p. 18
  • When P and Q are close in χ² distance, more samples are needed to reduce the relative estimation error.The statistical error is divided by the χ² distance in the relative-error expression.Source: E Theorem 3: Generalization Bounds · pp. 18, 21
  • For the simple linear function class, the local Rademacher-complexity bound uses effective dimension d(γ), typically much smaller than ambient dimension m.Without the local constraint, d(γ) is replaced by m.Source: E Theorem 3: Generalization Bounds · p. 22

F Hyper-parameters and Architectures of Discriminator and Generators

The appendix reports optimization settings for CIFAR-10, LSUN, and CelebA and describes initialization and weight-decay choices. The authors report stable optimization across a range of learning rates and penalty weights.

F Hyper-parameters and Architectures of Discriminator and Generators

F.1 Inception score WGAN-GP baselines: comparison of architecture and weight initialization

The appendix documents architecture and initialization variants for Inception-score comparisons, alongside network specifications for the LSUN, CelebA, CIFAR-10 generation, and CIFAR-10 semi-supervised experiments. The supplied passages identify comparison setups but do not report numerical outcomes for the figures.

F.1 Inception score WGAN-GP baselines: comparison of architecture and weight initialization

  • For WGAN, the 64-feature-map architecture starts with some instability but catches up to the 128-feature-map architecture.F64 uses 64 feature maps after the first convolution; F128 has 128 and omits two extra generator and discriminator layers.Source: §F.1 · p. 23
  • Figure 6 compares architecture and initialization variations trained with WGAN-GP, while Figure 7 compares architecture variations trained with WGAN.The main-text Figure 5 comparison uses the best architecture, F64 with initialization 0.02.Source: Figure 6 · p. 23, Figure 7 · p. 23, §F.1 · p. 23

F.2 LSUN and CelebA.

  • The appendix provides network-layer specifications for the LSUN and CelebA experiments.Source: §F.2 · pp. 23–24

F.3 CIFAR-10: Sample Quality and Inceptions Scores Experiments

  • For samples and inception, the text lists network-layer configurations, including generator transposed-convolution blocks and an S layer specified as Linear(6144 -> 10).Source: §F.3 · pp. 24–25

F.4 CIFAR-10: SSL Experiments

  • The CIFAR-10 semi-supervised experiment uses a discriminator described as following the OpenAI Improved GAN and ALI flavor, with the generator as above.Source: §F.4 · p. 25

G Sample implementation in PyTorch

The appendix supplies a minimal PyTorch implementation of Fisher GAN and identifies the changes needed to adapt the reference Wasserstein GAN code. Its training loop constrains the critic's pooled second moments with a Lagrange multiplier and updates the generator using the fake-sample mean score.

G Sample implementation in PyTorch

  • The sample implementation is based on the WassersteinGAN repository at commit d92c503 and lists optional additions, including monitoring and a class-conditioned generator.It also notes separate weight decay for the last layer v.Source: G Sample implementation in PyTorch · p. 26

G.1 Main loop

  • The discriminator loop computes real and fake mean scores and squared-score means, then uses their pooled second moment in a penalized constraint with a Lagrange multiplier.The multiplier is updated after the discriminator step.Source: §G.1 · p. 27
  • The generator update minimizes the negative mean critic score on generated samples.Source: §G.1 · p. 27

G.2 Full diff from reference

  • The reference-code diff replaces weight clipping with the Fisher moment constraint and introduces the multiplier and penalty parameter.The diff also changes the discriminator and generator objectives to use critic scores.Source: §G.2 · pp. 27–29
Loading 1705.09675v3…