PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Training Methods for GANs do actually Converge?

Lars Mescheder, Andreas Geiger, Sebastian Nowozin

arXiv:1801.04406v4cs.LGcs.AIcs.GT

TL;DR

The paper asks when GAN training converges, particularly when data and generator distributions lie on lower-dimensional manifolds. It analyzes a prototypical counterexample and regularization strategies, showing that unregularized training and some finite-update Wasserstein methods can fail while instance noise and zero-centered penalties converge. It then extends convergence results to more general GANs and simplified gradient penalties.

  • Problem

    GAN training is often difficult to make convergent, while existing local-convergence results assume absolutely continuous data and generator distributions despite common lower-dimensional manifold supports.

  • Method

    The paper analytically studies GAN dynamics on a simple prototypical counterexample, evaluates stabilization techniques, and extends the analysis to general GANs with simplified gradient penalties.

  • Results

    Unregularized GAN training is not always locally convergent; WGANs and WGAN-GP with finitely many discriminator updates may also fail, whereas instance noise and zero-centered gradient penalties lead to local convergence.

  • Takeaways & Limitations

    Simplified zero-centered gradient penalties have proved local convergence under suitable assumptions and worked well for high-resolution generative image models with little hyperparameter tuning.

  • Takeaways & Limitations

    The existence conditions for stable equilibria of unregularized GAN training remain unclear for high-dimensional data distributions.

Abstract

from arXiv · show

Recent work has shown local convergence of GAN training for absolutely continuous data and generator distributions. In this paper, we show that the requirement of absolute continuity is necessary: we describe a simple yet prototypical counterexample showing that in the more realistic case of distributions that are not absolutely continuous, unregularized GAN training is not always convergent. Furthermore, we discuss regularization strategies that were recently proposed to stabilize GAN training. Our analysis shows that GAN training with instance noise or zero-centered gradient penalties converges. On the other hand, we show that Wasserstein-GANs and WGAN-GP with a finite number of discriminator updates per generator update do not always converge to the equilibrium point. We discuss these results, leading us to a new explanation for the stability problems of GAN training. Based on our analysis, we extend our convergence results to more general GANs and prove local convergence for simplified gradient penalties even if the generator and data distribution lie on lower dimensional manifolds. We find these penalties to work well in practice and use them to learn high-resolution generative image models for a variety of datasets with little hyperparameter tuning.

1. Introduction

GAN training is powerful but often fails to converge, especially when data and generator distributions are not absolutely continuous. This paper analyzes the failure and identifies regularization methods that restore local convergence.

  • Motivation: GAN optimization is often observed not to converge despite GANs' power for learning complex real-world distributions.This motivates research into improved training algorithms and theoretical analyses of GAN dynamics.
  • Existing theory: Prior theory established local convergence for absolutely continuous data and generator distributions, but this assumption excludes distributions supported on lower-dimensional manifolds.Such manifold-supported distributions are common use cases, making the assumption practically restrictive.
  • Main analysis: The paper gives a prototypical counterexample showing that unregularized GAN training is not always locally convergent.It analytically examines how recently proposed stabilization techniques affect convergence on this example.
  • Regularization results: WGANs, WGAN-GP, and DRAGAN do not converge on the example with a fixed number of discriminator updates per generator update.The result concerns practical finite-update training rather than WGAN training with a discriminator optimized fully to convergence.
  • Regularization results: Instance noise, zero-centered gradient penalties, and consensus optimization lead to local convergence in the analyzed setting.The paper further introduces simplified gradient penalties and proves local convergence for regularized GAN dynamics in more general cases.

2. Instabilities in GAN training

The paper uses the Dirac-GAN as a prototypical counterexample showing that unregularized GAN training can fail to converge when distributions are concentrated on lower-dimensional manifolds. Its dynamics oscillate or become unstable because the local gradient structure lacks a convergent component.

  • 2.3. Where do instabilities come from?: The counterexample violates support and discriminator-continuity assumptions used by earlier convergence results, while lower-dimensional-manifold distributions remain common in GAN applications.Even absolutely continuous distributions concentrated near a lower-dimensional manifold can yield eigenvalues close to the imaginary axis and an ill-conditioned problem.
  • 2.2. The Dirac-GAN: The Dirac-GAN has one generator parameter and one discriminator parameter, with generator distribution δθ, linear discriminator Dψ(x)=ψ·x, and data concentrated at 0.The unique equilibrium is θ=ψ=0.
  • 2.2. The Dirac-GAN: The equilibrium’s gradient-field Jacobian has eigenvalues ±f′(0)i, placing both eigenvalues on the imaginary axis rather than in the negative-real-part region required for local convergence.This explains why the standard local-convergence criterion does not apply to the example.
  • 2.2. The Dirac-GAN: Every continuous-system trajectory preserves θ(t)^2+ψ(t)^2, so integral curves do not converge to the Nash-equilibrium.The conserved norm implies persistent motion around the equilibrium.
  • 2.2. The Dirac-GAN: For alternating gradient descent, α≤2 places all update eigenvalues on the unit circle, while α>2 produces eigenvalues outside it.The continuous system’s nonconvergence and observed stable cycles make sublinear convergence unlikely.

3. Regularization strategies

The analysis compares regularization strategies for the Dirac-GAN. Instance noise and zero-centered gradient penalties yield local convergence, whereas WGANs and WGAN-GP do not generally converge under fixed discriminator-update schedules.

  • Wasserstein GAN: WGANs generally do not converge to the Nash equilibrium with fixed discriminator updates per generator update and fixed learning rate.The result applies to simultaneous or alternating gradient descent for the Dirac-GAN.
  • WGAN-GP: WGAN-GP also does not converge for the Dirac-GAN, although the paper leaves its successful practical applications for future theoretical analysis.The negative result is stated for the analyzed Dirac-GAN example.
  • Instance noise: Gaussian instance noise makes all Jacobian eigenvalues have negative real parts when σ > 0, yielding local convergence for sufficiently small learning rates.This applies to both simultaneous and alternating gradient descent.
  • Instance noise: σcritical^2 = |f′(0)|/|f′′(0)| removes the rotational component when σ = σcritical and can yield quadratic convergence for h = |f′(0)|^-1.Below the critical level, eigenvalues retain nonzero imaginary parts; above it, they become real-valued.
  • Zero-centered gradient penalties: Zero-centered gradient penalties produce local convergence for γ > 0 because all Jacobian eigenvalues have negative real parts.At γcritical = 2|f′(0)|, the local vector field becomes rotation-free and behaves like a normal optimization problem.

4. General convergence results

The paper extends convergence analysis to general GANs using simplified zero-centered gradient penalties. Under stated regularity and realizability assumptions, both simultaneous and alternating updates converge locally at least linearly, including when distributions lie on lower-dimensional manifolds.

  • Simplified gradient penalties: Simplified zero-centered gradient penalties extend convergence proofs to generator and data distributions with different local supports.The penalties are introduced as simplified versions of Roth et al. (2017), with variants penalizing gradients on true data or generated data.
  • Simplified gradient penalties: The regularized convergence proof applies when generator and data distributions lie on lower-dimensional manifolds.This addresses the more realistic setting excluded by absolute-continuity assumptions.
  • Assumptions: The analysis assumes realizability, with generator parameters capable of producing the true data distribution and an equilibrium discriminator equal to zero near the data support.It also assumes the specified regularity properties of the generator and discriminator manifolds and the discriminator objective.
  • Convergence theorem: Under Assumptions I, II, and III, simultaneous and alternating gradient descent converge to MG × MD near the equilibrium for sufficiently small learning rates.The convergence rate is at least linear.
  • Convergence theorem: The result is local rather than global: sufficiently close initialization is required, and the theorem establishes well-behaved dynamics near the equilibrium.The proof handles reparameterization by defining equilibrium manifolds MG and MD rather than a single parameter point.
  • Stable equilibria for unregularized GAN training: Unregularized GANs may still possess stable equilibria, but the conditions under which energy and full-rank solutions exist for high-dimensional data remain unclear.The paper identifies these stable-equilibrium forms without establishing their general existence conditions.

5. Experiments

The experiments compare five training methods across four 2D distributions and six architectures using Wasserstein-1 distance. R1 and R2 regularizers perform similarly and slightly better than unregularized training and WGAN-GP, while R1 enables stable high-resolution image-model training.

  • 2D-Problems: The study evaluates five training algorithms on four 2D examples and six GAN architectures using estimated Wasserstein-1 distance.Each method is tested with SGD and RMSProp across learning rates and regularization parameters, with results reported after 50k iterations using the best hyperparameter setup.
  • 2D-Problems: R1- and R2-regularizers perform similarly and achieve slightly better results than unregularized training or WGAN-GP.The reported comparison is based on Wasserstein-1 distance to the true data distribution.
  • 2D-Problems: R1- and R2-regularizers find solutions with discriminator value 0 near the true data distribution, unlike unregularized training and WGAN-GP, which converge to energy solutions.This distinction is reported in the supplementary analysis of the 2D experiments.
  • Images: On high-resolution image tasks, unregularized GAN training quickly leads to mode collapse, whereas the simple R1-regularizer enables stable training.The models include all 1000 ImageNet classes and CelebA-HQ at 1024×1024 resolution.

6. Conclusion

The paper analyzes GAN stability on a prototypical example, shows that several common training schemes need not converge, and proves convergence for instance noise and simplified zero-centered penalties under suitable assumptions. It also extends the analysis to more general GANs and identifies future work on non-realizable settings and finite sampling.

  • Conclusion: The analysis shows that unregularized gradient-based GAN optimization is not always locally convergent on a simple prototypical example.The same example is used to study the stability of several regularization strategies.
  • Conclusion: WGANs and WGAN-GP do not always lead to local convergence, whereas instance noise and zero-centered gradient penalties do.The conclusion concerns the analyzed training dynamics and regularization settings.
  • Conclusion: The paper proves local convergence for simplified zero-centered gradient penalties under suitable assumptions, including more general GAN settings.The extension addresses cases where generator and data distributions can lie on lower-dimensional manifolds.
  • Conclusion: Future work includes extending the theory to the non-realizable case and examining finite sampling sizes in GAN training dynamics.

Supplementary Material

The supplementary material formulates GAN training as discrete fixed-point dynamics and analyzes local convergence through Jacobian spectra. It extends the convergence theorem to manifolds of fixed points and relates simultaneous and alternating updates to eigenvalue conditions.

  • A.1. Discrete dynamical systems: A continuously differentiable fixed-point iteration is locally convergent when every Jacobian eigenvalue at the fixed point has absolute value below 1.The convergence rate is at least linear and is governed by the eigenvalue with largest absolute value.
  • A.1. Discrete dynamical systems: The framework distinguishes stable, asymptotically stable, and exponentially stable fixed points, with exponential stability implying linear convergence.
  • A.1. Discrete dynamical systems: A discrete convergence theorem extends the fixed-point result to a local manifold of equivalent fixed points when transverse Jacobian eigenvalues have absolute value below 1.The theorem gives linear convergence toward the manifold, with rate determined by the largest eigenvalue modulus.
  • A.2. Simultaneous and Alternating Gradient Descent: Simultaneous gradient descent has update eigenvalues λ = 1 + hµ, so convergence requires the resulting eigenvalues to lie inside the unit circle.Here µ denotes an eigenvalue of the gradient-vector-field Jacobian and h is the learning rate.
  • A.2. Simultaneous and Alternating Gradient Descent: For alternating gradient descent, negative-real-part gradient-field eigenvalues imply update eigenvalues inside the unit circle for sufficiently small learning rates.The update operator is composed from separate generator and discriminator update operators.
  • A.2. Simultaneous and Alternating Gradient Descent: Local-coordinate and spectrum arguments allow the convergence theorem to apply when stationary points lie on lower-dimensional manifolds rather than coordinate subspaces.

B. Proofs for the Dirac-GAN

The Dirac-GAN analysis shows that unregularized dynamics cycle or expand rather than converge, while instance noise and gradient regularization move the Jacobian spectrum into the convergent region. WGANs also generally fail to converge with fixed update counts and learning rates.

  • Unregularized dynamics: The Dirac-GAN has a unique equilibrium at θ = ψ = 0, while the gradient-vector-field Jacobian has eigenvalues ±f′(0)i on the imaginary axis.
  • Unregularized dynamics: Every continuous gradient-flow trajectory preserves θ(t)^2 + ψ(t)^2, so it does not converge to the Nash equilibrium.
  • Unregularized dynamics: Simultaneous gradient descent is unstable for every learning rate, with iterates whose norms increase monotonically from every initial condition.
  • Unregularized dynamics: For alternating updates, eigenvalues lie on the unit circle when α ≤ 2 and some lie outside it when α > 2.Here α = √(n_g n_d) h f′(0).
  • WGAN: WGAN training with a fixed discriminator-update count and fixed learning rate generally does not converge to the Dirac-GAN Nash equilibrium.The result applies to both simultaneous and alternating gradient descent.
  • Regularization: With Gaussian instance noise, all Jacobian eigenvalues have negative real parts when f′′(0) < 0 and σ > 0, yielding local convergence for small learning rates.
  • Regularization: For gradient-regularized Dirac-GANs, γ > 0 makes all Jacobian eigenvalues have negative real parts, so both update schemes converge locally for sufficiently small learning rates.

C.1. Nonsaturating GAN

The nonsaturating objective prevents generator-loss saturation, but its local Jacobian still has imaginary-axis eigenvalues, so discrete simultaneous updates are not locally convergent. In continuous time, however, the example converges logarithmically, while several stabilizing regularizers fail under fixed discriminator-update counts.

  • Nonsaturating objective: The nonsaturating objective replaces the generator objective to avoid vanishing gradient information when the discriminator confidently rejects generated samples.This addresses the saturation problem observed early in GAN training.
  • Local dynamics: The unique Nash equilibrium is θ = ψ = 0, but the Jacobian eigenvalues are ±f′(0)i on the imaginary axis.The eigenvalue condition explains why local convergence is not obtained from the linear analysis.
  • Local dynamics: Simultaneous gradient descent is not locally convergent for the nonsaturating GAN at any learning rate, and alternating descent is likewise covered by the same argument.The discrete iterates fail to converge to the equilibrium because their updates do not form a Cauchy sequence.
  • Continuous dynamics: The continuous nonsaturating Dirac-GAN dynamics converge with logarithmic convergence rate when f is concave and f′′(0) < 0.For the standard choice f(t) = −log(1 + exp(−t)), f′′(0) = −1/4, so the stated result applies.
  • Regularization comparisons: WGAN-GP and DRAGAN do not converge on the Dirac-GAN example when trained with a fixed number of discriminator updates.For WGAN-GP, the regularized gradient field is discontinuous at equilibrium, and the result applies to simultaneous or alternating descent.
  • Regularization comparisons: Consensus optimization shifts the equilibrium Jacobian eigenvalues to negative real part −γf′(0)^2, yielding local convergence for sufficiently small learning rates.The result holds for both simultaneous and alternating gradient descent, although consensus optimization can introduce spurious attractive points.

D. General convergence results

The paper extends convergence analysis beyond absolutely continuous distributions by introducing gradient penalties that remove the shared-support assumption. This makes the theory applicable when generator and data distributions lie on lower-dimensional manifolds.

  • General convergence results: The convergence proof is extended using gradient penalties that eliminate the assumption that generator and data distributions locally have the same support.The extension targets cases where both distributions lie on lower-dimensional manifolds.

D.1. Convergence proof

The proof analyzes the regularized gradient field near the manifold of equilibria and shows that its transverse Jacobian has eigenvalues with negative real parts. Consequently, both simultaneous and alternating updates converge locally at least linearly for sufficiently small learning rates.

  • Proof strategy: The proof studies the regularized gradient field and restricts its Jacobian to directions orthogonal to the equilibrium manifolds’ tangent spaces.This isolates the directions in which convergence toward the equilibrium manifold must be established.
  • Convergence theorem: For small enough learning rates, simultaneous and alternating gradient descent converge locally to MG × MD with a rate of at least linear.The result applies to both regularized vector fields considered in the theorem.
  • Equilibrium structure: The regularizers preserve the equilibrium manifold MG × MD because the regularized gradient field vanishes at every point on it.This follows from the discriminator and input-gradient conditions on the data support.
  • Jacobian analysis: The restricted Jacobian has only eigenvalues with negative real parts under the stated assumptions.This is the key spectral condition used to invoke the local convergence theorem.
  • Jacobian analysis: The restricted discriminator block is symmetric negative definite, while the generator–discriminator block has full column rank.These properties provide the matrix conditions needed for negative-real-part eigenvalues.

D.2. Extensions

The convergence framework extends to Wasserstein objectives and a broader class of regularizers, while the paper also characterizes stable equilibria that can occur without regularization under additional discriminator assumptions.

  • Wasserstein extension: The convergence proof extends to f(t) = t, the Wasserstein objective, after modifying Assumption III slightly.The modified setting uses an adjusted discriminator equilibrium manifold.
  • Regularizer extensions: The adaptive regularizer of Roth et al. can be covered by the convergence result when its two weights are positive at zero.This regularizer is a weighted combination of the two simplified gradient penalties.
  • Regularizer extensions: Zero-centered gradient penalties make general GANs locally convergent under suitable assumptions, even though unregularized training is not always locally convergent.The convergence claim concerns the regularized training dynamics rather than all GAN architectures without assumptions.
  • Stable equilibria: Unregularized GANs can nevertheless have stable equilibria when additional assumptions constrain the class of representable discriminators.The paper identifies energy solutions and full-rank solutions as two such equilibrium types.
  • Stable equilibria: Energy solutions associate the discriminator with an energy function, while full-rank solutions associate it with a representation of the data distribution having specific properties.The paper connects these equilibrium types to possible explanations for autoencoder-based discriminators and batch normalization, respectively.

E.1. Energy Solutions

Energy solutions can yield locally convergent unregularized GAN dynamics under manifold and discriminator regularity assumptions, but their applicability and conditioning are limited.

  • Energy solutions require the discriminator to form a potential function for the true data distribution.
  • The assumptions require smooth equilibrium and reparameterization manifolds, nonzero off-manifold generator responses, and discriminator gradients detecting departures from equilibrium.
  • Under Assumptions I′, II, and III′, simultaneous and alternating gradient descent converge locally for small learning rates, with at least linear convergence.
  • For the Dirac-GAN, a quadratic discriminator produces locally convergent energy solutions when θ = 0, ψ2 = 0, and ψ1 > 0.
  • Energy solutions may be poorly conditioned, and discretized training can fail to attract even when continuous dynamics are locally convergent.
  • Complete full-rank representations also yield local linear convergence, but they may require powerful discriminators or limited generator classes and can be difficult to find.

F. Experiments

Experiments evaluate regularized and unregularized GAN training on two-dimensional distributions and image datasets, finding that simplified gradient penalties stabilize training across resolutions.

  • 2D-Problems: On two-dimensional problems, R1- and R2-regularizers converge to solutions with zero discriminator near the data, whereas unregularized training and WGAN-GP converge to energy solutions.
  • CIFAR-10: On CIFAR-10, R1- and R2-regularizers produce stable training, while unregularized GAN training is unstable.
  • CIFAR-10: WGAN-GP with one or five discriminator updates per generator update achieves similar final inception scores on the tested CIFAR-10 architecture.
  • Imagenet: Without regularization, the Imagenet GAN quickly collapses modes, whereas R1-regularization yields stable training and convincing samples from all 1000 classes.
  • Imagenet: On Imagenet, R1-regularization achieves higher inception scores than WGAN-GP for the tested dataset and architecture.
Loading 1801.04406v4…