PaperlayerView exact revision on arXiv

Source-linked AI summary

On Convergence and Stability of GANs

Naveen Kodali, Jacob Abernethy, James Hays, Zsolt Kira

arXiv:1705.07215v5cs.AIcs.CVcs.GTcs.LGcs.NE

TL;DR

GAN training is unstable and often suffers mode collapse, while common analyses assume a discriminator that is optimal at each step. This paper studies training as regret minimization and introduces DRAGAN, reporting faster training and improved stability and modeling performance across architectures and objectives.

  • Problem

    GAN training is unstable and often suffers mode collapse, while much theory assumes consistent divergence minimization with an optimal discriminator at each step.

    Source: §1 · p. 1

  • Method

    The paper analyzes GAN training as a repeated game with no-regret players and proposes DRAGAN, a gradient penalty motivated by sharp discriminator gradients around real data points.

    Source: §1 · pp. 1–2, §2.4 · p. 5

  • Results

    DRAGAN enables faster training and improves stability and modeling performance over WGAN-GP across varied architectures and objectives.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that constraining discriminator gradients around real points can mitigate mode collapse associated with undesirable local equilibria.

    Source: §2.4 · pp. 5–6

    The asymptotic convergence result in the convex-concave analysis requires a fresh sample on every round.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

We propose studying GAN training dynamics as regret minimization, which is in contrast to the popular view that there is consistent minimization of a divergence between real and generated distributions. We analyze the convergence of GAN training from this new point of view to understand why mode collapse happens. We hypothesize the existence of undesirable local equilibria in this non-convex game to be responsible for mode collapse. We observe that these local equilibria often exhibit sharp gradients of the discriminator function around some real data points. We demonstrate that these degenerate local equilibria can be avoided with a gradient penalty scheme called DRAGAN. We show that DRAGAN enables faster training, achieves improved stability with fewer mode collapses, and leads to generator networks with better modeling performance across a variety of architectures and objective functions.

1 INTRODUCTION

The introduction reframes GAN training as regret minimization and investigates how undesirable local equilibria may relate to mode collapse. It motivates DRAGAN as a gradient-penalty approach and reports faster training with improved stability and modeling performance.

1 INTRODUCTION

  • The paper models GAN training as a repeated game in which both players use no-regret algorithms, rather than assuming consistent divergence minimization.It argues regret minimization better captures training dynamics, while the divergence view may still help reason about asymptotic equilibrium.Source: §1 · p. 1
  • In the non-parametric limit, the paper gives an asymptotic convergence proof without requiring an optimal discriminator at each step.For the artificial convex-concave case, it also describes a unique solution and convergence of averaged iterates; most analysis applies to simultaneous updates as well.Source: §1 · pp. 1–2
  • The authors hypothesize that undesirable local equilibria in non-convex games contribute to mode collapse, which often accompanies sharp discriminator gradients around real points.They argue that sharp gradients can encourage multiple latent vectors to map to one output.Source: §1 · p. 2
  • The convergence results do not extend to practical non-convex games; the authors focus on mode collapse rather than the cycling they also discuss.They note that global regret minimization and equilibrium computation are generally hard in non-convex games.Source: §1 · p. 2
  • DRAGAN enables faster training and improves stability and modeling performance over WGAN-GP across varied architectures and objectives.The paper presents it as a way to mitigate mode collapse.Source: §1 · p. 2
  • The introduction identifies a need for stable GAN training that is fast and versatile, given limitations in architecture requirements, framework changes, or computational overhead of prior approaches.The authors position their work as addressing this gap.Source: §1 · p. 2

2 THEORETICAL ANALYSIS OF GAN TRAINING DYNAMICS

The analysis reframes GAN training as regret minimization, then contrasts convergence in the convex-concave setting with local equilibria and mode collapse in practical non-convex games. It uses sharp discriminator gradients around real samples to motivate DRAGAN and distinguishes local from coupled gradient penalties.

2.1 BACKGROUND

  • A GAN is a repeated zero-sum game in which the generator maps noise to samples and the discriminator estimates whether an input is real.The models may be drawn from arbitrary function classes; in practice, both typically use deep networks.Source: §2.1 · p. 3

2.2 CONVEX-CONCAVE CASE AND NO-REGRET ALGORITHMS

  • In the non-parametric limit, no-regret analysis gives asymptotic GAN convergence without assuming the discriminator is optimal at every step.The argument connects alternating gradient updates to online gradient descent and convergence of averaged iterates in the convex-concave case.Source: §2.2 · p. 4

2.3 NON-CONVEX CASE AND LOCAL EQUILIBRIA

  • With practical non-convex networks, guaranteed convergence no longer follows; training may cycle or reach a local equilibrium, which the authors hypothesize can underlie mode collapse.Mode collapse is described as multiple noise vectors mapping to the same output that the discriminator assigns a high probability of being real.Source: §2.3 · pp. 4–5

2.4 MODE COLLAPSE AND GRADIENT PENALTIES

  • Mode-collapse cases often accompany sharp discriminator gradients around real samples, motivating DRAGAN’s constraint on discriminator gradients near real data.The authors report that over-penalization can make the discriminator assign the same real probability to a data point and its noisy perturbation; the chosen penalty assumes small perturbations are likely off-manifold.Source: §2.4 · pp. 5–6

2.5 COUPLED VS LOCAL PENALTIES

  • Local penalties constrain gradients around real samples, whereas coupled penalties constrain gradients between real and generated samples.The authors say coupled penalties can restrict the function class and modeling performance, may require multiple discriminator updates, and use generated samples; they also question WGAN-GP’s arbitrary-pair penalty’s basis in KR-duality.Source: §2.5 · p. 7

3 EXPERIMENTAL RESULTS

The experiments compare DRAGAN with vanilla GAN training and WGAN variants across CIFAR-10, varied architectures, and multiple objectives. DRAGAN improves modeling performance in the DCGAN/CIFAR-10 comparison and is more stable across the tested architectures and most tested objectives.

3.1 INCEPTION SCORES FOR CIFAR-10 USING DCGAN ARCHITECTURE

  • On CIFAR-10 with DCGAN, DRAGAN beats WGAN variants under both iteration-matching configurations, while vanilla GAN is only slightly better.The authors note that DRAGAN is faster than WGANs, so practical performance may be closer to the DRAGANd case.Source: §3.1 · p. 8, Figure 5 · p. 9

3.2 MEASURING STABILITY AND PERFORMANCE ACROSS ARCHITECTURES

  • Across 100 hard CIFAR-10 architecture pairings, DRAGAN outperforms the other algorithms in both average final Inception score and area under the curve.The comparison uses a diverse pool of MLP, ResNet, and DCGAN architectures; the authors note that WGAN-GP had a slight advantage from the fixed-generator-iteration setup.Source: §3.2 · p. 9, Table 1 · p. 9
  • In a blind qualitative comparison across 50 hard architectures, DRAGAN receives an average score of 157.5 versus 92.5 for vanilla GAN.Two authors judged randomized, unlabeled side-by-side curves, allocating five points per architecture based on relative stability.Source: §3.2 · p. 9

3.3 STABILITY USING DIFFERENT OBJECTIVE FUNCTIONS

  • DRAGAN is stable for Forward KL, Reverse KL, Pearson χ2, and Squared Hellinger objectives, but not Total Variation; vanilla training fails in all five tested cases.Figure 6 shows two examples, with all five objectives shown in appendix Figure 15.Source: §3.3 · pp. 9–10, Figure 6 · p. 10

4 CONCLUSIONS

The paper reframes GAN training dynamics as regret minimization rather than consistent divergence minimization. It hypothesizes that undesirable local equilibria underlie mode collapse and motivates DRAGAN, reporting improved stability and modeling performance across settings.

4 CONCLUSIONS

  • The paper frames GAN training as regret minimization, hypothesizes undesirable local equilibria as a source of mode collapse, and reports improved stability and modeling performance with DRAGAN across settings.The authors also observe that mode-collapse situations often exhibit sharp discriminator gradients around some real data points and leave deeper exploration to future work.Source: §4 · p. 10

5 APPENDIX

The appendix presents generated samples and latent walks, a one-hidden-layer MNIST experiment, and an 8-Gaussians comparison. It also restates selected DCGAN architecture guidelines.

5.1 SAMPLES AND LATENT SPACE WALKS

  • DRAGAN-trained DCGAN models show smooth transitions between images in MNIST and CelebA latent walks, alongside additional CelebA and CIFAR-10 samples.The latent walks are shown in Figures 9 and 10; the additional CelebA samples are in Figure 7 and the earlier CIFAR-10 samples in Figure 8.Source: §5.1 · p. 12, Figure 7 · p. 12, Figure 8 · p. 13

5.2.1 ONE HIDDEN LAYER NETWORK TO MODEL MNIST

  • With fully connected generator and discriminator networks having one hidden layer, vanilla GAN shows severe mode collapse, while DRAGAN remains stable and produces decent-quality samples.The experiment uses MNIST and a constrained network setup.Source: §5.2.1 · p. 14, Figure 11 · p. 15, Figure 12 · p. 15

5.2.2 8-GAUSSIANS EXPERIMENT

  • On 8-Gaussians, WGAN-GP and DRAGAN both approximately converge to the real distribution, but WGAN-GP’s discriminator appears more constrained in data space than DRAGAN’s.Figure 13 shows real samples in orange, generated samples in green, and discriminator level sets in the background.Source: §5.2.2 · p. 15, Figure 13 · p. 15

5.2.3 STABILITY ACROSS DCGAN ARCHITECTURE VARIATIONS

  • The appendix restates DCGAN design guidance, including all-convolutional networks for learned spatial upsampling or downsampling and ReLU generator activations except for a tanh output layer.The cited guidelines concern architecture design; these passages do not report a comparison outcome for the architecture variations.Source: §5.2.3 · p. 16

3. Use batch normalization in both the generator and the discriminator

The listed DCGAN guideline specifies ReLU activations in the generator except at its output layer.

3. Use batch normalization in both the generator and the discriminator

  • Use ReLU activation in every generator layer except the output layer, which uses tanh.This is listed as the fourth DCGAN design guideline.Source: §3 · p. 16

5. Use LeakyReLU activation in the discriminator for all layers

The experiments report that DRAGAN remains stable where vanilla GAN training fails on difficult DCGAN variants. The section also compares training across objective functions and describes how BogoNet constructs hard games.

5. Use LeakyReLU activation in the discriminator for all layers

  • DRAGAN remains stable while vanilla GAN training fails across four difficult DCGAN variants.The variants remove batch normalization, change generator structure or nonlinearities, or use a different discriminator configuration.Source: §5 · p. 16

5.2.4 STABILITY ACROSS OBJECTIVE FUNCTIONS

  • The objective-function experiments compare DRAGAN and vanilla GAN training across five cases.Results are presented for all five cases; Figure 15 compares performance using different objective functions.Source: §5.2.4 · p. 17, Figure 15 · p. 18

5.3 BOGONET DETAILS

  • BogoNet samples DCGAN, ResNet, and MLP architectures, then varies their configurations to create hard games for testing training stability.The family probabilities are DCGAN 0.6, ResNet 0.2, and MLP 0.2.Source: §5.3 · p. 18
  • BogoNet details include bounty-split examples, with averages listed in DRAGAN-then-vanilla order.Figure 14 plots are among the examples discussed.Source: Figure 14 · p. 17, §5.3 · p. 18

Related papers

Loading 1705.07215v5…