PaperlayerView exact revision on arXiv

Source-linked AI summary

Stabilizing Generative Adversarial Networks: A Survey

Maciej Wiatrak, Stefano V. Albrecht, Andrew Nystrom

arXiv:1910.00927v2cs.LG

TL;DR

GAN training is unstable, with failures including mode collapse, vanishing gradients, and non-convergence. This survey categorizes and compares stabilization approaches; it finds architectural variants have the highest empirical improvement, while approaches retain important limitations.

  • Problem

    GAN training remains vulnerable to instability, including mode collapse, non-convergence, and vanishing or exploding gradients, while evaluation lacks a comprehensive metric.

    Source: §1 · p. 1, §2.1.3 · pp. 4–5

  • Method

    The survey reviews GAN stabilization methods, categorizes variants, compares their advantages and disadvantages, and outlines research directions.

    Source: §6 · p. 18

  • Results

    Architecture variants currently show the highest empirical improvement, while loss-function variants often yield more general training improvement and multi-agent methods mainly target mode collapse.

    Source: §3.7 · pp. 14–15

  • Takeaways & Limitations

    Architectural changes can improve stability, but improving training without added complexity and addressing the game beyond architecture remain challenges.

    Source: §3.1.5 · p. 8

    BigGAN's training improvement comes at great computational cost, and architectural changes alone do not resolve the non-convex-concave game challenge.

    Source: §3.1.5 · p. 8

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) are a type of generative model which have received much attention due to their ability to model complex real-world data. Despite their recent successes, the process of training GANs remains challenging, suffering from instability problems such as non-convergence, vanishing or exploding gradients, and mode collapse. In recent years, a diverse set of approaches have been proposed which focus on stabilizing the GAN training procedure. The purpose of this survey is to provide a comprehensive overview of the GAN training stabilization methods which can be found in the literature. We discuss the advantages and disadvantages of each approach, offer a comparative summary, and conclude with a discussion of open problems.

1. Introduction

GANs can model complex real-world data, but their training remains unstable and often requires manual tuning. This survey organizes stabilization methods, compares their trade-offs, and identifies open problems, focusing on training stability more directly than many prior surveys.

1. Introduction

  • GAN training can fail through mode collapse, non-convergence, or vanishing and exploding gradients, despite strong generation results across several domains.These failures can reduce sample diversity, produce oscillatory or diverging training, or slow and stop learning.Source: §1 · p. 1
  • Existing stabilization methods have not fully solved instability, which typically leaves substantial manual parameter tuning and impedes off-the-shelf use.Proposed approaches include changes to architectures, loss functions, and training strategies.Source: §1 · pp. 1–2
  • The survey describes stability problems, categorizes methods into five groups, compares them, and highlights open problems; many methods may be complementary.The categories are presented separately, though the survey notes that methods can potentially be combined.Source: §1 · p. 2

1.1 Related Surveys

  • Unlike many broad GAN surveys that omit detailed training procedures and instability, this review centers on stability while situating it within the wider research landscape.Related work also includes studies focused on training issues and evaluations of selected losses, architectures, normalization, and regularization schemes.Source: §1.1 · p. 2

2. Generative Adversarial Networks

GANs train a generator and discriminator through a competitive objective, but practical training can struggle to reach equilibrium. The section traces instability to difficult optimization, weak generator feedback, and mode collapse, with evaluation itself also unresolved.

2. Generative Adversarial Networks

  • GANs cast generation as a competition between a generator that makes samples resemble the target distribution and a discriminator that separates generated from real samples.Goodfellow et al. (2014) proved a unique equilibrium, but practical GANs have struggled to reach it (Fedus et al., 2018; Salimans et al., 2016).Source: §2 · p. 3

2.1.1 Convergence

  • Neural-network parameterization and the game's non-convex-concave character make convergence difficult, with oscillations, cycles, local equilibria, and costly discriminator optimization.A local equilibrium can be arbitrarily far from the global one.Source: §2.1.1 · p. 4

2.1.2 Vanishing or Exploding Gradients

  • An overly accurate discriminator can drive generator gradients close to zero; non-saturating loss alleviates vanishing gradients but does not eliminate instability.This occurs when real and generated distributions have supports on low-dimensional manifolds that permit near-perfect discrimination.Source: §2.1.2 · p. 4

2.1.3 Mode Collapse

  • Mode collapse maps distinct inputs to the same output, reducing sample diversity, while addressing it involves a trade-off with gradient problems and requires discriminator generalization.The cited discussion identifies discriminator generalization as crucial for addressing mode collapse.Source: §2.1.3 · pp. 4–5
  • Instability problems are linked and often involve trade-offs; no single evaluation metric, including Inception or FID, provides a comprehensive solution.The section identifies accurate estimation of discriminator generalization as a challenge for evaluating the generator.Source: §2.1.3 · p. 5

3. GAN Variants and Stabilization Techniques

The survey groups GAN stabilization methods into five approaches and reviews architecture changes as one family. Architectural variants can improve stability, but their empirical gains vary and may carry computational or theoretical costs.

3. GAN Variants and Stabilization Techniques

  • The survey organizes stabilization methods into architecture, loss-function, game-theory, multi-agent, and gradient-based approaches, comparing their concepts and trade-offs.Figure 1 presents a taxonomy of GAN methods.Source: §3 · p. 5, Figure 1 · p. 6

3.1 Modified Architectures

  • Architecture variants change networks or training structure while keeping the loss largely intact, and many target specific applications.The survey focuses on general approaches reported to improve GAN training stability.Source: §3.1 · p. 5

3.1.1 Convolutional Architectures

  • DCGAN uses convolutional operations and generator upsampling; SAGAN adds attention to capture local and global dependencies, and BigGAN builds on SAGAN for ImageNet results.BigGAN also increases batch size and model parameters.Source: §3.1.1 · p. 5

3.1.2 Hierarchical Architectures

  • Progressive GAN expands resolution and network structure in phases, while Stacked GAN and GoGAN train multiple pairs or stages with additional losses or margin-based Wasserstein distance.Progressive GAN also uses pixel normalization and mini-batch standard deviation; Stacked GAN combines adversarial, conditional, and entropy losses.Source: §3.1.2 · p. 7

3.1.3 Autoencoder Architectures

  • Autoencoder discriminators score samples by reconstruction energy, aiming to avoid greedy discriminator behavior and provide generator feedback on important image features.The methods described include ignoring high-error outliers, relaxing a fixed margin in MAGAN, and BEGAN's balance hyperparameter γ.Source: §3.1.3 · p. 7

3.1.4 Latent Space Decomposition Architectures

  • Latent-space methods address discriminator advantage from disjoint low-dimensional supports by adding feature information, either unsupervised through InfoGAN or supervised through CGAN labels.The CGAN approach is restricted by the need for a labeled dataset.Source: §3.1.4 · p. 8

3.1.5 Summary

  • Architecture variants improve GAN training stability, but BigGAN's gains incur high computational cost, and the non-convex-concave game remains a challenge beyond architectural changes.Improving training without increasing complexity remains an open challenge in the survey's assessment.Source: §3.1.5 · p. 8

3.2 Modified Loss Functions

Standard GAN losses can suffer non-convergence and vanishing gradients when generated and target distributions are disjoint; the survey reviews alternative divergences, IPM losses, and auxiliary terms. It compares their benefits and constraints, concluding that loss variants have not resolved training instability.

3.2.1 f -Divergence

  • The f-divergence family includes f-GAN, which estimates divergences for a chosen function f, and LSGAN, which uses least-squares loss to penalize generated samples far from the decision boundary.LSGAN replaces the standard GAN’s sigmoid cross-entropy loss and is associated with Pearson X^2 divergence.Source: §3.2.1 · p. 9
  • Estimating f-divergences becomes harder as data dimension grows, and low-dimensional distribution supports can make the divergence infinite.The survey identifies these as limitations of the f-divergence family.Source: §3.2.1 · p. 9

3.2.2 Integral Probability Metric

  • IPM losses measure maximal distribution distance with a scalar-valued critic, and the survey presents Wasserstein distance as one such loss used by WGAN.The IPM critic is framed as a regression task rather than a binary classifier.Source: §3.2.2 · p. 9
  • Wasserstein distance can improve stability and convergence on low-dimensional supports, but clipping can cause slow convergence and unstable training; WGAN-GP substitutes a gradient penalty.The survey also notes a proposed energy-function remedy for WGAN’s biased gradients.Source: §3.2.2 · p. 10
  • Other IPM approaches trade among representation, stability, and cost: MMDGAN uses adversarial kernel learning but scales exponentially with sample count, while RGAN compares realness probabilities and Geometric GAN requires expensive matrix computation.Loss Sensitive GAN uses margins, and Fisher GAN adds a data-dependent constraint while avoiding weight clipping or gradient penalties.Source: §3.2.2 · p. 10

3.2.3 Auxiliary Loss Functions

  • Auxiliary loss methods include unrolling possible discriminator reactions into the generator’s gradient and MRGAN’s encoder-based latent vectors, which aim to prevent mode collapse.MRGAN adds a loss term intended to ensure different modes are captured in the latent vector.Source: §3.2.3 · p. 11

3.2.4 Summary

  • Loss-function variants can predominantly overcome vanishing gradients and offer more generalization than architecture variants, but restrictions and hyperparameter tuning leave stabilization challenging.The survey notes that architectural approaches with proper hyperparameter tuning can yield superior results.Source: §3.2.4 · p. 11

3.3 Game Theory for GANs

Game-theoretic GAN methods use mixed-strategy Nash equilibria to address convergence, but the proposed methods remain restrictive and rarely directly applicable.

3.3 Game Theory for GANs

  • Game-theoretic GAN research centers on mixed-strategy Nash equilibria, which under certain circumstances can equal a global Nash equilibrium, unlike the local guarantee described for pure strategies.The survey describes this as a contrast with standard GANs’ pure-strategy Nash equilibrium.Source: §3.3 · p. 11
  • Proposed equilibrium-finding methods include regret minimization, Parallel Nash Memory, Mirror Descent, and mixtures of historical models.ChekhovGAN’s convergence guarantee requires a single-layer discriminator; Mirror Descent is described as less restrictive than the earlier finite-game formulation.Source: §3.3 · p. 12

3.3.1 Summary

  • The literature remains limited, and its highly restrictive methods are rarely directly applicable, though theoretical work opens avenues for further research.Source: §3.3.1 · p. 12

3.4 Multi-Agent GANs

Multi-agent GANs expand the generator-discriminator pair, often to hinder mode collapse, but computational cost and choosing the number of agents remain unresolved concerns.

3.4 Multi-Agent GANs

  • Multi-agent GANs use multiple generators or discriminators to improve convergence properties and prevent mode collapse; MAD-GAN and MGAN encourage diverse outputs through generator-specific feedback or classification.Stackelberg GAN instead makes multiple generators followers to a discriminator leader.Source: §3.4 · p. 12
  • Other approaches combine multiple generator-discriminator pairs or mixed strategies, including reward averaging and dynamically exchanged discriminators.The survey also notes multi-agent communication as a way to regularize both networks.Source: §3.4 · pp. 12–13

3.4.1 Summary

  • Multi-agent methods often incur significant computational costs, and the appropriate number of generators or discriminators remains unclear.The surveyed literature mainly uses added agents to hinder mode collapse.Source: §3.4.1 · p. 13

3.5 Modified Gradient Optimization

Modified gradient optimization explores alternatives to gradient descent ascent for GANs as games with interacting losses. Initial results report better convergence and less mode collapse than GDA, while leaving the field open to further study.

3.5 Modified Gradient Optimization

  • Proposed alternatives include Optimistic Mirror Descent, ConOpt, decompositions into potential and Hamiltonian games, and Competitive Gradient Descent for two-player competitive games.The methods have differing conditions, including a bilinear zero-sum restriction for Optimistic Mirror Descent.Source: §3.5 · p. 13

3.5.1 Summary

  • Gradient-based GAN variants initially improve convergence and prevent mode collapse compared with GDA.The survey cautions that suitable hyperparameter tuning might also achieve these results.Source: §3.5.1 · p. 13
  • Gradient variants may offer an additional edge in general games and can be used alongside other stabilization methods, but remain largely unexplored.Source: §3.5.1 · pp. 13–14

3.6 Other Modifications

This section surveys methods outside the preceding categories, including population-based adaptation, boosting, energy-based potential fields, and Bayesian weight marginalization.

3.6 Other Modifications

  • Evolutionary GAN adapts to its environment through operations on populations of generators and discriminators.(Wang et al.)Source: §3.6 · p. 14
  • AdaGAN uses boosting to iteratively address missing modes.(Tolstikhin et al., 2017)Source: §3.6 · p. 14
  • Unterthiner et al. (2018) proposed a proof and method using energy-based potential fields to escape local equilibria.Source: §3.6 · p. 14
  • Bayesian GAN marginalizes generator and discriminator weights as another technique for preventing mode collapse.Source: §3.6 · p. 14

3.7 Comparative Summary

The comparison finds architecture variants show the highest empirical improvement, while emphasizing tuning, limited generality, and unresolved alternatives across GAN variants.

3.7 Comparative Summary

  • Architecture variants show the highest empirical improvement, though their relative publication volume, low entry barrier, and meticulous parameter optimization may contribute to that result.Much architecture research lacks rigorous theoretical justification and usually improves only one or two training problems (Wang et al., 2019).Source: §3.7 · p. 14
  • A comprehensive comparison found only a small improvement in general metrics for state-of-the-art loss-type GANs over architectures such as DCGAN.The survey notes WGAN results could also be attained through appropriate hyperparameter optimization of architecture variants.Source: §3.7 · p. 14
  • Loss-function variants may improve training more generally, but significant breakthroughs may require alternatives to gradient descent ascent; gradient-based variants remain too early for reliable predictions.Game-theory variants may address the non-convex-concave game but current methods have strict limitations and cannot compete with architecture- and loss-function variants.Source: §3.7 · p. 14
  • Multi-agent GANs have mostly targeted mode collapse, with promising but not state-of-the-art results, while broader multi-agent learning remains unexplored.The area has only a handful of published studies.Source: §3.7 · p. 15

4. General Stabilization Heuristics

The survey reviews general heuristics that can potentially stabilize different GAN types, spanning regularization, normalization, and architectural or training procedures.

4. General Stabilization Heuristics

  • General stabilization heuristics can potentially apply across GAN types, and studies report effects of regularization and normalization across loss-function and architecture variants.The survey notes that hyperparameter optimization can yield superior results to applying specific methods such as modified architectures and loss functions (Lucic et al., 2018).Source: §4 · p. 15

4.1 Regularization

  • Gradient penalties constrain discriminator objectives or gradients, with variants targeting 1-Lipschitzness, data-manifold constraints, or non-overlapping distribution support.WGAN-GP uses interpolation between real and generated distributions to calculate the gradient norm; the regret-minimization variant limits penalty calculation to around a data manifold.Source: §4.1 · p. 15

4.2 Normalization

  • Discriminator normalization can stabilize optimization by improving feedback and enrich layer representations; spectral normalization constrains layer spectral norms to impose a Lipschitz condition.Batch and layer normalization are optimization-oriented approaches, while spectral normalization is described as the most effective representation-oriented method proposed up to that point.Source: §4.2 · pp. 15–16

4.3 Architectural Features and Training Procedures

  • Feature matching trains the generator against an intermediate discriminator layer so generated data also matches the target distribution, rather than only seeking real classifications.It was proposed to prevent generator overtraining on the discriminator (Salimans et al., 2016).Source: §4.3 · p. 16
  • Minibatch discrimination penalizes similarity among generated samples using cross-sample distances, while historical averaging adds past parameter values to the player's cost.Minibatch discrimination supplies the computed data as side information while the discriminator still outputs one probability; historical averaging is designed to avoid oscillating and cyclical behavior.Source: §4.3 · p. 16
  • One-sided label smoothing restricts discriminator output probabilities to 0.1–0.9 instead of 0–1, alleviating vanishing gradients by limiting discriminator confidence.(Salimans et al., 2016)Source: §4.3 · p. 16
  • Noise injection smooths probability mass and reduces non-overlapping support, particularly for standard GANs, but can lower generated-image quality.It injects continuous noise into discriminator inputs and is described as computationally efficient and practical.Source: §4.3 · pp. 16–17

5. Open Problems

The survey identifies open questions about combining stabilization approaches and drawing on related fields, including reinforcement learning, opponent modeling, and VAEs.

5.1 Combining Training Stabilization Methods

  • How stabilization techniques align and how combinations affect performance remain unclear, motivating study of joint methods.BigGAN achieved state-of-the-art results by combining known methods and scaling them; game-theory and gradient-based approaches are often omitted from architecture- and loss-focused GAN literature.Source: §5.1 · p. 17

5.2 Actor-Critic Methods for GAN Training

  • Replay-buffer ideas from actor-critic methods could inspire ways to identify and improve networks on samples with large real-versus-generated discrepancies.The passage describes prioritized experience replay as a conceptual inspiration for this possible method.Source: §5.2 · p. 17

5.3 Discriminator Generalization

  • A GAN objective that fully incorporates discriminator generalization remains a challenge, since discriminator loss alone cannot assess feedback quality or generalization to unseen samples.A discriminator may distinguish real from generated images using only a small set of features without proper generalization.Source: §5.3 · pp. 17–18

5.4 Opponent Modeling in GANs

  • Modeling GANs as agents with limited information and accounting for awareness of the other agent could potentially lead to more stable convergence.Few studies have explored the connection between GAN games and opponent modeling or broader multi-agent learning (Sch¨afer et al., 2019).Source: §5.4 · p. 18

5.5 Variational Autoencoder GAN Hybrid

  • VAE-GAN hybrids combine VAEs and GANs to alleviate mode collapse, and this relatively unexplored intersection is described as promising for preventing it.VAEs have stronger theoretical grounding and are less prone to mode collapse than GANs, though they are known to produce lower-quality images.Source: §5.5 · p. 18

6. Conclusion

The survey categorizes GAN stabilization techniques and offers a comparative summary and research directions. It concludes that most methods address only one training issue and often lack rigorous theoretical justification.

6. Conclusion

  • Most surveyed methods prevent only one GAN training issue and often lack rigorous theoretical justification.The conclusion frames these as limitations across the majority of studies.Source: §6 · p. 18
  • The article categorizes current approaches to stabilizing GAN training and outlines their techniques and key concepts.Source: §6 · p. 18
  • The survey provides a comparative summary of methods and describes potential research directions for addressing open problems.Source: §6 · p. 18
Loading 1910.00927v2…