PaperlayerView exact revision on arXiv

Source-linked AI summary

How Generative Adversarial Networks and Their Variants Work: An Overview

Yongjun Hong, Uiwon Hwang, Jaeyoon Yoo, Sungroh Yoon

arXiv:1711.05914v10cs.LG

TL;DR

GANs address the difficulty of modeling complex, high-dimensional data without explicitly assuming a probability distribution. This survey explains adversarial training, objectives, variants, autoencoder combinations, and applications, while noting unresolved issues in convergence and mode collapse.

  • Problem

    Explicit probability-density models require distributional assumptions that may fail to represent complex, high-dimensional real data.

  • Method

    The paper surveys GANs through their generator–discriminator mechanism, objective functions, theoretical behavior, autoencoder combinations, variants, and applications.

  • Results

    GANs avoid explicitly defining pθ(x), generate samples through a simple latent-to-data mapping, and support parallel generation.

  • Takeaways & Limitations

    GANs provide a flexible framework for generating real-like samples and have been applied to image synthesis, translation, domain adaptation, and sequential data.

  • Takeaways & Limitations

    GAN training remains unstable and difficult to converge, with unresolved questions about convergence and whether mode collapse can be completely overcome.

Abstract

from arXiv · show

Generative Adversarial Networks (GAN) have received wide attention in the machine learning field for their potential to learn high-dimensional, complex real data distribution. Specifically, they do not rely on any assumptions about the distribution and can generate real-like samples from latent space in a simple manner. This powerful property leads GAN to be applied to various applications such as image synthesis, image attribute editing, image translation, domain adaptation and other academic fields. In this paper, we aim to discuss the details of GAN for those readers who are familiar with, but do not comprehend GAN deeply or who wish to view GAN from various perspectives. In addition, we explain how GAN operates and the fundamental meaning of various objective functions that have been suggested recently. We then focus on how the GAN can be combined with an autoencoder framework. Finally, we enumerate the GAN variants that are applied to various tasks and other fields for those who are interested in exploiting GAN for their research.

1 Introduction

Generative models learn complex data distributions, but explicit likelihood models require distributional assumptions that may limit high-dimensional modeling. GANs address this through adversarial generator–discriminator learning, enabling simple sampling and broad applications; this paper surveys their objectives, architectures, variants, and uses.

  • Generative models represent complex, high-dimensional data across images, videos, music, language, medical imaging, and security.
  • Maximum-likelihood models optimize the probability assigned to training data but require an explicit form for pθ(x).
  • GANs replace explicit likelihood maximization with adversarial learning between a generator that maps latent variables to data and a discriminator that distinguishes real from generated samples.
  • GANs avoid explicitly defining pθ(x), sample through a simple feed-forward mapping, and support parallel generation unlike Markov-chain and autoregressive alternatives.
  • The paper explains GAN theory, objective functions, autoencoder combinations, variants, applications, and challenges such as mode collapse.

2 Generative Adversarial Networks

A GAN is a minimax game in which the generator produces samples from latent variables and the discriminator classifies them against real data. The survey presents the objective, its theoretical equilibrium, related interpretations, and practical shortcomings motivating GAN variants.

  • The generator maps latent variables into data-space samples, while the discriminator distinguishes generated samples from real data.
  • The standard objective is binary cross entropy, with the discriminator maximizing correct real/fake classification and the generator minimizing the same minimax value to deceive it.
  • Under sufficient model capacity, equilibrium occurs when the generated distribution matches the real distribution, and the optimal discriminator yields a Jensen–Shannon divergence objective.
  • The discriminator can estimate a density ratio, allowing GAN training to address likelihood intractability through relative behavior between real and generated distributions.
  • GANs face practical shortcomings, including unstable training, difficult convergence, and mode collapse, motivating changes to objectives and architectures.

2.1 Object Functions

Generative-model training seeks to match the real data distribution, and GAN objectives can replace the standard Jensen–Shannon divergence with other distributional distances or divergences.

  • Generative models train by minimizing differences between the real distribution pdata(x) and the generated distribution pg(x).
  • Standard GAN minimizes Jensen–Shannon divergence estimated through the discriminator, while alternative distances or divergences may improve GAN performance.

2.1.1 f-divergence

The f-divergence framework measures distribution differences through convex functions and generalizes GAN objectives via tractable variational lower bounds. Its variants include standard divergences and least-square losses that additionally encourage generated samples toward real data.

  • f-divergence: The f-divergence D_f(p_data||p_g) measures differences between distributions using a convex function f satisfying f(1) = 0.When the distributions are equivalent, their ratio is 1 and the divergence is 0.
  • f-GAN: f-GAN estimates f-divergences through expectations of a variational lower bound using a parameterized discriminator.It maximizes the lower bound with respect to the discriminator and minimizes the resulting divergence with respect to the generator.
  • f-GAN: The f-GAN framework derives KLD, reverse KLD, JSD, and other divergences by selecting the corresponding convex generator function.This unifies multiple GAN objectives under one f-divergence formulation.
  • Least-square GAN: LSGAN replaces sigmoid cross-entropy with least-square loss, directly penalizing generated samples for remaining far from the real data distribution.Its discriminator targets designated values for real and generated samples rather than probabilities, while retaining an f-divergence interpretation.

2.1.2 Integral probability metric

Integral probability metrics measure distribution discrepancies by maximizing differences over a chosen critic function class. GAN variants such as WGAN, MMD-based GANs, and Fisher GAN select different constraints or representations to obtain useful distance properties and training behavior.

  • Integral probability metric: An integral probability metric defines the maximum discrepancy between p_data and p_g over a specified class of measurable, bounded critic functions.The function class determines the resulting distance and its properties.
  • Wasserstein GAN: WGAN uses the Earth-mover distance and trains a 1-Lipschitz critic to estimate the distance before optimizing the generator.Weight clipping is used to keep the critic parameter space compact, although it can restrict the critic to a limited subset of Lipschitz functions.
  • Wasserstein GAN: The Earth-mover distance is continuous and differentiable almost everywhere under the Lipschitz condition, making it suitable for distributions supported on low-dimensional manifolds.The paper describes it as more tolerant for convergence than KLD and total variation distance.
  • Maximum mean discrepancy: MMD matches kernel embedding means and can represent diverse feature spaces through a kernel, including infinite-order moments with a Gaussian kernel.Its computational cost grows quadratically with the number of samples.
  • Fisher GAN: Fisher GAN maximizes mean discrepancy under a data-dependent hyperellipsoid constraint, avoiding weight clipping and reducing computation relative to gradient-penalty training.It computes covariances instead of critic gradients for the constraint.
  • Comparison to f-divergence: IPM estimators are more consistently convergent than f-divergence estimators and do not diverge when distribution supports are disjoint.The paper attributes these properties to the IPM family’s behavior across data dimensions and finite i.i.d. samples.

2.1.3 Auxiliary object functions

GAN objectives can be combined with auxiliary losses to stabilize training, preserve image content, or exploit labeled data. Reconstruction and classification objectives assign these additional roles to the generator, discriminator, or classifier.

  • Auxiliary objectives: Adversarial objectives estimate a distance between p_g and p_data, while the generator reduces that estimated distance during minimax training.Additional objective functions can support training stability and related goals.
  • Reconstruction: Reconstruction loss encourages the generator to preserve the contents of the original input image, usually through an L1 difference.This is important for image translation and autoencoder reconstruction when semantic content and modes should remain intact.
  • Reconstruction: Some GAN variants apply reconstruction objectives to the discriminator by adopting an autoencoder architecture and interpreting the discriminator as an energy function.These variants are described as extensions of the discriminator design.
  • Classification: Cross-entropy loss is added when labeled data exists, especially for semi-supervised learning and domain adaptation.It can give the discriminator a classification role or train a separate classifier jointly with the generator and discriminator.

2.2 Architecture

GAN architectures use generator–discriminator designs, stacking, progressive growth, or autoencoder-based energy functions to improve sample quality, resolution, and training behavior. These variants divide complex generation into stages or reshape the discriminator’s role.

  • DCGAN stabilizes GAN training through a transposed-CNN generator, CNN discriminator, batch normalization, and selected activation functions.
  • StackedGAN uses multiple generator–discriminator pairs and encoders to learn hierarchical, level-specific representations adversarially.
  • Hierarchical stacks divide a complex generator mapping step by step, making high-resolution image generation easier despite discriminator and memory challenges.
  • Progressive GAN starts at 4×4 resolution and incrementally doubles image resolution by adding generator and discriminator layers.Pixel normalization, equalized learning rate, and mini-batch standard deviation further stabilize training.
  • Autoencoder-based GANs treat the discriminator as an energy function, assigning low energy near the data manifold and high energy to contrastive samples.Pixelwise reconstruction loss provides the discriminator’s energy, while BEGAN balances generator and discriminator training with γ = E[L(G(z))]/E[L(x)].
  • EBGAN and MAGAN use margin-based energy objectives, while BEGAN matches reconstruction-loss distributions and trades image diversity against artifacts as γ changes.Higher γ values are associated with increased diversity but some artifacts in the illustrated BEGAN samples.

2.3 Obstacles in Training GAN

GAN training faces theoretical and practical obstacles because distribution supports may be low-dimensional, neural-network optimization lacks the theory’s convexity assumptions, and alternating updates can behave like maximin optimization. These issues contribute to vanishing gradients, nonconvergence, mode collapse, and oscillations, motivating stabilization techniques.

  • Theoretical issues: The standard GAN objective relates the optimal discriminator to JSD, while asymmetric KLD objectives behave differently for mode coverage and sharpness.JSD accounts for both mode dropping and sharpness and does not diverge when generated support excludes a point.
  • Theoretical issues: When real and generated distributions lie on disjoint or low-dimensional manifolds, a perfect discriminator can produce vanishing gradients for the generator.This removes useful information from discriminator gradients as real and fake samples become perfectly separable.
  • Practical issues: Deep neural networks create a gap between convergence theory and practice because optimization occurs in parameter space rather than the theoretically analyzed function space.The convexity required by the theoretical convergence proof does not hold for the neural-network parameterization.
  • Practical issues: Alternating discriminator and generator updates can behave like a maximin problem, causing the generator to concentrate on one mode instead of covering multimodal real data.The discriminator is updated for only a finite number of steps, and minimax and maximin solutions are not generally equal.
  • Practical issues: GAN training can oscillate and require excessive time because generator and discriminator optimize the same objective in opposite directions within a non-convex minimax game.Finding a Nash equilibrium with high-dimensional parameters is difficult under gradient descent.
  • Training techniques to improve GAN training: Spectral normalization constrains each discriminator layer’s spectral norm to promote Lipschitz continuity and is computationally beneficial relative to gradient-penalty regularization.Label smoothing instead softens real-data labels; PatchGAN produces grid outputs over local patches to help generate sharper results.

2.4 Methods to Address Mode Collapse in GAN

Mode collapse prevents GANs from representing the diversity of real data, so the paper surveys objective-function, architectural, and mini-batch methods to address it.

  • Problem: Mode collapse occurs when a GAN represents only one or a few modes of a multimodal real-data distribution.The generator may switch between modes during training rather than converge to a distribution covering them all.
  • Overview: The paper organizes mode-collapse remedies into new objectives, architecture modifications, and mini-batch discrimination.Mini-batch discrimination is described as a notably practical technique because it exposes diversity directly to the discriminator.
  • Objective-function methods: Unrolled GAN uses a surrogate generator objective that unrolls k discriminator updates to anticipate the discriminator’s response.This differs from standard GAN training, which updates the generator using the discriminator after its immediate update.
  • Objective-function methods: DRAGAN constrains discriminator gradients around the real-data manifold by penalizing deviations from gradient norm 1.The method attributes mode collapse to a spurious local Nash equilibrium in the nonconvex problem.
  • Objective-function methods: EBGAN adds a repelling regularizer that encourages orthogonal feature vectors within a mini-batch, discouraging generation concentrated in a few modes.The regularizer uses cosine similarity at an encoder representation level.
  • Architectural and mini-batch methods: MAD-GAN uses multiple generators with one discriminator and an additional cosine-similarity objective to move generators toward different modes.MRGAN instead adds an encoder to match generated and real data manifolds, while mini-batch discrimination lets the discriminator assess sample diversity jointly.

3 Treating the Latent Space

GAN latent-space methods make data attributes more tractable to manipulate by structuring or learning compressed representations, including through conditions, semantic codes, and autoencoders.

  • Motivation: Latent-space manipulation is more tractable than direct image-space editing because image manifolds are high-dimensional and complex.The latent space contains compressed representations that can support attribute changes such as pose, age, and expression.
  • Latent-space decomposition: A standard GAN latent vector z is highly entangled, motivating decomposition into meaningful information c and a standard latent vector z.The paper distinguishes supervised and unsupervised approaches to this decomposition.
  • Supervised methods: CGAN supplies a known condition such as a class label to both generator and discriminator, enabling controlled generation.AC-GAN additionally makes the discriminator predict class labels alongside real-versus-fake probabilities.
  • Unsupervised methods: InfoGAN samples an unknown latent code c and maximizes mutual information I(c; G(z, c)) so c captures salient semantic features.Unlike CGAN, InfoGAN does not require semantically labeled conditions during training.
  • Applications of latent decomposition: StackGAN generates text-conditioned images in two stages: low-level feature generation followed by detail painting.Other variants decompose latent variables for facial identity and pose or use attribute vectors for scenery generation.
  • Autoencoder frameworks: Autoencoder-GAN frameworks combine an encoder that maps x to z with a decoder that reconstructs x, helping learn latent representations and stabilize GAN training.The paper links this structure to reduced mode collapse and easier interpolation or conditional concatenation in latent space.
  • Autoencoder frameworks: ALI and BiGAN learn the joint distribution of data x and latent z using an encoder, while AGE performs adversarial learning between the generator and encoder in latent space.VAE instead uses an encoder-decoder mapping and a variational lower bound to address marginal-likelihood intractability.

4 Applications Using GANs

GANs are applied across image translation, enhancement, object manipulation, multidomain synthesis, and video generation because they can generate real-like samples without an explicit data-distribution form.

  • Overview: GAN applications span image translation, super resolution, object manipulation, image blending, multidomain synthesis, and video generation.The paper presents these as applications enabled by GAN’s ability to generate real-like samples without explicitly specifying the real data distribution.
  • Image translation: Pix2pix performs supervised image translation by conditioning a CGAN generator on an input image paired with its target image.The example converts Cityscapes labels into real photographs and compares them with ground truth.
  • Image translation: CycleGAN and DiscoGAN perform unpaired translation by adding cyclic consistency to adversarial loss, reducing meaningless mappings and mode collapse.Their setting uses separate image domains without paired examples.
  • Image enhancement: Super-resolution GAN methods add perceptual feature losses to adversarial losses because pixelwise mean-squared error can produce smoothed, perceptually poor details.The perceptual loss compares feature differences from an intermediate discriminator layer rather than pixels alone.
  • Object manipulation and blending: GANs support object-focused tasks including small-object super-resolution, occluded-object completion, object transfiguration, and realistic image blending.These methods respectively enhance discriminability, reconstruct invisible regions, separate object and background features, or combine GAN outputs with gradient constraints.
  • Multidomain synthesis: Coupled GAN jointly generates images from multiple domains by sharing high-level semantic weights between GAN pairs.The method seeks to learn a joint multidomain distribution from samples drawn from marginal domain distributions.
  • Video generation: Video GAN uses separate generators for moving foreground and static background, while Pose-GAN models content and motion components with VAE and recurrent structures.The motion component is modeled with an RNN to capture temporal dependency.

4.2 Sequential Data Generation

Sequential GAN generation must handle discrete outputs and evaluate partially generated sequences, so variants use reinforcement-learning policy gradients or ranking objectives, alongside applications to semi-supervised learning.

  • Sequential-generation challenges: Discrete-value GANs cannot directly back-propagate through noncontinuous outputs, motivating policy-gradient reinforcement learning methods.For sequences, conventional GANs evaluate whole sequences rather than partially generated prefixes.
  • Music generation: C-RNN-GAN models both generator and discriminator with LSTMs but evaluates complete music sequences rather than partial sequences.SeqGAN generates sequences through policy-gradient updates instead of producing the whole sequence at once.
  • Policy-gradient methods: SeqGAN and related methods treat the generator as an agent policy and the discriminator’s output as a reward.Policy gradients allow evaluation during sequential generation rather than only after an entire sequence is produced.
  • Language generation: RankGAN replaces the conventional discriminator with a ranker that compares generated sentences with human-written reference sentences.The generator seeks a high relative rank, incorporating language expressiveness alongside authenticity.
  • Voice conversion: VAW-GAN combines VAE and WGAN frameworks for voice conversion, encoding phonetic content and decoding a target-speaker voice.It uses WGAN to address sharpness limitations associated with VAE’s Gaussian assumption.
  • Semi-supervised learning: GAN-based semi-supervised learning assigns generated data to a K + 1 class while using labeled real data for classes 1 through K.Unlabeled and generated data inform the space where real data resides and guide the supervised objective’s optimum.
  • Semi-supervised learning: Triple-GAN addresses incompatible discrimination and classification convergence points, plus class-specific generation, through a three-player generator-discriminator-classifier formulation.Its formulation uses generated, labeled, and unlabeled data with predicted labels.

4.4 Domain Adaptation

Domain adaptation uses GAN-based adversarial objectives to reduce source–target domain shift while preserving task-relevant content. The surveyed approaches align feature spaces or transform source images, extending GAN applications to simulation-to-real transfer, navigation, and other domains.

  • Domain adaptation: Domain adaptation addresses domain shift by adapting source-domain data to a target domain while preserving classification performance.
  • Feature-space alignment: DANN learns domain-invariant features that remain discriminative for classification by combining a classifier and a domain discriminator over a shared feature extractor.
  • Feature-space alignment: CyCADA adds cycle consistency to DANN-style adaptation to preserve content whose characteristics are important for determining labels.
  • Image-level adaptation: Simulation-to-real grasping adapts source images toward the target domain and preserves source content, enabling better performance than the supervised method.Unlike feature-space methods, transformed images can be inspected during training.
  • Image-level adaptation: Cycle consistency and style loss reduced outdoor domain shift enough for autonomous navigation in a real environment without real labeled data.
  • Other applications: GAN variants extend adversarial learning beyond domain adaptation to medical segmentation, steganography, and continual learning.The surveyed examples use segmentor–critic structures, multiple discriminators, or generative replay to address task-specific objectives.

5 Discussion

The discussion reviews how GANs are evaluated, applied to discrete data and related learning frameworks, and weighed against their practical advantages and optimization difficulties. GANs offer flexible, fast, sharp generation, but evaluation, convergence, discrete outputs, and mode collapse remain unresolved concerns.

  • Evaluation: GAN evaluation seeks to capture both generated-data quality and diversity, commonly using the Inception score rather than likelihood.
  • Evaluation: The Inception score can miss mode collapse because a generator producing one plausible sample per class may still receive a high score.MS-SSIM and an independently trained Wasserstein critic are discussed as ways to detect collapse or overfitting more reliably.
  • Discrete structured data: GANs struggle with discrete outputs because non-differentiability prevents direct back-propagation through discrete sequences or images.Policy-gradient methods and latent-space autoencoder approaches are presented as alternatives.
  • Advantages: GAN avoids explicit likelihood assumptions and maps latent variables to data with a feed-forward generator, enabling parallel sampling.
  • Advantages: Adversarial training can capture high-frequency image details, contributing to sharper generated results.PatchGAN is cited as a technique that helps produce and capture sharper results.
  • Limitations: GAN training is highly unstable because iterative minimax updates can improve one player’s objective while worsening the other’s, preventing convergence.

6 Conclusion

The paper surveys GAN objectives, architectures, variants, theoretical foundations, and applications across image and other domains. It concludes that GANs offer substantial opportunities while retaining unresolved theoretical challenges, including convergence and mode collapse.

  • The paper examines how GAN objective functions and architectures influence behavior across applications including image translation, attribute editing, and domain adaptation.
  • GAN research has produced many variants through practical and mathematical approaches extending the standard minimax formulation.
  • GANs have been connected with imitation learning and other generative models, producing broader theoretical and application techniques.
  • GANs remain promising for applications because deep neural networks can learn highly nonlinear mappings from latent space into data space.
  • Convergence and whether GANs can completely overcome mode collapse remain unsolved theoretical problems.
Loading 1711.05914v10…