PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Adversarial Networks: An Overview

Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, Anil A Bharath

arXiv:1710.07035v1cs.CV

TL;DR

GANs address the challenge of learning deep representations from largely unannotated data. This review synthesizes GAN architectures, training methods, and applications, finding useful representations and effective results across tasks while identifying unresolved theoretical and training challenges.

  • Problem

    Deep representation learning remains limited by the availability of extensively annotated training data, despite vast quantities of unlabelled image data.

  • Method

    The paper reviews GAN architectures, training strategies, and applications using familiar signal-processing analogies and concepts.

  • Results

    GAN-learned representations and adversarial methods achieve good or state-of-the-art results across classification, image translation, and pose and gaze estimation tasks.

  • Takeaways & Limitations

    GANs provide a framework for learning nonlinear representations and supporting applications including synthesis, translation, super-resolution, and classification.

  • Takeaways & Limitations

    GAN training faces unresolved equilibrium and convergence problems, including cases where apparent convergence leaves the learned distribution far from the target.

Abstract

from arXiv · show

Generative adversarial networks (GANs) provide a way to learn deep representations without extensively annotated training data. They achieve this through deriving backpropagation signals through a competitive process involving a pair of networks. The representations that can be learned by GANs may be used in a variety of applications, including image synthesis, semantic image editing, style transfer, image super-resolution and classification. The aim of this review paper is to provide an overview of GANs for the signal processing community, drawing on familiar analogies and concepts where possible. In addition to identifying different methods for training and constructing GANs, we also point to remaining challenges in their theory and application.

I. INTRODUCTION · II. PRELIMINARIES · A. Terminology

GANs learn high-dimensional data distributions by training differentiable generator and discriminator networks in competition, with the discriminator’s feedback improving generated samples. Their learned representations support synthesis and downstream tasks including editing, transfer, classification, retrieval, and augmentation.

  • I. INTRODUCTION: GANs are semi-supervised and unsupervised learning methods that implicitly model high-dimensional data distributions through competition between two networks.Proposed in 2014, GANs commonly frame the generator as an art forger and the discriminator as an art expert.
  • I. INTRODUCTION: The generator creates synthetic samples without direct access to real images and learns only through interaction with the discriminator.The discriminator compares generated samples with real samples using ground-truth real-versus-fake labels.
  • I. INTRODUCTION: The discriminator’s error signal is propagated to train the generator, leading it toward higher-quality forgeries.This feedback mechanism allows generator improvement despite the absence of direct real-image access.
  • I. INTRODUCTION: Generator and discriminator networks typically use convolutional and/or fully-connected layers and must be differentiable, but need not be directly invertible.The generator maps a latent representation space to image data space.
  • I. INTRODUCTION: In a basic GAN, the discriminator maps image data to a probability that an image belongs to the real data distribution rather than the generator distribution.For a fixed generator, it classifies training images as real, close to 1, and generated images as fake, close to 0.
  • I. INTRODUCTION: GAN representations can be used beyond the generator and discriminator themselves, supporting varied subsequent applications.The paper specifically motivates exploring these representations in later application-focused sections.
  • A. Terminology: Generative models synthesize novel samples from learned statistical distributions for semantic image editing, data augmentation, and style transfer.Their learned representations are also used for classification and image retrieval.
  • A. Terminology: Fully connected and convolutional layers generalize perceptrons and spatial filter banks with nonlinear postprocessing, with weights learned through backpropagation.These layer types are part of the deep-network terminology used throughout the paper.

B. Notation · C. Capturing Data Distributions · D. Related Work

The sections establish notation for GAN variables and coupled objectives, explain GANs as implicit approaches to capturing data distributions, and situate them among signal-processing representations and generative models. GANs differ from PCA, ICA, Fourier, and wavelet methods through highly complex nonlinear mappings from latent space to image space.

  • B. Notation: GAN notation uses bold lowercase symbols for multidimensional vectors, with pdata(x) denoting the density over a random vector x in R^|x|.This convention emphasizes the multidimensional nature of variables.
  • B. Notation: The generator and discriminator objectives, JG(ΘG; ΘD) and JD(ΘD; ΘG), are codependent because both networks’ parameter sets evolve iteratively.The notation highlights that each objective depends on the other network’s changing parameters.
  • C. Capturing Data Distributions: GANs are motivated by density estimation, which represents real-world data distributions and supports inference tasks such as categorization, detection, recognition, tracking, and registration.The data generating distribution refers to the underlying probability density or mass function of observations.
  • C. Capturing Data Distributions: Because likelihood functions for high-dimensional real-world images are difficult to construct, GANs avoid explicit density evaluation and can implicitly capture data distributions.This requires a generator-discriminator pair with suitable capacity.
  • D. Related Work: PCA uses covariance-matrix eigenvectors and a linear reconstruction mapping, making it shallow and limiting the complexity of representable data.Its basis functions emerge from the covariance matrix over input observations.
  • D. Related Work: ICA relaxes PCA’s orthogonality requirement by treating mixing coefficients as statistically independent, with formulations differing in objectives or generative models.ICA constructs data examples by blending signal components.
  • D. Related Work: GAN latent space is analogous to transform-space coefficient space, but deep nonlinear generator networks can map it to image space with extraordinarily high complexity.The mapping may use nonlinearities and almost arbitrary depth.

III. GAN ARCHITECTURES · A. Fully Connected GANs · B. Convolutional GANs

GAN architectures progressed from fully connected networks on simple image datasets to convolutional designs for more capable image synthesis. Convolutional approaches addressed training and spatial-sampling challenges through multiscale generation, DCGAN operators, and extensions to volumetric data.

  • A. Fully Connected GANs: Early GANs used fully connected generator and discriminator networks on MNIST, CIFAR-10, and the Toronto Face Dataset.These datasets included handwritten digits and natural images.
  • B. Convolutional GANs: CNNs were a natural extension for image data, but early CIFAR-10 experiments found them harder to train at supervised-learning capacity.The difficulty concerned training generator and discriminator networks with comparable capacity and representational power.
  • B. Convolutional GANs: LAPGAN addressed this difficulty by decomposing image generation across multiple scales with conditional convolutional GANs.Each pyramid layer was generated given the layer above, matching the corresponding decomposition of a ground-truth image.
  • B. Convolutional GANs: DCGAN introduced architectures for training deep convolutional generator and discriminator networks.The approach used strided and fractionally-strided convolutions for learned spatial down-sampling and up-sampling.
  • B. Convolutional GANs: During training, GANs encourage the generated distribution pg(x) to match the real-data distribution pdata(x), with learned representations stored in network weights.For an appropriately parameterized and trained GAN, the two distributions can become nearly identical.
  • B. Convolutional GANs: Strided and fractionally-strided convolutions learn sampling-rate and spatial-location changes needed to map between image space and lower-dimensional latent or discriminator representations.These operators support the spatial transformations required by convolutional GANs.
  • B. Convolutional GANs: Volumetric convolutions extended GAN image synthesis to 3D samples, including novel chairs, tables, and cars, and mappings from 2D images to 3D objects.The method synthesized 3D object versions from objects portrayed in 2D images.

C. Conditional GANs · D. GANs with Inference Models · E. Adversarial Autoencoders (AAE)

The merged sections extend GANs through class conditioning, explicit inference networks, and adversarially regularized autoencoders. These approaches improve controllability or latent-space structure while exposing reconstruction-fidelity and training-objective tradeoffs.

  • C. Conditional GANs: Conditional GANs make both generator and discriminator class-conditional, enabling class-conditional image synthesis and better representations for multi-modal data generation.InfoGAN instead decomposes noise into an incompressible source and latent code to discover latent variation factors.
  • D. GANs with Inference Models: Original GANs lacked a mechanism mapping observations x into latent vectors, motivating inversion methods and ALI/BiGAN inference networks.The discriminators in these extensions examine joint data-and-latent representations.
  • D. GANs with Inference Models: ALI/BiGANs use an encoder and decoder jointly trained against a discriminator that distinguishes real image-encoding pairs from generated image-input pairs.The encoder maps image space to latent space, while the decoder maps latent space to image space.
  • D. GANs with Inference Models: ALI/BiGAN reconstructions typically have poor fidelity, although an additional adversarial cost on data samples and reconstructions can improve it.The added cost targets the distribution of data samples and their reconstructions.
  • E. Adversarial Autoencoders (AAE): Autoencoders learn encoder and decoder mappings between data and latent spaces, with their composition producing reconstructions trained to remain close to the original.They learn a deterministic data-to-latent mapping and a latent-to-data mapping.
  • C. Conditional GANs: InfoGAN augments discrimination by estimating the class label while seeking latent factors of variation through a structured noise source.Its noise source is divided into an incompressible component and a latent code.
  • E. Adversarial Autoencoders (AAE): Adversarial autoencoders apply adversarial training between latent representations and a desired prior, combining reconstruction error with a measure of distributional difference.This latent-space GAN supports useful latent organization and feedforward ancestral sampling.
  • E. Adversarial Autoencoders (AAE): Adversarial Variational Bayes unifies variational autoencoders with adversarial training by replacing the variational autoencoder’s Kullback-Leibler divergence with an adversarial objective.AVB aims to optimize the same criterion as variational autoencoders.

IV. TRAINING GANS · A. Introduction

GAN training alternates between optimizing a discriminator to classify accurately and a generator to confuse it, with theoretical optimality when generated and real distributions match. Despite this framework, training is often unstable because of convergence failures, mode collapse, vanishing discriminator gradients, and optimization difficulties.

  • A. Introduction: GAN training alternates discriminator updates that improve classification accuracy with generator updates that increasingly confuse the discriminator.The main training loop may update the discriminator k times before updating the generator.
  • A. Introduction: The training cost is evaluated with a value function V (G, D) that depends jointly on the generator and discriminator.Training is formulated as an optimization problem over both models’ parameters.
  • A. Introduction: For a fixed generator, the unique optimal discriminator is D∗(x) = pdata(x) pdata(x)+pg(x), while the generator is optimal when pg(x) = pdata(x).At generator optimality, the discriminator predicts 0.5 for every sample.
  • A. Introduction: In practice, the discriminator may receive only a few updates while the generator is updated simultaneously, typically using the non-saturating criterion maxG log D(G(z)).This replaces minG log(1 −D(G(z))) for generator training.
  • A. Introduction: GAN training remains challenging and unstable despite the theoretical existence of unique solutions, motivating heuristic approaches based on empirical training symptoms.These heuristics are discussed in Section IV-B.
  • A. Introduction: Observed failure modes include difficulty converging, generator collapse to highly similar samples, and discriminator loss rapidly reaching zero.These symptoms respectively concern model convergence, sample diversity, and discriminator behavior.
  • A. Introduction: Gradient descent can be inappropriate because GAN optimization may have a saddle-point solution, although stochastic gradient descent is often used for neural-network updates.Early explanations of instability were proposed by Goodfellow and Salimans et al.
  • A. Introduction: Because real and generated distributions may lie on lower-dimensional, nonoverlapping supports, a discriminator can distinguish them with 100% accuracy and provide ineffective generator gradients.This explains why discriminator error may quickly converge to zero; when the discriminator is optimal, generator training corresponds to minimizing Jensen-Shannon divergence, whereas a nonoptimal discriminator can yield less meaningful or inaccurate updates.

B. Training Tricks

GAN training was improved through architectural guidelines and heuristic techniques that stabilize optimization, increase generator information, prevent mode collapse, and avoid overly confident discriminators. These methods include DCGAN design principles, feature matching, mini-batch discrimination, parameter averaging, virtual batch normalization, label smoothing, and input noise.

  • Architectural guidelines: DCGANs introduced CNN architecture guidelines for constructing and training both the generator and discriminator.The approach emerged from extensive exploration of CNN architectures used in computer vision and emphasized strided and fractionally-strided convolutions.
  • Stabilization heuristics: Feature matching modifies the generator objective to increase the information available during GAN training.The discriminator continues to distinguish real from fake samples while the generator objective is adjusted.
  • Stabilization heuristics: Mini-batch discrimination gives the discriminator features encoding distances among samples, helping prevent mode collapse from repeated generator outputs.The discriminator can identify when the generator produces the same outputs.
  • Stabilization heuristics: Heuristic averaging penalizes deviations from a running parameter average, while virtual batch normalization uses statistics from a fixed reference mini-batch to reduce sample dependence.Both techniques are intended to improve training behavior: averaging can help convergence to equilibrium, and reference-batch statistics reduce interactions among samples.
  • Discriminator regularization: One-sided label smoothing sets the discriminator target to 0.9 rather than 1, reducing overconfidence and preserving stronger generator gradients.Adding noise to discriminator inputs was also proposed as a way to challenge the discriminator.

C. Alternative formulations

Alternative GAN formulations generalize the training objective beyond Jensen-Shannon divergence and introduce cost functions intended to address vanishing gradients.

  • Alternative cost functions: Alternative cost functions are designed to directly address vanishing gradients.The section considers these cost functions separately from information-theoretic interpretations and generalizations.
  • Information-theoretic interpretations and generalizations: f-GANs generalize GAN training to estimate f-divergences, including the Jensen-Shannon and Kullback-Leibler divergences.The approximation applies Fenchel conjugates of the desired f-divergence to generated samples after passing them through the discriminator.

1) Generalisations of the GAN cost function: … V. THE STRUCTURE OF LATENT SPACE

The paper surveys alternative GAN objectives, variant-specific training and conditioning strategies, and the structured latent spaces learned by GANs. It emphasizes WGANs’ more useful generator gradients and latent-space operations such as attribute manipulation and inference through encoders.

  • 2) Alternative Cost functions to prevent vanishing gradients:: WGAN replaces the original GAN cost with an approximation of the Wasserstein distance, making useful generator-update gradients more likely.Its critic must be k-Lipschitz continuous, implemented practically by clipping parameters.
  • D. A Brief Comparison of GAN Variants: GANs are difficult to train partly because of vanishing gradients, and all discussed models require careful hyperparameter tuning and model selection.The AAE and WGAN are identified as comparatively easier to train.
  • D. A Brief Comparison of GAN Variants: AAEs are relatively easy to train because their adversarial loss operates on a simpler, lower-dimensional distribution than image data.The passage contrasts AAE training with the complexity of image-data distributions.
  • D. A Brief Comparison of GAN Variants: Conditional GANs enable synthesis with user-specified content, whereas GAN or WGAN samples may belong to any class represented in the training data.This distinguishes unconditional class membership from user-directed content specification.
  • D. A Brief Comparison of GAN Variants: GAN latent spaces contain meaningful organization, but vanilla GANs lack an inference model for mapping samples back to latent representations.BiGANs and ALI provide inference mechanisms, although reconstruction quality does not necessarily imply faithful encoding and decoding.
  • V. THE STRUCTURE OF LATENT SPACE: GANs learn structured geometric vector spaces with fewer dimensions than the modeled data space, supporting trajectories such as face rotations.The passage relates this property to VAEs and word2vec.
  • V. THE STRUCTURE OF LATENT SPACE: Latent-space image analogies can add visual attributes, such as eyeglasses, to a bare face.This illustrates semantic manipulation through operations in the learned latent space.
  • V. THE STRUCTURE OF LATENT SPACE: Encoders support inverse mapping from modeled data to latent space, enabling labelled-image collections to reveal concept vectors for high-level attributes.Examples include concepts such as smiling or wearing attributes.

VI. APPLICATIONS OF GANS · A. Classification and Regression

GANs support diverse computer-vision applications, including classification, image synthesis, and super-resolution. In classification and regression, discriminator or encoder representations enable downstream learning, while adversarially refined synthetic data supports strong performance when labels are scarce.

  • VI. APPLICATIONS OF GANS: GAN applications span image manipulation, analysis, and characterization, although surveyed examples do not capture their full potential breadth.The review examines selected computer-vision applications and notes that discovering new adversarial-training applications remains active research.
  • VI. APPLICATIONS OF GANS: GANs are used for classification, image synthesis under constraints, and super-resolution with adversarial loss components.Classification provides quantitative assessment of unsupervised features, while super-resolution illustrates supplementing existing methods with adversarial loss.
  • A. Classification and Regression: After training, discriminator convolutional outputs can serve as features for linear models trained with modest labeled datasets.This enables GAN representations to support downstream tasks using limited (image, label) pairs.
  • A. Classification and Regression: Regularized L2-SVM classification of DCGAN discriminator features achieved good scores on supervised and semi-supervised datasets, including data disjoint from original training.The feature vector was extracted from the trained discriminator and evaluated across both supervised and semi-supervised settings.
  • A. Classification and Regression: ALI representations from its last three hidden encoder layers achieved a significantly lower misclassification rate than DCGAN using a similar L2-SVM classifier.ALI also achieved state-of-the-art classification results when label information was incorporated into training.
  • A. Classification and Regression: GAN-refined synthetic images preserved annotations and enabled state-of-the-art pose and gaze estimation using no real training data.The approach synthesizes additional labeled samples for settings where labeled training data is limited.
  • A. Classification and Regression: Classification tasks remain important quantitative tools for assessing GAN performance because generated-sample quality is difficult to compare directly across models.This role is expected to persist as computer-vision applications become more diverse.

B. Image Synthesis · C. Image-to-image translation

GAN research advances image synthesis through multiscale generation, conditioning on labels, text, locations, and attributes, and supports intuitive image editing. For image-to-image translation, pix2pix learns general mappings with a training loss, while CycleGAN uses cycle consistency without requiring matched image pairs.

  • B. Image Synthesis: LAPGAN generates images coarse-to-fine by cascading convolutional networks within a Laplacian pyramid framework.
  • B. Image Synthesis: LAPGAN extends conditional GANs by providing both generator and discriminator networks with additional label information, a common practice for improving image quality.
  • B. Image Synthesis: Text-conditioned GANs can synthesize several plausible bird images matching descriptions such as color, markings, and beak shape.
  • B. Image Synthesis: GAWWN conditions image generation on location, enabling users to build large images incrementally with textual part descriptions and supplied bounding boxes.
  • B. Image Synthesis: Conditional GANs synthesize samples with specific attributes and support intuitive edits such as changing hairstyles, adding glasses, or making people look younger.
  • C. Image-to-image translation: Conditional adversarial networks translate input images into output images, and pix2pix provides a general-purpose model that learns the mapping and its training loss.
  • C. Image-to-image translation: CycleGAN introduces cycle consistency loss to preserve the original image after translation and reverse translation.
  • C. Image-to-image translation: CycleGAN does not require matching image pairs, simplifying data preparation and broadening applications such as artistic style transfer.

D. Super-resolution · VII. DISCUSSION · A. Open Questions

SRGAN generates photo-realistic high-resolution images from low-resolution inputs by combining adversarial, perceptual, and regularization losses. Despite progress using unlabelled data and addressing training and evaluation, GANs retain open challenges including mode collapse.

  • D. Super-resolution: Super-resolution generates high-resolution images from lower-resolution inputs, with SRGAN adding adversarial loss to constrain outputs to the natural-image manifold.The model infers photo-realistic details while up-sampling.
  • D. Super-resolution: SRGAN conditions its generator on a low-resolution image and performs 4x up-scaling while combining adversarial, perceptual, and regularization losses.Perceptual loss comes from a pretrained classifier, while regularization encourages spatial coherence.
  • D. Super-resolution: SRGAN is straightforward to customize because training pairs can be constructed by down-sampling high-resolution images from domain-specific corpora.Generated photo-realistic details vary with the image domain represented in the training set.
  • A. Open Questions: GANs attract attention by leveraging vast amounts of unlabelled data, although several open challenges remain despite progress in training and evaluation.The passage frames these issues as continuing research challenges.
  • A. Open Questions: GAN applications include images conditioned on text and location, as well as bidirectional translations between unordered image collections.GAWWN uses keypoints or bounding boxes, while CycleGAN maps examples such as Monet paintings to landscapes and zebras to horses.
  • A. Open Questions: A common GAN failure is mode collapse, where the generator produces a small family of similar samples or, in the worst case, a single sample.The passage distinguishes partial collapse from complete collapse.

1) Mode Collapse: … B. Conclusions

The review identifies mode collapse, saddle-point convergence, equilibrium existence, and generative-model evaluation as unresolved GAN challenges, while highlighting opportunities enabled by unlabelled data and deep networks.

  • 1) Mode Collapse:: Mode collapse can be alleviated by balancing discriminator sample distributions, using multiple GANs, or changing the statistical distance measure.These approaches aim to increase generator diversity or cover different probability-distribution modes.
  • 1) Mode Collapse:: Unrolled objectives give the generator foresight into discriminator updates, helping prevent endless mode hopping caused by one-step generator training.With the usual objective, the discriminator can force the generator to move between convergence and mode-hopping behavior.
  • 2) Training instability – saddle points:: GAN objectives have saddle points because their loss Hessians are indefinite, making convergence dependent on good initialization for first-derivative optimizers.Stable-manifold analysis links convergence difficulties to Jacobian eigenvalues with zero real parts.
  • 2) Training instability – saddle points:: Second-order optimizers may address saddle-point instability, but Newton-type methods scale quadratically or cubically with parameter dimension.This creates an open problem in applying and scaling second-order optimization for adversarial training.
  • 2) Training instability – saddle points:: A GAN equilibrium may not exist below a certain model capacity, and apparent convergence can still leave the learned distribution far from the target.The existence of equilibrium is connected to representing a finite mixture of neural networks.
  • 3) Evaluating Generative Models:: Evaluating GAN sample fidelity and comparing models remain open questions because different evaluation measures can produce conflicting quality conclusions.The passage raises likelihood estimation and cross-method model comparison as unresolved evaluation issues.
  • B. Conclusions: GANs offer opportunities to learn nonlinear latent-to-data mappings from abundant unlabelled image data, motivating further theoretical, algorithmic, and application-oriented development.Their potential extends deep representation learning to image data that remains unused because it lacks labels.
Loading 1710.07035v1…