PaperlayerView exact revision on arXiv

Source-linked AI summary

NIPS 2016 Tutorial: Generative Adversarial Networks

Ian Goodfellow

arXiv:1701.00160v4cs.LG

TL;DR

The tutorial addresses why generative modeling matters and how GANs operate and compare with other generative models. It presents GANs as supervised methods for approximating difficult cost functions, while identifying training equilibrium and differentiability constraints. GANs support state-of-the-art image generation and manipulation systems, but their broader potential remains bounded by these limitations.

  • Problem

    Generative modeling matters for representing high-dimensional distributions, supporting multiple valid outputs, and producing realistic samples for applications such as reinforcement learning and image synthesis.

  • Method

    GANs use supervised learning and ratio estimation to approximate intractable cost functions, including the KL divergence used for maximum likelihood.

  • Results

    GANs are crucial to many state-of-the-art image generation and manipulation systems and may enable additional applications.

  • Takeaways & Limitations

    GANs provide a framework for generative modeling that connects supervised ratio estimation with practical image generation and manipulation systems.

  • Takeaways & Limitations

    The GAN framework requires a differentiable generator, preventing direct generation of discrete data such as one-hot word or character representations.

Abstract

from arXiv · show

This report summarizes the tutorial presented by the author at NIPS 2016 on generative adversarial networks (GANs). The tutorial describes: (1) Why generative modeling is a topic worth studying, (2) how generative models work, and how GANs compare to other generative models, (3) the details of how GANs work, (4) research frontiers in GANs, and (5) state-of-the-art image models that combine GANs with other methods. Finally, the tutorial contains three exercises for readers to complete, and the solutions to these exercises.

Introduction

This report summarizes a NIPS 2016 tutorial on GANs, covering why generative modeling matters, how generative models and GANs work, research frontiers, and combined state-of-the-art image models. It also includes exercises and their solutions.

  • The tutorial is a report of the author's NIPS 2016 presentation and is not intended as a comprehensive review of GAN research.
  • The tutorial is organized around motivation, generative-model fundamentals, GAN mechanics, research frontiers, and combined image-generation methods.
  • Density-estimation models evaluate pmodel(x) for particular x values, whereas sample-generating models produce new examples from the learned distribution.
  • Generative models learn an estimate pmodel of a distribution from training samples drawn from pdata, either explicitly or through generated samples.

1 Why study generative modeling?

The tutorial motivates generative modeling through its role in representing high-dimensional distributions, handling multiple valid outputs, and supporting applications that require realistic samples. Examples span prediction, super-resolution, artistic assistance, photo editing, and image-to-image translation.

  • Generative models test the ability to represent and manipulate high-dimensional probability distributions used across applied mathematics and engineering.
  • Generative models support reinforcement learning by simulating possible futures for planning and model-based algorithms.
  • Generative models can use incomplete data, including missing labels in semi-supervised learning, to make predictions on inputs with missing information.
  • GANs help model multi-modal outputs, avoiding the blurry averages produced when mean squared error forces one prediction among multiple correct answers.
  • Single-image super-resolution requires generative modeling because the model must synthesize information absent from the low-resolution input.
  • Generative applications include interactive sketch-to-image creation, realistic photo modification, and image-to-image translation such as satellite-photo-to-map conversion.
  • These applications provide reasons to invest time and resources in improving generative models.

2 How do generative models work? How do GANs compare to others?

Generative models learn to represent a data distribution, and the tutorial compares GANs with maximum-likelihood, explicit-density, variational, and Markov-chain approaches. GANs avoid several computational and modeling constraints, but introduce the difficulty of finding a game-theoretic equilibrium.

  • Maximum likelihood: Maximum likelihood chooses model parameters that maximize the probability assigned to training data, equivalently minimizing KL divergence from the data distribution.The tutorial discusses maximum likelihood in log space for simpler derivatives and better numerical stability.
  • Comparing model families: FVBNs generate samples sequentially with cost O(n), whereas GANs can generate samples in parallel.Sequential generation first produces x_1, then x_2, and so on, making runtime proportional to sample dimensionality.
  • Explicit-density models: Explicit-density models are effective because they permit direct optimization of training-data log-likelihood, but tractable density families are limited.The limitation is addressed through carefully structured models or tractable approximations to likelihoods and gradients.
  • Comparing model families: GANs require no Markov chains, allow few restrictions on generator design, and require no variational bound.The tutorial contrasts these properties with Boltzmann machines, GSNs, nonlinear ICA, and VAEs.
  • Comparing model families: GANs are known to be asymptotically consistent with universal-approximator model families, while consistency of some VAEs remains unproven.The tutorial also notes that GANs are designed to recover pdata at Nash equilibrium with sufficient model capacity and infinite data.
  • Comparing model families: GANs are subjectively regarded as producing better samples than other methods, but training requires finding a Nash equilibrium rather than optimizing one objective.This equilibrium problem is described as more difficult than ordinary objective-function optimization.

3 How do GANs work?

After reviewing other generative models, the tutorial turns to how GANs work and emphasizes that their operation differs from those models.

  • The tutorial has reviewed several other generative models before introducing GANs' operation.
  • GANs do not work in the same way as the other generative models discussed.
  • The section focuses on explaining how GANs themselves work.

3.1 The GAN framework

The GAN framework uses differentiable generator and discriminator functions in an adversarial game: the generator maps latent samples to data, while the discriminator distinguishes real from generated examples. Training updates both players’ parameters using simultaneous stochastic gradient steps.

  • The GAN framework: The generator creates samples intended to match the training distribution, while the discriminator classifies inputs as real or fake.The generator is trained to fool the discriminator, analogous to a counterfeiter competing against police.
  • The GAN framework: The discriminator D receives x, and the generator G receives latent variable z; both are differentiable with respect to inputs and parameters.Typically, both functions are implemented as deep neural networks.
  • The GAN framework: Each player minimizes its own cost while controlling only its own parameters, so the coupled objectives define a game rather than a single optimization problem.A local differential Nash equilibrium is a local minimum of each player’s cost with respect to that player’s parameters.
  • The GAN framework: Sampling z from a simple prior and applying G(z) yields samples from the model distribution, with the generator’s inputs allowed at different points within the network.The generator is commonly represented by a deep neural network.
  • The GAN framework: Under standard backpropagation, GANs cannot have discrete generator outputs, whereas VAEs cannot have discrete inputs to the generator.The two frameworks otherwise have a large intersection, with each able to train some models the other cannot.
  • The GAN framework: Training uses minibatches of data x and latent variables z, followed by one gradient step for each player in simultaneous SGD.The tutorial notes that Adam is usually a good choice and that, as of late 2016, simultaneous one-step updates worked best in practice according to the author.

3.2 Cost functions

GANs share a discriminator objective while varying the generator objective, yielding games with different theoretical properties and practical training behavior. Their density-ratio estimate distinguishes GAN approximations from other deep generative-model approximations, while divergence choice alone does not explain sharp samples or mode collapse.

  • Discriminator and generator objectives: All GAN games use the same discriminator cost and differ only in the generator cost.The discriminator objective is standard binary-classification cross-entropy over data and generated minibatches.
  • Density-ratio estimation: The discriminator estimates the ratio of data to model densities, enabling computation of divergences and their gradients.This supervised density-ratio approximation distinguishes GANs from variational autoencoders and Boltzmann machines, which use lower bounds or Markov chains.
  • Minimax game: The minimax game is theoretically tractable and resembles Jensen-Shannon divergence minimization, but its convergence analysis does not apply to deep networks updated in parameter space.The generator’s minimax gradient can vanish when the discriminator confidently rejects generated samples.
  • Alternative generator objectives: The non-saturating generator objective maximizes the discriminator’s probability of being mistaken, providing a strong gradient when the generator is losing.This modification is heuristically motivated and makes the game non-zero-sum.
  • Maximum likelihood: GANs can approximate maximum likelihood under an optimal discriminator, with the equivalence holding in expectation despite sampling variance.Other maximum-likelihood approximations within the GAN framework are also possible.
  • Divergence, sharpness, and mode collapse: Maximum-likelihood-trained GANs still produce sharp samples and select few modes, while GAN mode collapse appears driven by training defects rather than divergence choice.The reason GANs produce sharp samples remains unclear.
  • Comparison of cost functions: The non-saturating cost has lower sample variance than the maximum-likelihood game, which may explain its greater practical success and motivates variance-reduction research.In the maximum-likelihood game, most gradient comes from a few generated samples most likely to be real.

3.3 The DCGAN architecture

DCGAN is a deep convolutional GAN architecture built around normalization, all-convolutional structure, transposed convolutions, and Adam optimization. It enabled single-shot high-resolution generation in restricted domains and showed that latent-space arithmetic can encode semantic attributes.

  • Architecture: DCGAN denotes a specific deep convolutional GAN architecture that became a loose foundation for many later GANs.Its name refers to this architectural style even though earlier GANs were also deep and convolutional.
  • Architecture: Batch normalization is used in most discriminator and generator layers, except at the generator output and discriminator input.The two discriminator minibatches are normalized separately so the model can learn the data distribution’s mean and scale.
  • Architecture: The network omits pooling and unpooling, using transposed convolutions with stride greater than one to increase generator spatial dimensions.This structure is largely borrowed from the all-convolutional network.
  • Optimization: Adam is used instead of SGD with momentum.
  • Results: DCGANs were the first GANs to generate high-resolution images in a single shot, whereas LAPGANs required multi-stage Laplacian-pyramid generation.On restricted image domains such as bedrooms, DCGANs generated high-quality samples.
  • Results: DCGAN latent codes support meaningful arithmetic, with vector operations corresponding to semantic attribute changes.The tutorial illustrates this behavior with operations on attributes such as those represented in figure 19.

3.4 How do GANs relate to noise-contrastive estimation and maximum likelihood?

The tutorial compares GANs with noise-contrastive estimation and maximum likelihood through their shared minimax-game view. Although minimax GANs use the NCE cost as a value function, the methods learn different objects because they focus on different players.

  • Noise-contrastive estimation: Minimax GANs and noise-contrastive estimation use the same cost function as a value function but learn different things by focusing on different players.The tutorial contrasts the generator-centered role of GANs with the discriminator-centered role of NCE.
  • Latent representations: DCGAN latent representations can disentangle semantic attributes, with vector arithmetic combining concepts such as gender and wearing glasses.The tutorial presents this as a demonstrated connection between latent-code arithmetic and semantic attributes.
  • Shared game formulation: GANs, NCE, and maximum likelihood can all be interpreted as minimax games with the same value function, differing in where the model distribution appears.For GANs, the generator is the model; for NCE and maximum likelihood, the model is part of the discriminator.

4 Tips and Tricks

The tutorial presents labels, one-sided label smoothing, and batch-normalization variants as useful GAN techniques, while emphasizing that their effects and trade-offs are context-dependent. It also cautions against simplistic assumptions about balancing the generator and discriminator.

  • 4.1 Train with labels: Label information usually produces a dramatic improvement in the subjective quality of GAN samples.This can occur through explicit class conditioning or by training the discriminator to recognize real-object classes.
  • 4.1 Train with labels: Comparisons involving labels must use the same labeling setup, because comparing labeled and unlabeled models is considered an unfair benchmark.Class-conditional models should likewise be compared only with other class-conditional models.
  • 4.2 One-sided label smoothing: One-sided label smoothing replaces the real target with a value slightly below one, such as .9, while leaving the fake target at zero.The modification is intended to encourage soft discriminator probabilities and prevent extreme logit extrapolation.
  • 4.2 One-sided label smoothing: Smoothing fake labels can alter the optimal discriminator and reinforce spurious generator modes, whereas smoothing only real labels merely scales down its optimal value.The tutorial therefore warns against smoothing labels for fake samples.
  • 4.3 Virtual batch normalization: Batch normalization can improve optimization but, with small minibatches, its fluctuating statistics may influence generated images more than their individual z codes.This can correlate samples within a minibatch, motivating reference batch normalization, which uses fixed reference-batch statistics.
  • Training dynamics: There is no compelling demonstration that balancing the generator and discriminator is necessary or feasible; discriminator overpowering can be acceptable because it estimates a density ratio.If generator gradients vanish, the tutorial recommends changing the game parameterization rather than limiting discriminator power.

5 Research Frontiers

The tutorial identifies non-convergence and mode collapse as major GAN research problems, while surveying methods, applications, evaluation challenges, and limitations that define open research frontiers.

  • 5.1 Non-convergence: Non-convergence is GANs’ largest research problem because players can undo each other’s progress instead of reaching a useful equilibrium.Simultaneous gradient descent may converge in some games, but there is no theoretical guarantee for deep-network parameter updates, and GANs often oscillate in practice.
  • 5.1.1 Mode collapse: Mode collapse occurs when multiple latent inputs map to the same output; partial collapse is common even though complete collapse is rare.The tutorial illustrates generators cycling among individual modes rather than representing all modes in the target distribution.
  • 5.1.1 Mode collapse: Mode collapse does not appear to result from any particular cost function, including Jensen-Shannon divergence or approximations to DKL(pdata∥pmodel).The tutorial notes that generators can collapse to fewer modes than Jensen-Shannon divergence would prefer.
  • 5.1.1 Mode collapse: Mode collapse limits GAN applications to settings where producing a small number of distinct outputs is acceptable, although newer approaches can improve output diversity.The tutorial cites text-to-image synthesis as an example and reports higher diversity for StackGANs than previous GAN-based approaches.
  • 5.1.1 Mode collapse: Minibatch features reduce mode collapse enough that counting, perspective, and global-structure defects become the most visible problems in CIFAR-10 samples.The method lets the discriminator compare generated examples with one another and detect unusually similar samples.
  • 5.2 Evaluation: Quantitative evaluation remains unresolved because likelihood and sample quality can disagree, while GAN likelihood is difficult to estimate.The tutorial also notes that no clearly justified quantitative score for samples exists.
  • 5.3 Discrete outputs: GAN generators must be differentiable, which prevents direct generation of discrete data such as one-hot words or characters.Removing this restriction is presented as an important direction for applying GANs to natural-language processing.
  • 5.4 Semi-supervised learning: Feature matching GANs achieve state-of-the-art semi-supervised performance across MNIST, SVHN, and CIFAR-10 with label counts ranging from 20 to 8,000.These methods perform well despite typical training setups using 50,000 or more labels.

6 Plug and Play Generative Networks

Plug and play generative networks combine approximate Langevin sampling with a denoising autoencoder and GAN loss to generate diverse, high-resolution ImageNet images.

  • 6 Plug and Play Generative Networks: PPGNs substantially improve the diversity of high-resolution images generated from ImageNet classes.The tutorial presents PPGNs as a recently released generative model whose results are illustrated in figure 33.
  • 6 Plug and Play Generative Networks: PPGNs remain poorly understood because their design recommendations are based mainly on empirical observation rather than theory.The tutorial describes the model as complicated and expects its operation to become clearer through future research.
  • 6 Plug and Play Generative Networks: PPGNs use approximate Langevin sampling with a Markov chain, estimating sampler gradients through a denoising autoencoder trained with several losses.One of those losses is the GAN loss.
  • 6 Plug and Play Generative Networks: The GAN loss is crucial for obtaining high-quality PPGN images.Removing it prevents the denoising autoencoder from producing compelling images.

7 Exercises

The tutorial includes three exercises covering optimal discriminator behavior, game dynamics, and a generator cost related to maximum likelihood, with solutions provided later.

  • Exercises: The tutorial provides solutions to all three exercises in section 8.The exercises are intended to check readers’ understanding.
  • Exercise 1: The first exercise asks readers to derive the discriminator’s optimal strategy in function space and identify the assumptions required.It concerns the discriminator objective described in equation 8.
  • Exercise 2: The second exercise studies equilibrium and simultaneous-gradient-descent trajectories in a scalar minimax game.The exercise treats gradient descent as a continuous-time process with an infinitesimal learning rate.
  • Exercise 3: The third exercise derives a generator cost whose expected gradient matches the gradient of DKL(pdata∥pmodel) when the discriminator is optimal.Readers determine the form of a function f in the proposed cost.

8 Solutions to exercises

The exercises derive GANs’ density-ratio estimation mechanism, characterize a saddle-point game, and examine gradient-descent dynamics and divergence estimation under stated assumptions.

  • 8.1 The optimal discriminator strategy: The optimal discriminator estimates the ratio of data density to the sum of data and model densities.This ratio is the key approximation mechanism used by GANs.
  • 8.2 Gradient descent for games: The value function V(x,y)=xy has a saddle-point equilibrium at x=y=0, but not every saddle point is necessarily an equilibrium.This game’s saddle point satisfies the requirement that an infinitesimal unilateral perturbation cannot reduce that player’s cost.
  • 8.2 Gradient descent for games: Simultaneous gradient descent with an infinitesimal learning rate follows a circular orbit and never approaches the equilibrium.With a larger learning rate, the trajectory can spiral outward forever.
  • 8.2 Gradient descent for games: For GANs, convergence of simultaneous gradient descent has no theoretical prediction, making convergent algorithms an open research problem.The tutorial notes that simultaneous gradient descent converges for some games but not for others.
  • 8.3 Using the discriminator to estimate divergences: The divergence-estimation derivation assumes nonnegative model density and conditions allowing differentiation and integration to be exchanged.The discriminator-based construction also requires handling expectations drawn from the generator rather than the data.
  • 8.3 Using the discriminator to estimate divergences: The density-ratio estimate can be used to calculate a variety of divergences, including the KL divergence used for maximum likelihood estimation.Importance sampling reweights generator samples to compensate for their being drawn from the generator rather than the data.

9 Conclusion

GANs use supervised density-ratio estimation to approximate otherwise intractable costs, including KL divergence, and underpin state-of-the-art image generation and manipulation systems. Their broader potential depends on improved understanding and training methods for high-dimensional non-convex games.

  • 9 Conclusion: GANs use supervised learning to approximate an intractable cost function through density-ratio estimation.The tutorial compares this role with Markov-chain approximation in Boltzmann machines and variational lower bounds in VAEs.
  • 9 Conclusion: GANs can approximate multiple cost functions, including the KL divergence used for maximum likelihood estimation.
  • 9 Conclusion: Training GANs remains difficult because it requires finding Nash equilibria in high-dimensional, continuous, non-convex games.The tutorial calls for better theoretical understanding and training algorithms.
  • 9 Conclusion: GANs are crucial to many state-of-the-art image generation and manipulation systems and may enable additional applications in the future.
Loading 1701.00160v4…