NIPS 2016 Tutorial: Generative Adversarial Networks

Scope

• The tutorial prioritizes frequently asked audience questions and explicitly does not provide a comprehensive review of GAN research.Its two-hour oral format and limited scope excluded papers that were not relevant to the most frequent questions.Source: Introduction · p. 1

Coverage

• The tutorial covers generative modeling, comparisons with other models, GAN mechanisms, research frontiers, combined image models, and three exercises with solutions.These topics define the tutorial’s overall structure and intended coverage.Source: Introduction · p. 1

Generative models

• Generative models learn an estimate pmodel of a data distribution pdata from training samples, either explicitly, through sampling, or through both capabilities.GANs focus primarily on generating samples, although they can also be designed to estimate distributions explicitly.Source: Introduction · p. 2, Figure 1 · p. 2, Figure 2 · p. 2

• Generative modeling tests the ability to represent and manipulate high-dimensional probability distributions important across applied mathematics and engineering.Source: §1 · p. 3
• Generative models can simulate possible futures for planning, enable learning in imaginary environments, guide exploration, and support inverse reinforcement learning.Source: §1 · p. 3
• Generative models, especially GANs, can learn from missing data and improve semi-supervised learning by exploiting abundant unlabeled examples.Source: §1 · p. 3
• Generative models enable multi-modal outputs, avoiding the blur caused by averaging multiple valid predictions with mean squared error.For video-frame prediction, an additional GAN loss produces sharp, recognizable images corresponding to multiple possible futures.Source: §1 · p. 4, Figure 3 · p. 4
• Generative modeling is required for single-image super-resolution because low-resolution inputs correspond to many possible high-resolution images, making averaged outputs blurry.GAN-based super-resolution improves realism by sampling plausible images rather than imposing one averaged reconstruction.Source: §1 · pp. 4–5, Figure 4 · p. 5
• Generative models support creative applications including interactive realistic image creation, photo modification, and image-to-image translation between sketches, photographs, and maps.These applications illustrate a broad and difficult-to-anticipate range of useful uses for generative models.Source: §1 · p. 5, Figure 5 · p. 6, Figure 6 · p. 7

2.1 Maximum likelihood estimation

• Maximum likelihood chooses model parameters that maximize training-data likelihood, equivalently minimizing KL divergence from the empirical data distribution to the model.Log space converts products into sums, simplifying derivatives and reducing numerical underflow; if the true distribution lies within the model family, exact recovery is possible in principle.Source: §2.1 · p. 9

2.2 A taxonomy of deep generative models

• The taxonomy separates explicit-density models from implicit models, which interact with the represented distribution indirectly, typically by generating samples; GANs use single-step implicit sampling.Explicit models may have tractable or intractable densities, while implicit models may use Markov chains or generate samples directly.Source: §2.2 · p. 9, Figure 9 · p. 12

2.3 Explicit density models

• Explicit-density models directly define pmodel(x; θ), but must balance expressive power against computational tractability through structural design or likelihood approximations.Tractable models optimize likelihood directly, whereas intractable models use variational or Monte Carlo approximations.Source: §2.3 · p. 11, §2.3.2 · p. 14

2.3.1 Tractable explicit models

• Tractable explicit models enable direct log-likelihood optimization but are limited by family-specific design constraints, including sequential generation or invertibility requirements.FVBNs generate entries one at a time, while nonlinear ICA requires latent and data spaces to have equal dimensionality; GANs were designed to avoid these restrictions.Source: §2.3.1 · pp. 11, 13

2.3.2 Explicit models requiring approximation

• Approximate explicit models use variational or Markov chain methods; variational bounds can bias learned distributions, while Markov chains can converge slowly and become inefficient in high dimensions.GANs were designed to avoid both variational-bound bias and Markov-chain costs, though their better sample quality is described as a subjective judgment rather than an empirical fact.Source: §2.3.2 · pp. 14, 16

2.4 Implicit density models

• Implicit-density models train through indirect interaction with pmodel, usually sampling; GANs avoid the high-dimensional scaling and computational costs of Markov-chain sampling.GANs belong to the single-step implicit-model family, later joined by kernelized moment-matching models.Source: §2.4 · pp. 16–17

2.5 Comparing GANs to other generative models

• GANs were designed to avoid disadvantages of other models: sequential generation, restrictive generator design, Markov chains, and variational bounds, while introducing harder game-based training.The generator need not be invertible or match latent and sample dimensionalities, but training requires solving a Nash equilibrium rather than optimizing a single objective.Source: §2.5 · p. 17
• GANs generate samples in parallel, impose few generator restrictions, avoid Markov chains and variational bounds, and are subjectively regarded as producing better samples, but require finding a Nash equilibrium.These advantages are relative to FVBNs, nonlinear ICA, Boltzmann machines, GSNs, and VAEs; asymptotic consistency is established for known universal-approximator GAN families but remains conjectural for some VAEs.Source: §2.5 · p. 17

3 How do GANs work?

• The section introduces an explanation of how GANs work, emphasizing that they differ from other generative models.Source: §3 · p. 17

The GAN framework

• GANs pit a generator against a discriminator: the generator creates samples intended to match the training distribution, while the discriminator distinguishes real from fake.The discriminator uses supervised learning, and the generator is trained to fool it; the framework is illustrated in Figure 12.Source: §3.1 · pp. 17–18, Figure 12 · p. 19
• GANs are structured probabilistic models with latent variables z and observed variables x, represented by differentiable functions G(z; θ(G)) and D(x; θ(D)).The generator maps prior samples z to modeled samples x, while the discriminator takes x as input.Source: §3.1 · p. 18
• Because each player minimizes its own cost while controlling only its parameters, the solution is a local differential Nash equilibrium rather than an ordinary optimization minimum.The equilibrium is a tuple (θ(D), θ(G)) that is locally minimizing for J(D) with respect to θ(D) and J(G) with respect to θ(G).Source: §3.1 · p. 18

The generator

• The generator can be implemented with a deep neural network and allows flexible placement or injection of noise, with differentiability as its principal requirement.To give pmodel full support over x, the latent dimension must be at least the dimension of x; GAN generators and VAEs also differ in which discrete variables standard backpropagation permits.Source: §3.1 · pp. 18, 20

The training process

• Training uses simultaneous stochastic gradient descent: each step samples minibatches of data x and latent values z, then updates θ(D) and θ(G) with one gradient step each.Adam is usually a good optimizer, and the author identifies one simultaneous step per player as the best practical protocol as of late 2016.Source: §3.1 · p. 20

3.2.1 The discriminator’s cost, J(D)

• GANs use the same discriminator cost across games: binary-classifier cross-entropy trained on real examples labeled 1 and generated examples labeled 0.The discriminator estimates a density ratio, enabling divergence and gradient computation, but this supervised approximation can suffer from overfitting and underfitting.Source: §3.2.1 · p. 21

3.2.2 Minimax

• The minimax game is theoretically analyzable and resembles Jensen-Shannon divergence minimization, but its function-space convergence results do not apply directly to neural-network parameter updates.The minimax generator cost is useful for theory but performs poorly in practice.Source: §3.2.2 · p. 22, §3.2.3 · p. 22

3.2.3 Heuristic, non-saturating game

• The non-saturating game flips the generator’s target, giving it a strong gradient when the discriminator confidently rejects generated samples.This heuristic game is no longer zero-sum and is motivated solely by preventing weak gradients for the losing player.Source: §3.2.3 · pp. 22–23

3.2.4 Maximum likelihood game

• With an optimal discriminator, a suitable GAN objective is equivalent in expectation to minimizing the data-to-model KL divergence, although sampling makes practical gradients stochastic.Other maximum-likelihood approximations within GANs are also possible, including Nowozin et al. (2016).Source: §3.2.4 · p. 23

3.2.5 Is the choice of divergence a distinguishing feature of GANs?

• Maximum-likelihood-trained GANs still produce sharp samples and select few modes, while GAN mode collapse often exceeds the limitation imposed by model capacity, so divergence choice alone does not explain either behavior.The tutorial concludes that mode selection likely reflects a training defect, while the cause of sharp samples remains unclear and may involve model families or approximation differences.Source: §3.2.5 · pp. 24–25, Figure 15 · p. 25

3.2.6 Comparison of cost functions

• The generator receives discriminator-mediated rewards and gradients, with a non-stationary reward function; non-saturating costs avoid the flat-gradient problem affecting minimax and maximum-likelihood games.The maximum-likelihood game concentrates most gradient on very few likely-real samples, whereas the non-saturating cost has lower sample variance.Source: §3.2.6 · pp. 25–26, Figure 16 · p. 26

Overview

• Most contemporary GANs are at least loosely based on DCGAN, a specific deep convolutional architecture despite earlier GANs also being deep and convolutional.DCGAN stands for “deep, convolution GAN.”Source: §3.3 · p. 27

Architecture

• DCGANs use batch normalization in most generator and discriminator layers, excluding the generator’s last and discriminator’s first layers to preserve learnable data-distribution mean and scale.The discriminator’s two minibatches are normalized separately.Source: §3.3 · p. 27
• The architecture largely follows all-convolutional networks, omitting pooling and unpooling while using transposed convolutions with stride greater than 1 to enlarge generator representations.This design is borrowed from Springenberg et al. (2015).Source: §3.3 · p. 27

Optimization

• DCGAN training uses the Adam optimizer rather than SGD with momentum.This is one of the architecture’s key implementation choices.Source: §3.3 · p. 27

Results

• DCGANs were the first GANs to generate high-resolution images in a single shot, producing high-quality bedroom images on restricted domains and meaningful latent-space arithmetic.Before DCGANs, LAPGANs were the only GANs reported to scale to high-resolution images and required multi-stage Laplacian-pyramid generation; DCGAN latent arithmetic corresponded to semantic attributes.Source: §3.3 · pp. 27–28, Figure 18 · p. 28

Connections to NCE and maximum likelihood

• GANs learn the generator sampler, whereas NCE learns the discriminator’s density model, producing substantially different gradients despite their shared value function.Maximum likelihood is closely related to NCE and uses the same minimax-game formulation with a heuristic generator update rather than gradient descent.Source: §3.4 · p. 28
• Figure 20 shows that GANs, NCE, and maximum likelihood differ in which component represents pmodel and how each player is updated.GANs use gradient descent for both players; maximum likelihood uses gradient descent for the discriminator and copies its learned density model into the generator; NCE leaves the generator as fixed noise.Source: §3.4 · p. 28, Figure 20 · p. 29

4 Tips and Tricks

• GAN tricks can help in some contexts and hurt in others, so practitioners should treat them as techniques worth trying rather than ironclad best practices.Source: §4 · p. 30

4.1 Train with labels

• Labels almost always improve subjective sample quality: class-conditional GANs outperform class-free GANs, while discriminator class recognition can suffice without generator class conditioning.These findings were reported by Denton et al. (2015) and Salimans et al. (2016); labeled models should only be compared with other labeled models.Source: §4.1 · p. 30

4.2 One-sided label smoothing

• One-sided label smoothing replaces real targets near 1 with values such as .9, discouraging extreme discriminator logits while leaving fake targets at 0.Smoothing fake labels can distort the optimal discriminator and reinforce spurious generator modes, so only real labels should be smoothed.Source: §4.2 · p. 31

4.3 Virtual batch normalization

• Virtual batch normalization reduces minibatch-induced correlations by using reference-batch statistics, allowing generated samples to remain independent at the cost of possible reference-batch overfitting.Batch-statistic fluctuations can dominate the input z, especially with small minibatches; virtual batch normalization computes statistics from each example together with a fixed reference batch.Source: §4.3 · pp. 32–33, Figure 21 · p. 32

4.4 Can one balance G and D?

• Balancing generator and discriminator strength has no compelling demonstrated recipe; extra discriminator steps usually do not clearly improve results, and vanishing or exploding gradients require reparameterization or smoothing instead.The discriminator may appropriately overpower the generator when estimating the density ratio, while generator capacity is often left smaller because current methods suffer from mode collapse.Source: §4.4 · p. 33

5.1 Non-convergence

• GANs lack a convergence theory for parameterized deep-network games, which can oscillate instead of reaching equilibrium and may benefit other game-based applications.Simultaneous gradient descent converges for some games, but the function-space proof for the minimax GAN game does not apply to parameter space.Source: §5.1 · p. 34

5.1.1 Mode collapse

• Mode collapse commonly produces repeated outputs or cycles among modes, limiting GAN applications; minibatch features reduce it, while unrolled GANs reduce mode dropping with even 10 or fewer discriminator steps.Minibatch GANs reduce collapse enough that counting, perspective, and global-structure defects become more apparent, while unrolled GANs have not yet scaled to ImageNet.Source: §5.1.1 · pp. 34, 35, 36, 38, Figure 22 · p. 35

5.1.2 Other games

• Improved convergence algorithms could benefit other games in AI, including AlphaGo, adversarial security, domain adaptation, privacy, and cryptography.The tutorial presents these as applications of general game-solving advances rather than an exhaustive list.Source: §5.1.2 · p. 42

5.2 Evaluation of generative models

• Generative models lack a clearly justified quantitative score because good likelihood and good samples can disagree, while GAN likelihood is especially difficult to estimate.Likelihood estimation for GANs is possible, as shown by Wu et al. (2016), but evaluation remains a major research area; Theis et al. (2015) describe broader difficulties.Source: §5.2 · p. 42

5.3 Discrete outputs

• GAN generators must be differentiable, preventing direct discrete outputs such as one-hot words and motivating REINFORCE, concrete or Gumbel-softmax methods, and continuous-to-discrete decoding.Removing this limitation could unlock GAN applications in NLP.Source: §5.3 · p. 42

5.4 Semi-supervised learning

• Feature matching GANs achieve state-of-the-art semi-supervised performance on MNIST, SVHN, and CIFAR-10 with 20 to 8,000 labels.Their discriminator adds a fake-image class to the real classes, allowing unlabeled real and generated examples to train real-versus-fake discrimination while labeled data trains classification; this approach was developed by Salimans et al. (2016) and Odena (2016).Source: §5.4 · p. 43

5.5 Using the code

• Using GAN latent codes remains difficult because inferring z from x lacks a fully general demonstrated encoder, while InfoGANs (Chen et al., 2016a) make code entries correspond to semantic attributes.Donahue et al. (2016) trained a deterministic encoder, and Dumoulin et al. (2016) trained an encoder sampling from a Gaussian posterior.Source: §5.5 · pp. 43–44

5.6 Developing connections to reinforcement learning

• GANs connect to actor-critic methods, inverse reinforcement learning, and imitation learning, suggesting continuing mutual benefits for GANs and reinforcement learning.The cited connections include Pfau and Vinyals (2016), Finn et al. (2016a), and Ho and Ermon (2016).Source: §5.6 · p. 44

• PPGNs dramatically improve the diversity of high-resolution images generated for ImageNet classes (Nguyen et al., 2016).Source: §6 · p. 44
• PPGNs are complicated and not yet well understood, with design recommendations based mostly on empirical observation rather than theoretical understanding.Source: §6 · p. 44
• PPGNs approximately sample images with a Langevin-based Markov chain whose gradients are estimated by a denoising autoencoder trained with several losses, including a GAN loss.Source: §6 · p. 44
• The GAN loss is crucial for obtaining high-quality images, as demonstrated by figure 34.Source: §6 · p. 44

7 Exercises

• The tutorial includes three exercises, with their solutions given in Section 8.Source: §7 · p. 44

7.1 The optimal discriminator strategy

• The first exercise asks readers to derive the optimal discriminator strategy in function space and state the assumptions required.It frames the discriminator as minimizing the objective in equation 8, with D(x) specified independently for each x.Source: §7.1 · p. 44

7.2 Gradient descent for games

• The second exercise analyzes whether a scalar minimax game has an equilibrium and solves the trajectory produced by continuous-time simultaneous gradient descent.The exercise treats the minimizing and maximizing players as controlling scalars x and y and uses infinitesimal learning rates.Source: §7.2 · p. 46

7.3 Maximum likelihood in the GAN framework

• The third exercise derives a GAN cost whose expected generator gradient matches that of D_KL(p_data∥p_model) when the discriminator is optimal.Readers determine the form of f in the proposed cost.Source: §7.3 · p. 46

8.1 The optimal discriminator strategy

• The optimal discriminator estimates the ratio of data density to the sum of data and model densities, guiding the generator toward underrepresented regions.Where D(G(z)) is large, model density is too low; where it is small, model density is too high.Source: §8.1 · p. 47, Figure 35 · p. 47

8.2 Gradient descent for games

• For V(x, y) = xy, x = y = 0 is an equilibrium, but simultaneous gradient descent circles it indefinitely or spirals outward rather than converging.Whether simultaneous gradient descent converges for GANs has no theoretical prediction, and guaranteed-convergent algorithms remain open problems.Source: §8.2 · pp. 48–49

8.3 Maximum likelihood in the GAN framework

• The discriminator’s density ratio can be used with importance sampling to match the expected generator gradient to the gradient of DKL(pdata∥pg), supporting calculations of multiple divergences.The construction assumes pg(x) ≥ 0 and permits exchanging differentiation and integration; with D(x) = σ(a(x)), a numerically stable implementation is f(x) = −exp(a(x)).Source: §8.3 · p. 51

9 Conclusion

• GANs use supervised ratio estimation to approximate intractable cost functions, including the KL divergence used for maximum likelihood estimation.This parallels Boltzmann machines’ use of Markov chains and VAEs’ use of the variational lower bound.Source: §9 · p. 51
• Training GANs requires finding Nash equilibria in high-dimensional, continuous, non-convex games, motivating better theory and training algorithms.Progress could improve applications beyond GANs.Source: §9 · p. 51
• GANs are crucial to many state-of-the-art image generation and manipulation systems and could enable many future applications.Source: §9 · p. 51