Source-linked AI summary
From GAN to WGAN
Lilian Weng
TL;DR
GAN training can be unstable, and its JS-based loss does not provide a meaningful value when the real and generated distributions are disjoint. The paper presents Wasserstein distance as a minimum-cost transport metric for GANs; it remains meaningful and smooth for non-overlapping low-dimensional distributions, although weight clipping leaves training challenges.
Problem
Method
The paper presents Wasserstein distance as the minimum cost among joint-probability transport plans for moving probability mass between distributions.
Source: §6.1 · p. 9Results
Wasserstein distance remains a meaningful, smooth measure of separation between distributions on non-overlapping low-dimensional manifolds.
Source: §6.2 · p. 9Takeaways & Limitations
The Wasserstein metric is presented as a smoother distance measure for GAN training, but weight clipping can still cause instability, slow convergence, or vanishing gradients.
Source: §6.2 · p. 9, §6.3.2 · p. 11Weight clipping is described as a poor way to enforce the Lipschitz constraint; large clipping windows can slow convergence, while small ones can cause vanishing gradients.
Source: §6.3.2 · p. 11
Abstract
from arXiv · showhide
This paper explains the math behind a generative adversarial network (GAN) model and why it is hard to be trained. Wasserstein GAN is intended to improve GANs' training by adopting a smooth metric for measuring the distance between two probability distributions.
1 Introduction
GANs use competing generator and discriminator models to produce realistic content, but training can be unstable or fail to converge. The paper explains this difficulty and introduces a modified GAN intended to address it.
1 Introduction
- GANs have produced results across image, language, and music generation, but training can be unstable or fail to converge.The paper frames its contribution as explaining the framework's mathematics and training difficulty before introducing a modified GAN intended to address it.Source: §1 · p. 1
2 Kullback–Leibler and Jensen–Shannon Divergence
KL divergence is asymmetric, while JS divergence is symmetric and smoother. The section reviews these metrics for comparing probability distributions and notes a proposed connection between JS and GAN success.
2 Kullback–Leibler and Jensen–Shannon Divergence
- KL divergence is asymmetric, and when p(x) is near zero while q(x) is nonzero, q's effect is disregarded, potentially skewing comparisons of equally important distributions.KL divergence measures how one distribution p diverges from an expected distribution q.Source: §2 · p. 1
- JS divergence is symmetric and smoother than KL divergence; Figure 1 illustrates both metrics using Gaussian distributions with means 0 and 1.The figure labels their average as m = (p + q)/2.Source: §2 · p. 1, Figure 1 · p. 2
- Some researchers [2] attribute part of GANs' success to replacing the traditional maximum-likelihood approach's asymmetric KL divergence with symmetric JS divergence.The paper states that it will discuss this proposed explanation in the next section.Source: §2 · p. 1
3 Generative Adversarial Network
A GAN trains a generator and discriminator in opposition, with the discriminator's optimal value determined by the real and generated distributions. At the global optimum, the distributions match, and the optimal-discriminator loss represents their JS divergence.
3 Generative Adversarial Network
- The GAN generator produces synthetic samples from noise to approximate real data, while the discriminator estimates whether samples come from the real dataset.They compete during training: the generator seeks to fool the discriminator, which tries not to be fooled.Source: §3 · pp. 2–3
3.1 What is the Optimal Value for D?
- The optimal discriminator is D*(x) = pr(x)/(pr(x) + pg(x)); when the generated and real distributions match, D*(x) becomes 1/2.The discriminator's objective is to maximize the GAN loss for a given generator.Source: §3.1 · p. 3
3.2 What is the Global Optimal?
- When both generator G and discriminator D are at their optimal values, the generated distribution pg equals the data distribution pr, and the discriminator outputs D*(x) = 1/2.Source: §3.2 · p. 4
3.3 What does the Loss Function Represent?
- When the discriminator is optimal, GAN loss quantifies the similarity between generated and real data distributions through JS divergence, reaching -2 log 2 when they match.The section presents this minimum as aligned with the preceding equations.Source: §3.3 · p. 4
4 Problems in GANs
GAN training is difficult because simultaneous game updates may not converge, while data supports and discriminator behavior create further optimization problems. The section describes instability, vanishing gradients, mode collapse, and the lack of a useful evaluation metric.
4.1 Hard to Achieve Nash Equilibrium
- Independent cost updates in the two-player game do not guarantee convergence; simultaneous gradient steps in the example produce worsening oscillations.The example updates x to minimize xy and y to minimize −xy.Source: §4.1 · p. 4, Figure 3 · p. 5
4.2 Low Dimensional Supports
- Because real and generated data lie on low-dimensional manifolds, their supports are almost certainly disjoint, allowing a discriminator to separate them perfectly.The passage notes that a 100-dimensional noise input can define outputs over 4096 image pixels.Source: §4.2 · p. 5, Figure 4 · p. 6
4.3 Vanishing Gradient
- 5 orders of magnitude: gradient norms decayed in the best case after 4,000 discriminator iterations, illustrating the strong-discriminator side of GANs’ dilemma.A poor discriminator gives the generator inaccurate feedback, while a strong one drives gradients close to zero and can slow or jam learning.Source: §4.3 · p. 5, Figure 5 · p. 6
4.4 Mode Collapse
- Mode collapse occurs when the generator repeatedly produces the same outputs, failing to represent the complex real-world distribution or its variety.Source: §4.4 · p. 6
4.5 Lack of a Proper Evaluation Metric
- GANs lack a good evaluation metric, leaving no reliable sign for when to stop training or compare model performance.Source: §4.5 · p. 6
5 Improved GAN Training
The section lists training suggestions, with the first five presented as practical faster-convergence techniques [3] and the final two proposed for disjoint distributions [4].
Feature Matching
- Feature matching optimizes the discriminator to compare statistics of real and generated samples, such as feature means or medians.Its loss is the squared distance between the expected feature statistics for real and generated data.Source: §5 · p. 7
Minibatch Discrimination
- Minibatch discrimination lets the discriminator use relationships among samples in a batch rather than process each sample independently.It summarizes each sample’s closeness to other batch members and adds that summary to the model input.Source: §5 · p. 7
Training Stability
- An added parameter-history penalty slows training when model parameters change too dramatically over time.Source: §5 · p. 7
One-sided Label Smoothing
- For the discriminator, replacing hard labels 1 and 0 with softened values such as 0.9 and 0.1 is shown to reduce the networks’ vulnerability.Source: §5 · p. 7
Virtual Batch Normalization
- Virtual batch normalization normalizes each sample against a fixed reference batch chosen at the start of training.The same reference batch remains fixed throughout training.Source: §5 · p. 7
Adding Noise
- Adding continuous noise to discriminator inputs aims to spread out the distributions and increase the chance that they overlap.Source: §5 · p. 7
Use Better Metric of Distribution Similarity
- The vanilla GAN’s JS-divergence loss is uninformative for disjoint distributions, whereas the Wasserstein metric provides a smoother value space.Source: §5 · pp. 7–8
6 Wasserstein GAN (WGAN)
Wasserstein distance treats distribution mismatch as the minimum expected transport cost over joint plans. The section argues that its smooth value on disjoint supports motivates a dual-form WGAN objective evaluated by a Lipschitz-constrained critic.
6.1 What is Wasserstein Distance?
- Wasserstein distance is the minimum expected transport cost among joint plans, with each plan's cost reflecting the amount of dirt moved times its travel distance.The paper calls this Earth Mover’s distance.Source: §6.1 · pp. 8–9
6.2 Why Wasserstein is better than JS or KL Divergence?
- On disjoint low-dimensional supports, Wasserstein remains a meaningful smooth distance, while KL is infinite and JS jumps nondifferentiably at θ = 0.Source: §6.2 · pp. 9–10
6.3 Use Wasserstein Distance as GAN Loss Function
- Because enumerating all joint distributions is intractable, the paper invokes Kantorovich–Rubinstein duality to recast the distance as a supremum.Source: §6.3 · p. 10
6.3.1 Lipschitz Continuity
- The dual constrains f to be K-Lipschitz; continuously differentiable functions are Lipschitz here, but Lipschitz functions need not be differentiable everywhere.Source: §6.3.1 · p. 10
6.3.2 Wasserstein Loss Function
- WGAN uses a K-Lipschitz critic to estimate Wasserstein distance and clamps its weights after updates, replacing the vanilla GAN loss with a Wasserstein-derived loss.The source reports unstable training, slow convergence with large clipping windows, and vanishing gradients with small ones.Source: §6.3.2 · pp. 10–11