PaperlayerView exact revision on arXiv

Source-linked AI summary

Wasserstein GAN

Martin Arjovsky, Soumith Chintala, Léon Bottou

arXiv:1701.07875v3stat.MLcs.LG

TL;DR

GAN training lacks a distribution distance that remains well-defined and supports continuous optimization for distributions on low-dimensional manifolds. WGAN minimizes an efficient approximation to the Earth Mover distance, yielding a sound optimization problem and more stable training with meaningful learning curves and reduced mode collapse.

  • Problem

    Common divergences such as KL can be undefined or infinite for distributions supported on low-dimensional manifolds, complicating continuous optimization.

  • Method

    WGAN minimizes an efficient approximation to the Earth Mover distance and theoretically establishes that the corresponding optimization problem is sound.

  • Results

    WGANs improve training stability, drastically reduce mode dropping, and provide learning curves that support debugging and hyperparameter searches.

  • Takeaways & Limitations

    WGAN learning curves can be used to monitor training and correlate with observed sample quality, offering a practical diagnostic for GAN development.

  • Takeaways & Limitations

    The loss metric is difficult to compare across critics and may not closely estimate the Earth Mover distance because critics have finite capacity.

Abstract

from arXiv · show

We introduce a new algorithm named WGAN, an alternative to traditional GAN training. In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches. Furthermore, we show that the corresponding optimization problem is sound, and provide extensive theoretical work highlighting the deep connections to other distances between distributions.

1 Introduction

The paper motivates replacing density-based objectives with a distance between distributions that remains meaningful for low-dimensional supports and supports continuous optimization. It introduces WGAN as an EM-distance-based GAN with theoretically sound optimization and empirical improvements to GAN training.

  • Motivation: When real data lie on low-dimensional manifolds, the model density may not exist and the KL distance can be undefined or infinite.The model and true-support manifolds are unlikely to have a non-negligible intersection.
  • Motivation: Adding high-bandwidth Gaussian noise can cover examples but degrades image-generation quality by producing blurry samples.Noise terms are therefore common in classical generative models despite this sample-quality limitation.
  • Motivation: The paper studies how probability distances affect convergence and parameter-to-distribution continuity, noting that weaker distances make continuous model mappings easier to define.The relevant convergence notion depends on the distance used between distributions.
  • Contributions: WGAN minimizes a reasonable, efficient approximation of the Earth Mover distance, and the paper theoretically establishes that its optimization problem is sound.The paper also provides a comprehensive comparison of Earth Mover distance with popular probability distances and divergences.
  • Contributions: Empirically, WGANs reduce major GAN training problems, including mode dropping, while removing the need for careful discriminator-generator balance and delicate architecture design.They also enable continuous estimation of the Earth Mover distance during training.

2 Different Distances

The section contrasts distributional distances and shows that the Earth-Mover distance remains continuous and useful for optimization in settings where KL, JS, and TV fail. Its weaker topology supports convergence for distributions on low-dimensional manifolds and motivates practical Wasserstein optimization.

  • Distance definitions: The Earth-Mover (EM) distance is the optimal transport cost, where joint distributions specify how much mass moves between points to transform Pr into Pg.The coupling γ(x, y) has marginals Pr and Pg.
  • Learning parallel lines: For parallel lines Pθ approaching P0, EM converges while JS, KL, reverse KL, and TV do not converge.When θt →0, the sequence converges under EM but not under the other listed divergences.
  • Learning parallel lines: EM enables gradient-based learning because its loss is continuous, whereas the other distances produce a non-continuous loss in the low-dimensional-manifold example.The EM curve provides a usable gradient, while the JS curve does not.
  • Theoretical guarantees: Under local Lipschitzness and regularity assumption 1, W(Pr, Pθ) is continuous everywhere and differentiable almost everywhere.The corresponding continuity and differentiability statements are false for JS and all KL divergences.
  • Relative topologies: The induced topologies rank KL strongest, followed by JS and TV, with EM weakest; EM therefore remains sensible for distributions supported on low-dimensional manifolds.This motivates introducing a practical approximation for optimizing the EM distance.

3 Wasserstein GAN

WGAN replaces the intractable Wasserstein distance optimization with a parameterized family of Lipschitz critics, enabling principled estimation and gradient-based generator updates under an optimality assumption. Training the critic toward optimality yields more reliable gradients, avoids mode collapse, and motivates practical alternatives to weight clipping.

  • Lipschitz critics: A K-Lipschitz parameterized critic family estimates W(Pr, Pθ) up to a multiplicative constant through the Kantorovich-Rubinstein dual formulation.The critic family is constrained by a shared Lipschitz constant K.
  • Lipschitz critics: Under attainment of the dual supremum, backpropagating through the critic provides a principled Wasserstein gradient estimate for updating the generator.The gradient is estimated using Ez∼p(z)[∇θfw(gθ(z))].
  • Lipschitz critics: A neural critic with parameters in a compact space W can be trained and differentiated like a typical GAN while remaining K-Lipschitz for some K.The shared Lipschitz bound depends on W rather than individual weights.
  • Lipschitz critics: Weight clipping is a poor Lipschitz constraint: large clipping slows critic optimization, while small clipping can cause vanishing gradients in deep or unnormalized networks.The paper reports little difference from simple variants such as projecting weights to a sphere.
  • Training behavior: Because the Wasserstein distance is continuous and differentiable almost everywhere, training the critic toward optimality produces increasingly reliable gradients and prevents mode collapse.Mode collapse is attributed to the optimal generator for a fixed discriminator concentrating on points assigned the highest values.

4 Empirical Results

WGAN experiments on LSUN-Bedrooms show two main practical benefits: a loss metric that tracks convergence and sample quality, and improved optimization stability. The method also remains effective across generator architectures, although momentum optimizers and high learning rates can destabilize training.

  • Meaningful loss metric: WGAN’s loss estimate of the EM distance correlates with generated-sample quality across convolutional and 4-layer ReLU-MLP architectures.The critic is trained relatively well before each generator update, making the loss an EM-distance estimate up to constraint-dependent constant factors.
  • Limitations: The WGAN loss metric is useful for validating experiments but is not yet a quantitative generative-model evaluation method.Critic-dependent scaling and finite critic capacity make cross-critic comparison and proximity to the EM distance difficult to determine.
  • Meaningful loss metric: Unlike the WGAN estimate, the standard GAN’s JS estimate correlates poorly with sample quality and often saturates near log 2 ≈0.69.The JS estimate can remain constant or increase while samples improve or collapse.
  • Optimization stability: Training can become unstable with momentum-based critic optimizers such as Adam when β1 > 0, or with high learning rates.The authors identify critic-loss nonstationarity and frequently negative cosine similarity between the Adam step and gradient as a potential cause.
  • Optimization stability: WGAN training showed no evidence of mode collapse in the reported experiments and was more robust than standard GANs to generator architecture changes.It produced samples with a generator lacking batch normalization and constant filters, and with a 4-layer ReLU-MLP, where standard GANs performed poorly.

5 Related Work

The paper situates WGAN within integral probability metrics, showing that different function classes yield substantially different distances and optimization properties. It relates these choices to Wasserstein distance, total variation, EBGANs, MMD/GMMNs, and prior Wasserstein-based methods.

  • Integral Probability Metrics: Integral probability metrics are pseudometrics when their function class is closed under negation, but different classes can produce radically different metrics.The pseudometric properties include nonnegativity, symmetry, and the triangle inequality.
  • Wasserstein Distance: The Kantorovich-Rubinstein duality identifies the Wasserstein distance with the IPM over 1-Lipschitz functions, while K-Lipschitz functions scale it by K.Specifically, W(Pr, Pθ) = dF(Pr, Pθ) for 1-Lipschitz functions and K · W(Pr, Pθ) = dF(Pr, Pθ) for K-Lipschitz functions.
  • Total Variation and EBGANs: Using functions bounded between -1 and 1 recovers total variation distance, whose topology and loss regularity differ drastically from those of the Wasserstein distance.The paper connects EBGANs to total variation distance; because they share JS-like regularity, EBGANs face imperfect-gradient problems when the discriminator is not optimized fully.
  • MMD and GMMNs: MMD is an IPM over a bounded RKHS function class and becomes a proper metric with a universal kernel, but its standard evaluation cost grows quadratically with sample count.The kernel trick avoids training a separate maximizing network, while quadratic cost limits scalability; linear-cost estimates exist but have worse sample complexity.
  • MMD and GMMNs: GMMNs directly optimize MMD through the kernelized objective without a separate maximizing network, yet their applicability is limited in part by quadratic minibatch costs.The original GMMN paper used minibatches of size 1000, substantially larger than the standard 32 or 64, despite the quadratic computational cost.
  • Other Wasserstein Applications: Related work applies Wasserstein distances to learning in discrete Restricted Boltzmann Machines and develops algorithms for calculating Wasserstein distances between distributions.The latter direction is presented as potentially useful for evaluating generative models.

6 Conclusion

The paper introduces WGAN as an alternative to traditional GAN training, aiming to improve learning stability and address mode collapse. It also establishes a sound optimization problem and develops theory connecting WGAN to other distribution distances.

  • WGAN is introduced as an alternative algorithm for traditional GAN training.
  • The model improves learning stability and addresses problems such as mode collapse.
  • WGAN provides meaningful learning curves for debugging and hyperparameter searches.
  • The corresponding optimization problem is sound, with theoretical connections to other distances between distributions.

A Why Wasserstein is indeed weak

The section contrasts total variation’s strong norm topology with Wasserstein’s weak* topology on probability distributions. Total variation is identified with the dual-space norm, making it substantially stronger than Wasserstein distance.

  • A Why Wasserstein is indeed weak: Total variation on Prob(X) is exactly the norm distance on Cb(X)∗ via the Riesz Representation theorem.The mapping Φ(P)(f) := E_x∼P[f(x)] is an isometric immersion.
  • A Why Wasserstein is indeed weak: Probability measures are represented as a subset of the dual space Cb(X)∗, where total variation corresponds to the dual norm.This identification follows from the isometric immersion supplied by the Riesz Representation theorem.
  • A Why Wasserstein is indeed weak: Because total variation induces the strong norm topology, few parameterizations θ 7→Pθ are expected to be continuous under this distance.The section notes that the norm topology is very strong and previews equality with the Jensen-Shannon topology in Theorem 2.
  • A Why Wasserstein is indeed weak: For probability distributions, total variation gives the strong topology, whereas Wasserstein distance gives the weaker weak* topology.Dual spaces such as Cb(X)∗ carry both strong norm and weak* topologies.

B Assumption definitions

Assumption 1 requires the generator g to be locally Lipschitz between finite-dimensional vector spaces, with evaluations denoted g_θ(z) and local Lipschitz constants defined relative to a probability distribution p over Z.

  • Assumption 1: Assumption 1 treats g : Z × R^d → X as locally Lipschitz between finite-dimensional vector spaces and introduces the notation g_θ(z) for evaluation at (z, θ).For a probability distribution p over Z, the assumption further requires local Lipschitz constants L(θ, z).

C Proofs of things

The proofs establish continuity and almost-everywhere differentiability properties of the Wasserstein objective, characterize relationships among distributional distances, and justify its gradient formula under stated regularity and integrability conditions.

  • Theorem 1: A coupling of (g_θ(Z), g_θ′(Z)) bounds W(P_θ, P_θ′), providing the core argument for Theorem 1.The coupling γ belongs to Π(P_θ, P_θ′).
  • Theorem 1: Continuity of g in θ yields continuity of W(P_r, P_θ) through pointwise convergence, compactness, uniform boundedness, and bounded convergence.Compactness supplies a uniform bound M on distances between elements of X.
  • Theorem 1: Local Lipschitzness of g makes W(P_r, P_θ) locally Lipschitz, hence everywhere continuous and differentiable almost everywhere.The differentiability conclusion follows from Rademacher’s theorem.
  • Theorem 2: Theorem 2 proves convergence implications between δ, JS, and W, while recalling that W metrizes weak* convergence on probability measures over compact X.The proof uses mixture distributions, Radon–Nikodym derivatives, Pinsker’s inequality, and the triangular inequality.
  • Theorem 3: For Theorem 3, compactness ensures a 1-Lipschitz Kantorovich–Rubenstein optimizer exists, enabling an envelope-theorem gradient expression.The optimizer set X*(θ) is non-empty for every θ.
  • Theorem 3: Rademacher, Fubini, and dominated convergence establish the gradient expression almost everywhere when the relevant terms are well-defined and the integrability assumption holds.The proof also establishes existence of the differentiated expectation itself.

D Energy-based GANs optimize total variation

Under an optimal nonnegative discriminator, energy-based GANs optimize the total variation distance between real and generated distributions. An optimal discriminator exists, and the generator objective is proportional to this distance, reaching its minimum exactly when the distributions match.

  • EBGANs optimize the total variation distance between real and generated distributions under an optimal discriminator.
  • An optimal discriminator exists for any real and generated distributions under the stated measurable-function formulation.Theorem 4 states existence when the real distribution is over a compact space and the generator is measurable.
  • The generator loss under the optimal discriminator is proportional to total variation distance.
  • The generator objective is minimized exactly when total variation is minimized at 0, equivalently when the real and generated distributions are equal.

E Generator’s cost during normal GAN training

During normal GAN training, generator cost does not track sample quality consistently. Error increases for MLP and DCGAN generators, while an MLP generator–discriminator pair shows oscillating cost regardless of sample quality.

  • Both MLP and DCGAN generator curves have increasing error during normal GAN training.
  • For the DCGAN generator, samples improve while generator cost increases, indicating no significant correlation between sample quality and loss.
  • With both an MLP generator and discriminator, the cost curve goes up and down regardless of sample quality.

F Sheets of samples

The section presents WGAN and standard GAN sample procedures across three generator architectures, paired with DCGAN critics or discriminators. The configurations include DCGAN generators, batch-normalization-free constant-filter generators, and a four-hidden-layer MLP generator.

  • WGAN uses DCGAN generators and critics, while the corresponding standard GAN uses DCGAN generators and discriminators.
  • WGAN uses a DCGAN generator without batchnorm and with constant filter size alongside a DCGAN critic; standard GAN uses the same generator with a DCGAN discriminator.
  • WGAN uses an MLP generator with 4 hidden layers of 512 units and a DCGAN critic, while standard GAN uses the same MLP generator with a DCGAN discriminator.
Loading 1701.07875v3…