Source-linked AI summary
Wasserstein Divergence for GANs
Jiqing Wu, Zhiwu Huang, Janine Thoma, Dinesh Acharya, Luc Van Gool
TL;DR
GAN training requires reliable distribution comparison, but approximating the k-Lipschitz constraint in W-met is difficult. The paper introduces W-div, a relaxed Wasserstein formulation, and WGAN-div as its GAN objective. WGAN-div is reported as stable across settings and superior quantitatively and qualitatively on image-synthesis benchmarks.
Problem
GANs lack a reliable fake–real distribution measure, while enforcing the k-Lipschitz constraint required by Wasserstein objectives is difficult and can narrow the effective function space.
Method
The paper introduces W-div as a relaxed Wasserstein divergence and incorporates its directly usable objective into GANs as WGAN-div.
Results
WGAN-div is reported stable across settings and superior to compared methods quantitatively and qualitatively, including an FID score of 15.9 on LSUN, 4.4 lower than CTGAN.
Takeaways & Limitations
W-div provides a relaxed alternative to strict Lipschitz-constrained Wasserstein objectives, enabling a practical GAN formulation that faithfully approximates the divergence through optimization.
Takeaways & Limitations
The gradient-penalty-like objective corresponding to Eq. 6 is not generally a divergence and is not a valid metric.
Abstract
from arXiv · showhide
In many domains of computer vision, generative adversarial networks (GANs) have achieved great success, among which the family of Wasserstein GANs (WGANs) is considered to be state-of-the-art due to the theoretical contributions and competitive qualitative performance. However, it is very challenging to approximate the $k$-Lipschitz constraint required by the Wasserstein-1 metric~(W-met). In this paper, we propose a novel Wasserstein divergence~(W-div), which is a relaxed version of W-met and does not require the $k$-Lipschitz constraint. As a concrete application, we introduce a Wasserstein divergence objective for GANs~(WGAN-div), which can faithfully approximate W-div through optimization. Under various settings, including progressive growing training, we demonstrate the stability of the proposed WGAN-div owing to its theoretical and practical advantages over WGANs. Also, we study the quantitative and visual performance of WGAN-div on standard image synthesis benchmarks of computer vision, showing the superior performance of WGAN-div compared to the state-of-the-art methods.
1 Introduction
GANs can generate visually pleasing samples but lack a reliable measure of fake–real distribution differences, making training unstable. WGANs address this with Wasserstein objectives, yet approximating the required k-Lipschitz constraint remains difficult; this paper proposes W-div and WGAN-div as a relaxed alternative.
- Motivation: GANs formulate generation as a zero-sum game, but lack a reliable way to measure differences between fake and real data distributions.This limitation is associated with unstable training despite visually pleasing samples.
- Motivation: W-met is considered more sensible than JS or KL divergence for distributions supported by low-dimensional manifolds.
- Problem: The dual W-met formulation requires a k-Lipschitz constraint that is strict and non-trivial to approximate.Existing approximation methods generally provide unsatisfying regularization for this constraint.
- Related work: Prior work explored gradient regularization, deep architectures, and progressive growing to improve stability and high-resolution image synthesis.
- Contribution: The paper introduces W-div, a relaxed version of W-met, and incorporates it into GAN training as WGAN-div.
- Contribution: W-div is presented as a symmetric divergence, while WGAN-div is reported to approximate it through optimization and achieve stable, superior quantitative and qualitative results.
2 Background
GANs use adversarial objectives to distinguish real from synthetic data, while Wasserstein formulations target more stable distribution comparison. Existing methods approximate the Lipschitz constraint through weight or gradient controls, but these approaches can restrict the discriminator function space or remain difficult to guarantee in practice.
- GAN background: GANs train a generator and discriminator in an adversarial game, with the discriminator distinguishing real images from synthetic outputs.
- GAN background: The original GAN objective can be interpreted as JS divergence up to a constant and uses a discriminative function in a min-max problem.
- Wasserstein GANs: WGANs use the Wasserstein-1 metric because unstable GAN training is associated with gradient vanishing, and W-met is considered sensible for low-dimensional data manifolds.
- Lipschitz approximations: Weight clipping imposes a hard threshold on discriminator weights, but it can produce oversimplified functions; spectral normalization instead normalizes each layer's weights by its L2 matrix norm.
- Lipschitz approximations: Gradient penalties constrain gradients along lines between real and fake samples, but finite iterations and limited samples make the k-Lipschitz guarantee difficult.
- Related work: Consistency regularization, progressive growing, and f-divergence-based regularization provide alternative approaches to stability, dimensional mismatch, and high-resolution training.
- Related work: Information-geometry work connects Wasserstein distance with KL divergence by entropy regularization, which introduces a divergence and related geometric structures.
3 Proposed Method
The paper introduces W-div as a symmetric divergence that relaxes W-met’s strict Lipschitz requirement, then derives WGAN-div as an optimizable GAN objective. The objective is designed to approximate W-div while supporting neural-network parameterization and stable training.
- Wasserstein divergence: W-div is introduced as a symmetric divergence connected to the Wasserstein metric.The paper also relates W-div to an upper-bound construction involving the optimal W-met function.
- Wasserstein divergence: The proposed function space is less restrictive than the k-Lipschitz constraint because gradient norms need not obey a hard threshold k.The smooth compact-support function space can be parameterized by differentiable neural networks.
- Wasserstein divergence: The paper excludes a related construction because its corresponding quantity is not generally a divergence, and a specific case is neither a divergence nor a valid metric.The counterexample uses Ω = (−1, 1) and p = 2.
- Wasserstein divergence GANs: WGAN-div replaces the directly unusable W-met objective with a Wasserstein-divergence objective that can be optimized without imposing the strict 1-Lipschitz constraint.The discriminator parameterizes the function and the generator parameterizes the fake distribution; interpolated real and fake samples are used in training.
- Wasserstein divergence GANs: Minimizing L_DIV faithfully approximates W-div, so decreasing L_DIV indicates a better approximation of the target divergence.This contrasts with L_GP, whose decrease does not necessarily imply a better approximation of W1.
- Wasserstein divergence GANs: The WGAN-div algorithm trains generator and discriminator networks using sampled real data, Gaussian noise, interpolated points, and updates controlled by p and k.The default implementation uses Adam and studies the power p and coefficient k as gradient-term hyperparameters.
4 Experiments
The experiments evaluate WGAN-div on toy and standard image datasets, under standard and progressive-growing training, using FID and visual comparisons. WGAN-div shows stable training, strong FID results, and competitive qualitative performance across settings.
- Evaluation setup: WGAN-div is evaluated on Swiss roll, Gaussian mixtures, CIFAR-10, CelebA, and LSUN, with FID as the major image-quality criterion.The comparisons use default architectures and hyperparameters from the respective methods.
- Standard training: WGAN-div reaches the best FID scores among the compared approaches, with clearer improvements on CelebA and LSUN than on CIFAR-10.On LSUN, WGAN-div achieves an FID score of 15.9, 4.4 lower than CTGAN.
- Standard training: On CelebA, WGAN-div achieves FID scores of 15.2 with ResNet, 18.6 with ResNet without BN, 17.5 with ConvNet, and 21.5 with ConvNet without BN.These results compare WGAN-div against WGAN-GP and RJS-GAN across two architectures and batch-normalization settings.
- Training behavior: The WGAN-div training process is comparatively stable, converges fast, and reaches top FID scores with less than 60K iterations.The discriminator cost also shows a meaningful correlation with image quality.
- Progressive growing: In progressive growing, PGGAN-div slightly improves FID scores over PGGAN at higher resolutions and produces competitive 256 × 256 visual results on CelebA-HQ and LSUN.For CelebA-HQ, FID scores are not reported because artificial post-processing introduces distribution shift.
5 Conclusion
The paper concludes that Wasserstein divergence removes the 1-Lipschitz requirement and yields WGAN-div as a practical GAN objective. FID and qualitative evaluations report stable and superior performance compared with state-of-the-art methods.
- The proposed Wasserstein divergence does not require the 1-Lipschitz constraint.
- Equipping GANs with the Wasserstein divergence objective produces WGAN-div.
- FID scores and qualitative evaluations demonstrate the stability and superiority of WGAN-div over state-of-the-art methods.
A Proof of Theorem 1
The proof establishes that the proposed Wasserstein divergence is zero exactly when the probability measures agree and is symmetric.
- When Pr = Pg almost everywhere, the first two terms of Eq. (11) vanish and the divergence equals zero.
- When Pr ≠ Pg, there exists a function yielding a strictly negative value, supporting the converse identity claim.
- The proof derives symmetry by taking the infimum over all admissible functions.
B Study of Sampling Strategies
The sampling-strategy study compares six ways of constructing Pu and reports that WGAN-div consistently outperforms the compared methods.
- The study evaluates six sampling strategies for Pu, including real–fake combinations, same-source combinations, real-only, and fake-only sampling.
- Across the evaluated sampling strategies, WGAN-div consistently outperforms the compared methods.
C Interpolation Results
Figure 5 presents interpolation results for 64 × 64 images from CelebA and LSUN.
- Figure 5 presents interpolation results for 64 × 64 CelebA images.
- Figure 5 presents interpolation results for 64 × 64 LSUN images.
- The figure covers two image-synthesis datasets at the 64 × 64 resolution.