Source-linked AI summary
Generative Modeling using the Sliced Wasserstein Distance
Ishan Deshpande, Ziyu Zhang, Alexander Schwing
TL;DR
Generative modeling is difficult in high-dimensional spaces, and GAN training is often unstable, with mode dropping and challenging saddle-point optimization. The paper models distributions using random projections and the sliced Wasserstein distance, optionally using a discriminator for better projections. Across several datasets, the approach is reported as significantly more stable than conventional GANs and competitive in sample quality, while providing distance estimates and attainable-performance bounds.
Problem
Generative models must handle high-dimensional output spaces, while GANs remain difficult to optimize and can suffer from instability and mode dropping.
Method
The paper uses random projections to formulate generative modeling with the sliced Wasserstein distance, replacing the duality-based saddle-point approach with direct distance estimation.
Results
Across MNIST, Toronto faces, CIFAR-10, CelebA, and LSUN bedrooms, the approach is significantly more stable than conventional GANs and produces comparably good results.
Takeaways & Limitations
The formulation supports stable generative modeling, distance estimates, and a bound for attainable performance, with extensions to high-dimensional data.
Takeaways & Limitations
The reported experiments use at most 10,000 projected directions and sample sizes of at most 256, with the associated computation discussed as potentially worthwhile for improved stability.
Abstract
from arXiv · showhide
Generative Adversarial Nets (GANs) are very successful at modeling distributions from given samples, even in the high-dimensional case. However, their formulation is also known to be hard to optimize and often not stable. While this is particularly true for early GAN formulations, there has been significant empirically motivated and theoretically founded progress to improve stability, for instance, by using the Wasserstein distance rather than the Jenson-Shannon divergence. Here, we consider an alternative formulation for generative modeling based on random projections which, in its simplest form, results in a single objective rather than a saddle-point formulation. By augmenting this approach with a discriminator we improve its accuracy. We found our approach to be significantly more stable compared to even the improved Wasserstein GAN. Further, unlike the traditional GAN loss, the loss formulated in our method is a good measure of the actual distance between the distributions and, for the first time for GAN training, we are able to show estimates for the same.
1. Introduction
Generative modeling remains difficult in high-dimensional output spaces, and GAN optimization is often unstable because of its saddle-point formulation. This paper uses random projections and the sliced Wasserstein distance to obtain a more stable formulation with performance bounds and competitive sample quality.
- Motivation: High-dimensional output spaces make classical generative modeling difficult because they require sampling from complex distributions.The paper motivates function-approximator approaches that transform samples from simple distributions into desired outputs.
- Motivation: GANs produce sharp samples but are difficult to optimize, partly because their two-player game creates a saddle-point objective.Wasserstein GANs retain a saddle-point formulation, so their optimization remains challenging.
- Approach: Random projections yield a sliced Wasserstein formulation whose simplest version uses a single minimization instead of Kantorovich-Rubinstein duality.The sliced Wasserstein distance has also been applied to texture mixing, color transfer, and image classification.
- Approach: The proposed formulation provides a bound for attainable performance and extensions for modeling high-dimensional data beyond MNIST.These properties are presented as benefits beyond improved training stability.
- Results: Across MNIST, Toronto faces, CIFAR-10, CelebA, and LSUN bedrooms, the approach is significantly more stable than conventional GANs while producing comparably good results.The authors use these experiments to support stability and sample-quality claims.
2. Related Work
Related work frames GAN training as unstable and vulnerable to mode dropping, while Wasserstein-based methods address these issues but retain difficult discriminator optimization. The paper highlights computational and training-cost limitations in existing Wasserstein approaches.
- GAN limitations: GAN training is generally unstable because a well-trained discriminator can suppress generator learning.Task-specific tuning of generator updates has been proposed, but the passage says such efforts generalize poorly.
- GAN limitations: Mode dropping occurs when generated samples lack diversity, such as producing only a few of MNIST’s ten digits.The inverted Kullback-Leibler divergence is described as especially tolerant of missing modes and harsh toward novel samples.
- Wasserstein GAN: Wasserstein-1 distance was proposed as a replacement for Jensen-Shannon divergence to address these GAN problems.It is also known as the Earth mover’s distance.
- Wasserstein GAN: Wasserstein GANs use Kantorovich-Rubinstein duality and neural discriminators to approximate the distance through a constrained maximization.The discriminator is parameterized by weights that are clipped to enforce a Lipschitz condition.
- Limitations: Lipschitz-constraint enforcement remains open, while gradient clipping can converge slowly and exhibit high variance.The discriminator must be trained repeatedly because an inaccurate distance estimate can mislead the generator, increasing computation cost.
3. Approach
The approach directly estimates Wasserstein distance from samples using random one-dimensional projections, yielding a single-minimization training objective and an optional discriminator for more informative projections. It also provides an upper bound on the sliced Wasserstein distance and supports performance estimates, while introducing sorting-related computational costs.
- Random projections: The method estimates Wasserstein distance directly from samples by projecting high-dimensional data onto multiple one-dimensional subspaces and averaging the projected distances.The resulting metric is the sliced Wasserstein distance, which satisfies non-negativity, identity of indiscernibles, symmetry, and subadditivity.
- Training objective: Unlike the Kantorovich-Rubinstein formulation, the simplest proposed objective uses a single minimization rather than a saddle-point optimization.The paper motivates this formulation because non-convex, non-concave saddle-point problems are generally difficult to optimize.
- Random projections: In one dimension, sorting real and generated samples produces the optimal monotone assignment in O(|F| log |F|) time.The assignment pairs samples by rank, avoiding crossing pairings that would increase the matching cost.
- Training objective: Training samples random directions, projects real and generated distributions, computes projected Wasserstein distances, and differentiates their average for generator updates.The implementation replaces integration over all directions with a summation over randomly chosen unit vectors and can use stochastic gradient descent.
- Training objective as an upper bound: The training objective optimizes an upper bound on the sliced Wasserstein distance between the true and generated distributions.The bound becomes tighter as the relevant empirical-measure discrepancy tends to zero, according to the paper’s analysis.
- Scaling to high dimensional distributions: A discriminator can learn a feature space whose directions better separate real and fake samples, providing more informative projections in high-dimensional settings.This augments random projections because uniformly sampled directions may contain little useful information as training progresses.
4. Experimental Results
Experiments evaluate SWG across sample sizes, architectures, datasets, and training-time settings. The results indicate stable training, competitive sample quality, and a loss that tracks distributional quality and diversity.
- Effect of sample size: SWG estimates converge as sample size increases, with the empirical bound decreasing roughly at O(n^-1).The corresponding bound for the optimal generated distribution also decreases at O(n^-1).
- Effect of sample size: 128 samples suffice to train a fully connected MNIST generator that produces good images, while larger samples tighten the bound.Each sample-size configuration was trained five times, with small error bars indicating stable behavior.
- Stability of Training: SWG produces meaningful samples across all tested generator configurations, whereas WGAN fails in one configuration.The comparison uses identical hyperparameters across fully connected, convolutional, deconvolutional, and batch-normalized variants.
- Effectiveness of the sliced Wasserstein distance: SWG training is more stable than GAN and WGAN in the MNIST comparison, while its sliced Wasserstein distance reflects both sample quality and diversity.GAN training diverges and exhibits mode collapse around digit 1, whereas SWG produces good, diverse samples; SWG is also faster per generator update than WGAN because WGAN uses multiple discriminator updates.
- Stability of Training: A single default hyperparameter setting trains all tested architectures across CIFAR-10, LSUN Bedrooms, and CelebA.The experiments also include different discriminator architectures, with one discriminator update per generator update.
5. Conclusions
The paper proposes sliced Wasserstein distance for generative modeling and reports stable, competitive results across several image datasets. The implementation is publicly available.
- The paper uses the sliced Wasserstein distance for generative modeling.
- Experiments on MNIST, CIFAR-10, CelebA, and LSUN show stable results competitive with existing techniques.
- The implementation is publicly available.
A. Training objective as an upper bound
This section establishes an upper-bound analysis for the training objective by expressing sliced Wasserstein quantities through projected empirical samples and combining inequalities.
- Claim 1 assumes empirical measures formed from n i.i.d. samples of two distributions.
- The sliced Wasserstein distance is deconstructed using one-dimensional projections of the sample sets.
- The proof bounds the projected Wasserstein terms using an independent empirical copy and a cited theorem.
- Applying expectations and combining the intermediate equations completes the stated proof.
B. Bounds for generated distribution
This section derives bounds for the generated distribution by applying Claim 1 to empirical measures and rewriting the resulting expected sliced Wasserstein terms.
- Corollary 1 considers n-sample empirical measures and an independent copy of the data empirical measure.
- The corollary’s bound concerns the expected squared sliced Wasserstein distance between empirical data and generated measures.
- The proof derives the corollary from Claim 1 and rewrites the preceding expression using the resulting relation.
- The analysis states a minimization property for the generated distribution over Pf.
C. Discriminator update frequency experiments
The experiments test whether SWG remains robust when discriminator update frequency and training iterations vary across two discriminator configurations.
- The study varies discriminator updates per generator update and discriminator training iterations.
- The comparison uses two discriminator architectures with a DCGAN generator.
- Table 4 reports samples after 40 epochs on LSUN using sample size 64, learning rate 0.0005, and Adam.
- The four schemes include one discriminator update per generator update or per five generator updates, with one or five discriminator-training iterations.
- The table caption states that SWG is robust to different discriminator update schemes.
D. Network architectures for experiments on MNIST
This section summarizes the network architectures used for MNIST experiments and defines the notation for their layers and normalization.
- The MNIST experiments use separate generator and discriminator network architectures summarized in Table 5.The architectures are presented for the experiments in Sec. 4.2.
- “fc-n” denotes a fully connected layer with n output units.
- “conv2d-c-k-s” and “deconv2d-c-k-s” denote c convolutional filters of size k by k with stride s by s.
- “bn” denotes batch normalization.