Max-Sliced Wasserstein Distance and its use for GANs

1. Introduction

• High-dimensional sample sparsity motivates projecting onto low-dimensional subspaces, where distances between projected samples can better reflect true distance.The paper frames this as a way to mitigate artificial distances between point collections in high-dimensional space.Source: §1 · p. 1
• Max-sliced Wasserstein addresses projection complexity by maximizing projected distance over directions, and the paper proves it generalizable for Gaussian distributions.The paper presents it as a best-direction alternative to sliced Wasserstein’s random projections.Source: §1 · p. 2
• Using max-sliced Wasserstein, the paper trains GANs on high-resolution CelebA-HQ [16] and LSUN Bedrooms [36] images and reports improved performance in unpaired word translation [6].The paper also reports a significant reduction in the number of projection directions required for sliced-Wasserstein GANs.Source: §1 · p. 2
• The paper proves that, for a certain distribution class, Wasserstein distance has exponential sample complexity while sliced Wasserstein distance has polynomial sample complexity.For Gaussian distributions, sliced Wasserstein is generalizable while Wasserstein is not, partially explaining the better behavior reported for sliced-Wasserstein GANs.Source: §1 · pp. 1–2
• The paper studies sliced Wasserstein’s projection complexity: how the number of random directions affects estimation, since random directions can lose information.Its sliced Wasserstein analysis concerns averages of projected Wasserstein distances along randomly chosen directions [8] [20] [34].Source: §1 · pp. 1–2

• GANs recast generative modeling as finding a generated distribution that minimizes a distance to the data distribution, rather than maximizing likelihood.Source: §2 · p. 2
• Arjovsky et al. proposed Wasserstein distance for GANs; its estimate can use the Wasserstein-1 duality over 1-Lipschitz functions represented by a deep net.The Wasserstein-p distance is defined using joint distributions with the data and generated distributions as marginals.Source: §2 · p. 2
• Wasserstein estimates can be unstable under the curse of dimensionality [33], while sliced Wasserstein reduces estimation to one-dimensional distances and is more efficient.Sliced Wasserstein is a true metric, satisfying non-negativity, identity of indiscernibles, symmetry, and subadditivity [19].Source: §2 · p. 2
• In practice, sliced Wasserstein-2 replaces integration over directions with a sum over finitely many normalized random Gaussian directions, and computes one-dimensional transport by sorting.Source: §2 · p. 2
• The sliced-distance generator can operate in an adversarially learned feature space, where a discriminator classifies real and generated data.The feature space is parameterized by the discriminator, and the generator minimizes the distance between real and generated features; this approach was reported to work well on high-dimensional datasets.Source: §2 · p. 3

3.1. Sample complexity of the Wasserstein and sliced Wasserstein distances

• For the considered Gaussian family, sliced Wasserstein-2 is P-generalizable while Wasserstein-2 is not.P-generalizability means polynomial sample size in dimension and accuracy suffices for the empirical distance to approximate the population distance with high probability; under certain conditions, this can provide GAN training with a more accurate signal at fixed computational budget.Source: §3.1 · p. 3

3.2. Projection complexity of the Sliced Wasserstein Distance

• In high dimensions, random unit-norm projections are usually nearly orthogonal to the mean-shift direction, making sliced Wasserstein updates small and convergence slow.The paper reports that convergence slows further as dimension increases; using the important direction as the single projection instead yields fast convergence in the same mean-learning example.Source: §3.2 · p. 4, Figure 1 · p. 4

3.3. Max sliced Wasserstein distance

• Max-sliced Wasserstein distance compares distributions along the most meaningful projection direction, defining a proper distance that addresses projection complexity.The authors prove it retains the sliced Wasserstein distance’s sample-complexity benefit, while requiring estimation of a maximum.Source: §3 · p. 3, §3.3 · p. 4

13 end

• The max-sliced Wasserstein-2 distance is a well-defined distance and is P-generalizable.The paper says its validity permits using it directly for learning distributions.Source: §13 · p. 5

3.4. max-sliced GAN

• The method reduces the number of projection directions needed compared with [8] by using the max-sliced Wasserstein distance in a discriminator-learned feature space.The feature space comes from the discriminator’s penultimate layer, and the max is approximated because finding it is hard.Source: §3.4 · p. 5
• The projection and feature optimization is a saddlepoint problem: one-dimensional W2 requires sorting, while joint gradient search raises boundedness and regularization difficulties.A surrogate objective is used to address these difficulties and to make real and generated data easy to distinguish along the chosen direction.Source: §3.4 · pp. 5–6
• For the moment separator with identity transform, the paper analyzes the maximal suboptimality of the discriminator-derived projection direction.The supplied passage states this result but does not provide its bound.Source: §3.4 · p. 6

3.5. max-sliced GAN Algorithm

• Each training iteration optimizes the feature transform and projection direction for k steps on a surrogate loss, then trains the generator to minimize projected W2.The experiments use binary classification loss for the feature transform.Source: §3.5 · p. 6

4. Experiments

• The experiments report quantitative results for unpaired word translation and qualitative and quantitative image-generation results on CelebA-HQ and LSUN Bedrooms.Source: §4 · p. 6

4.1. Word Translation without Parallel Data

• Replacing the GAN adversary’s Jensen–Shannon objective with max-sliced Wasserstein yields CSLS retrieval precision@1 above [6] across all five tested language pairs.Evaluation tests 1,500 queries against 200k target-language words.Source: §4.1 · p. 7

4.2. Image Generation

• Using one projection, max-sliced Wasserstein GAN produces image samples comparable to or better than sliced Wasserstein GAN with 10,000 projections, reducing computational complexity and memory footprint.The comparison covers CelebA-HQ and LSUN Bedrooms.Source: Figure 3 · p. 7, §4.2 · p. 8, Figure 4 · p. 8

• MSWD has better sample complexity than Wasserstein distance and better projection complexity than sliced Wasserstein distance.Source: §5 · p. 8
• The authors develop a surrogate-loss method to approximate MSWD and analyze the approximation error for one such surrogate.Source: §5 · p. 8
• Empirically, the approach learns high-dimensional distributions using orders of magnitude fewer projection directions than sliced Wasserstein GAN, while working in a similar distance space.Source: §5 · p. 8

1. Experiments with Images at Higher Resolutions

• At 512×512, the max-sliced Wasserstein GAN converges faster than sliced Wasserstein GANs with 100, 1,000, or 10,000 projections, especially early in training.The advantage becomes more pronounced as distribution dimensions increase and is more obvious at 512×512 than at 256×256.Source: §1 · p. 10
• Generated samples use noise from N(0, I128) or 0.05 × N(0, I128), and the model trains end-to-end without progressive growing, stacking, or other tricks.The model uses a simple architecture described in Sec. 4.Source: §1 · p. 10

2. Proof of Claim 1 and Claim 3 in the paper

• For P = {N(a, I_d) | a ∈ R^d}, max-sliced Wasserstein-2 and Gaussian sliced Wasserstein are P-generalizable, but Wasserstein is not.Source: §2 · p. 10

2.1. Wasserstein distance is not P-generalizable

• With polynomially many samples from identical N(0, I) distributions, the empirical Wasserstein distance is high with high probability although the population distance is 0.Thus Wasserstein is not P-generalizable in this setting; the argument does not apply to exponentially many samples.Source: §2.1 · p. 11

2.2. Proof of max-sliced Wasserstein distance is P-generalizable

• For means separated by β along e1, the population max-sliced Wasserstein-2 distance is β, attained by projecting along e1.The optimal projection direction connects the centers of the two Gaussian distributions.Source: §2.2 · p. 13
• With polynomially many samples, the empirical max-sliced objective is close to β, with a lower bound of β − 4δ and a complementary upper bound.The proof estimates the objective at e1 for the lower bound and bounds its maximum for the upper bound.Source: §2.2 · p. 13

2.3. Proof of Lemma 2.1

• The Lemma 2.1 proof combines Gaussian order-statistic variance control with concentration for a Lipschitz function of Gaussian variables.Source: §2.3 · pp. 14–16
• The sample requirement for estimating the max-sliced distance grows as the distributions approach one another, approximately O(1/β^2).Source: §2.3 · p. 16

2.4. Proof of Corollary 2.1

• The proof concludes Corollary 2.1 by contradiction, using the growth of ψ(x) and the assumption n > 80.It shows that δ ≥ 21/10 would imply n < 80, contradicting the assumption, and then concludes the corollary.Source: §2.4 · p. 16

2.5.1 Preliminary Analysis

• For a fixed direction, the projected samples are independent Gaussians with means determined by the direction’s first coordinate, enabling a Lemma 2.1 probability bound.The passage gives probability at least 1 − 2n^-8 for that bound.Source: §2.5.1 · p. 17
• A fixed-direction bound does not establish that the inequality holds for every direction, so the proof proposes a covering argument and union bound.The argument covers the unit sphere with anchor directions and bounds the gap from each direction to a nearby anchor.Source: §2.5.1 · p. 17

2.5.2 Formal Proof

• The formal proof decomposes φ(ω) using centered samples into F1, a sorted projected-sample difference, and F2, a linear projection term.The passage defines F1(ω) = ∥(ω^T X̃)^ord − (ω^T Y)^ord∥ and F2(ω) = ω^T(X̃ − Y).Source: §2.5.2 · p. 17
• The proof extends the fixed-direction F1 bound over a δ-covering with O(1/δ^d) anchors, then controls the gap to nearby directions.It also bounds the largest eigenvalues of YY^T and X̃X̃^T with probability at least 1 − 2n^-8.Source: §2.5.2 · pp. 17–18
• Combining the F1 and F2 bounds proves the lemma’s final bound with probability at least 1 − exp(−d) − 3n^-8.The proof then simplifies the bound before concluding the lemma.Source: §2.5.2 · p. 19
• The lemma’s upper bound is loose because covering and union bounding ignore that only one optimal direction is effective.The proof says this introduces an extra multiplicative factor of d and an extra additive factor of m; stronger bounds may be attainable.Source: §2.5.2 · p. 19

2.6. Sliced Wasserstein distance is P-generalizable

• Proposition 2.2 establishes P-generalizability for sliced Wasserstein-2 using K random directions drawn from N(0, I_d) and empirical distributions of n samples.The passage introduces the empirical distance for two distributions and their size-n empirical versions.Source: §2.6 · p. 19
• Unlike max-sliced Wasserstein, the sliced Wasserstein generalization result imposes requirements on both the number of projection directions and the number of samples.Source: §2.6 · p. 19

3. Proof of Claim 2 in the paper

• The max-sliced Wasserstein-2 distance is a well-defined distance between distributions.This is the restated Claim 2, labeled Claim 3 in the supplement.Source: §3 · p. 21
• Non-negativity, symmetry, and identity of discernibles follow from Wasserstein-2, while the triangle inequality is shown along a maximizing projection using Wasserstein-2’s triangle inequality.The proof considers three distributions and applies the Wasserstein-2 triangle inequality in the direction that achieves the max-sliced distance.Source: §3 · p. 21

4. Architecture for Image Generation

• Tables 1–2 describe the 256x256 architectures and Tables 3–4 the 512x512 architectures; all experiments use Adam at learning rate 0.0001, with batch sizes 64 and 32, respectively.Source: §4 · p. 21