Distributional Sliced-Wasserstein and Applications to Generative Modeling

1 Introduction

• Optimal transport measures discrepancies between probability distributions, but its computational expense obstructs practical use.Entropic regularization methods have complexity of order O(k^2), which is expensive when OT must be computed repeatedly during distribution learning.Source: §1 · p. 1
• Slicing combines one-dimensional optimal transport’s closed form with projections of distributions through the Radon transform.The paper also notes that nonlinear projections yield generalized sliced-Wasserstein variants.Source: §1 · pp. 1–2
• SW’s uniform projection expectation is intractable and can require many samples, while Max-SW’s single direction can miss other high-dimensional distribution differences.The paper notes uniform sampling may draw many irrelevant directions, especially when data lie on a low-dimensional manifold.Source: §1 · p. 2
• Other approaches include optimal-subspace Max-SW and Projected Wasserstein, which averages costs from multiple one-dimensional optimal transport maps.The paper characterizes the former as computationally expensive because it cannot exploit the one-dimensional Wasserstein closed form.Source: §1 · p. 2
• DSW searches for an optimal, regularized distribution over projections to balance direction informativeness with exploration of distinct directions.The paper presents it as a generalization of Max-SW; without regularization, it recovers max-(generalized) SW.Source: §1 · p. 2
• In generative modeling, DSW significantly outperforms SW and Max-SW under similar computational time, with faster convergence and more realistic generated images.The experiments use real, large-scale datasets.Source: §1 · p. 3

2 Background

• The section situates these constructions within backgrounds on Wasserstein distance, Radon transforms, and sliced-Wasserstein distances.These topics provide the framework for the paper’s later distributional sliced distance.Source: §2 · p. 3

2.1 Wasserstein distance

• Wasserstein distance compares probability measures through transportation plans; for one-dimensional measures, it has a closed-form expression using their cumulative distribution functions.The paper considers measures with finite p-th moment.Source: §2.1 · p. 3

2.2 (Generalized) Radon transforms

• The Radon transform maps a function to values over lines, while the generalized Radon transform integrates over hypersurfaces defined by a function g.When g(x, θ) = ⟨x, θ⟩ and the feasible parameter set is the unit sphere, the generalized transform becomes the standard Radon transform.Source: §2.2 · pp. 3–4

2.3 (Generalized) sliced-Wasserstein distances

• SW and GSW use projection-based sliced distances, but their integrals are usually intractable and approximated by Monte Carlo sampling.Uniform sampling can draw many unimportant projections, increasing cost and reducing the effect of the distance.Source: §2.3 · p. 4

2.4 Max (generalized) sliced-Wasserstein distances

• Max-SW selects the single slice that best discriminates two distributions, and Max-GSW extends that idea to nonlinear projections.Because they focus on one direction, these distances can ignore useful information about high-dimensional distribution structure.Source: §2.4 · p. 4

3.1 Definition and metricity

• DSW searches for a probability measure over unit-sphere slices, between SW’s uniform weighting and Max-SW’s single-direction Dirac measure.It concentrates weight around important directions while discouraging those directions from lying in only one small area, where their projected distributions may become similar.Source: §3.1 · p. 5
• A small admissible regularization bound C favors directions farther apart, while C ≥ 1/d makes DSW, SW, Max-SW, and Wasserstein topologically equivalent.With C ≥ 1/d, convergence under DSW implies and is implied by convergence under the other listed metrics.Source: §3.1 · p. 5
• DSW is a well-defined metric on Borel probability measures with finite p-th moment.The stated properties are non-negativity, symmetry, identity, and the triangle inequality.Source: §3.1 · p. 5
• DSW’s statistical estimation error is C_d · n^-1/2 with high probability, and the paper states that it does not suffer from the curse of dimensionality.The constant C_d depends on dimension d.Source: §3.1 · p. 6

3.2 Computation of DSW distance

• The dual form replaces the constrained measure search with a regularized optimization over push-forwards of the uniform sphere measure.The reparameterization uses a Borel measurable function f from the unit sphere to itself; λ_C tunes the regularized term.Source: §3.2 · p. 6
• A neural network parameterizes the dual-form function, whose objective is updated by stochastic gradient ascent using Monte Carlo estimates.In generative-model applications, the gradient of the objective updates the parameters of interest, and λ_C can be tuned as a regularization parameter.Source: §3.2 · p. 6
• In the Gaussian illustration, increasing λ_C spreads optimal σ from one direction toward other directions, and at 1000 it concentrates near the two covariance eigenvectors that distinguish the distributions.Figure 1 shows empirical optimal-measure behavior on a circle for bivariate Gaussians sharing eigenvectors; at λ_C = 0, C = 1, and increasing λ_C makes C small.Source: §3.2 · pp. 6–7, Figure 1 · p. 7
• The paper extends DSW to DGSW using nonlinear projection through the generalized Radon transform.It also reports an appendix comparison with Max-GSW-NN using a neural-network defining function.Source: §3.2 · p. 7

4.1 Results on MNIST

• On MNIST, DSW-10 achieves better Wasserstein-2 model quality than Max-GSW-NN and Max-SW, runs an order of magnitude faster, and scores roughly 10% lower than SW at similar time.Except in the very-fast, low-quality regime, DSW outperforms the existing sliced-based baselines in both quality and speed; generated images are described as sharper and easier to classify.Source: §4.1 · p. 8, Figure 2 · p. 9
• On MNIST, DSW-10 has a better Wasserstein-2 score than PRW-5 while running 30 times faster; PRW-50 improves its score about 29% relative to DSW-10 but is around 40 times slower.The authors attribute DSW’s computational advantage to exact one-dimensional Wasserstein-distance calculation.Source: §4.1 · p. 8, Figure 2 · p. 9
• On MNIST, DSW learns better models faster than the other baseline distances with few projections; DSW-10 is the second-lowest curve among the sliced-distance methods.The comparison tracks Wasserstein-2 distance between learned and test distributions over training iterations.Source: §4.1 · p. 8, Figure 2 · p. 9
• On MNIST, DSW’s computational complexity scales as O(k log k), like the other sliced-based distances, where k is the minibatch sample count.Figure 2(c) compares computation speed as minibatch sample size changes.Source: §4.1 · p. 8, Figure 2 · p. 9
• Increasing λC lowers the mean absolute pairwise cosine similarity among DSW directions: it is near 1 without regularization and near 0.1 at λC = 1000.These values indicate direction collapse without regularization and near-orthogonality at λC = 1000.Source: §4.1 · p. 8, Figure 2 · p. 9
• Additional MNIST experiments find that more gradient steps for updating σ improve DSW performance but slow computation, and that the distributional approach also improves non-linear slicing quality.The paper also evaluates DGSW and encoder-generator training with joint contrastive inference in appendices.Source: Additional experiments · pp. 8–9

4.2 Results on Large-scale Datasets

• On CIFAR10, DSW’s FID scores are about 20 units lower than SW’s at the same computational time; on LSUN, DSW-100 scores 46 while SW-10000 exceeds 60.Across the large datasets, DSW outperforms the compared baselines in FID and computational efficiency; PRW was not compared because of its training cost.Source: §4.2 · p. 10, Figure 3 · p. 10
• On CIFAR10, CelebA, and LSUN, DSW outperforms SW, Max-SW, and Max-GSW-NN in FID and computational efficiency; its FID advantage is especially large on CIFAR10 and LSUN.Max-SW has the highest FID scores among the compared distances while requiring heavy computation; DSW samples useful directions without SW’s uniform-sampling inefficiency.Source: §4.2 · p. 10, Figure 3 · p. 10

• DSW searches for a distribution of important directions while regularizing toward orthogonal directions, and the paper establishes it as a well-defined metric with theoretical and computational properties.Source: §5 · p. 11
• Across generative-modeling tasks, experiments show that DSW yields significantly better models and training convergence behavior than previous sliced-based distances.Source: §5 · p. 11
• A stated future direction is to determine λC theoretically so DSW can capture all directions that distinguish the target measures.Source: §5 · p. 11

Supplement to “Distributional Sliced-Wasserstein and Applications to Generative Modeling”

• The supplement gathers main-text proofs, additional DSW properties, DGSW theory, generative-modeling details, and further experiments and settings.Source: Supplement to “Distributional… · p. 15

A.1 Proof of Theorem 1

• The appendix proof establishes non-negativity, symmetry, identity of indiscernibles, and the triangle inequality for DSW.The identity property uses Fourier-transform reasoning; the triangle inequality follows through Wasserstein’s triangle inequality and Minkowski’s inequality.Source: §A.1 · p. 15
• Consequently, for p ≥ 1 and admissible C > 0, DSW is a metric on Borel probability measures with finite p-th moment.Source: §A.1 · p. 15

A.2 Proof of Theorem 2

• For any p ≥ 1, Max-SW is bounded above by Wasserstein distance, and SW is bounded above by Max-SW.Source: §A.2 · p. 16
• For C ≥ 1/d and p ≥ 1, convergence in DSW is equivalent to convergence in Max-SW; under these conditions DSW, Max-SW, SW, and Wasserstein distances are equivalent.Source: §A.2 · p. 17

B Additional Studies with Distributional Sliced-Wasserstein Distance

• The supplement introduces further studies with DSW but this excerpt gives no specific additional finding.Source: B Additional Studies with… · p. 17

B.1 Discussion of the constraint in DSW

• The admissible set can contain both the uniform measure and a measure supported on an orthonormal basis under the stated constraint condition.The discussion also identifies the uniform measure as admissible in the stated regime.Source: §B.1 · p. 17

B.2 Approximation of dual value of DSW

• Monte Carlo sampling approximates the intractable outer expectations in the dual objective using i.i.d. directions sampled from the unit sphere.The appendix also describes using stochastic gradient ascent and a fixed point to approximate the dual value.Source: §B.2 · p. 19

B.3 Statistical guarantee of DSW

• Theorem 4's statistical guarantee is described as similar to those of other sliced distances and as not suffering from the curse of dimensionality.The theorem assumes i.i.d. samples from a probability measure supported on a compact subset of R^d, with admissible C > 0 and p ≥ 1.Source: §B.3 · p. 19

C An extension to distributional generalized sliced-Wasserstein distance

• DGSW uses nonlinear projections to capture more complex structures in target probability measures.For unit-sphere parameters, the dual form supports approximation through a parameterized function and stochastic gradient ascent.Source: C An extension to distributional… · pp. 20–21
• When the generalized Radon transform is injective, DGSW is a well-defined metric for Borel probability measures with finite p-th moment.The extension uses probability measures over feasible transform parameters, constrained by an upper bound C on expected absolute pairwise cosine similarity.Source: C An extension to distributional… · pp. 20–21

D Applications to Generative Modeling

• The paper presents DSW and DGSW for generative modeling settings that benefit from computationally efficient optimal-transport-type distances.The two settings described are minimum expected distance estimation and joint contrastive inference.Source: D Applications to Generative Modeling · p. 22

D.1 Minimum expected distributional sliced-Wasserstein estimator

• The minimum expected distributional sliced-Wasserstein estimator replaces SW and Max-SW with DSW for learning implicit generative models.The generator distribution may be formed by pushing a simple distribution, such as a standard Gaussian, through a neural network.Source: §D.1 · p. 22

D.2 Distributional sliced-Wasserstein joint contrastive inference

• The DSW-based joint contrastive inference method learns a generator and an inference model by minimizing a divergence between their structured joint distributions.The paper describes these distributions as pθ(z, x) and the data-induced aggregated joint inference model q̂φ(z, x).Source: §D.2 · p. 22

E Additional Experiments

• The appendix compares DGSW with generalized sliced-distance variants, DSW with ASW and PRW, and several sliced distances on joint contrastive inference.The described comparisons include varying projection counts for generated images and a Wasserstein-2 distance versus execution-time comparison on MNIST.Source: E Additional Experiments · p. 23

E.1 Generative models

• With 10 projections, DGSW outperformed GSW with 1,000 projections, Max-GSW, and Max-GSW-NN on MNIST’s Wasserstein-2 score.DGSW-10 was faster than the other reported methods except GSW-10, which had the worst Wasserstein-2 score.Source: §E.1 · p. 23, Figure 4 · p. 23
• Increasing minibatch sample size improved the performance of both DSW and DGSW in the MEDE framework.The reported comparisons appear in Figures 5(b) and 5(d).Source: §E.1 · pp. 23–24, Figure 5 · p. 24
• Increasing pushforward-measure gradient updates improved DSW and DGSW quality but considerably increased computation time.At 10 or 100 updates, both were slower than Max-SW, Max-GSW, and Max-GSW-NN.Source: §E.1 · p. 24, Figure 6 · p. 24
• DSW and circular DGSW achieved the best performance among the reported sliced distances on CIFAR10, CelebA, and LSUN FID results.The FID results average five runs; DSW-100 was faster than DSW-10000 and had lower FID.Source: §E.1 · p. 25, Table 1 · p. 25, Table 2 · p. 25
• Across MNIST, CelebA, CIFAR10, and LSUN, DSW and DGSW helped improve synthetic-image quality for linear and nonlinear projections.The paper presents generated-image comparisons in Figures 7–11.Source: §E.1 · p. 26, Figure 7 · p. 26, Figure 8 · p. 27
• With λC = 0, the tested Max-GSW-NN variant matched DSW and Max-SW performance, while DSW’s regularization prevents its projection distribution from collapsing to a Dirac measure.The comparison concerns the special defining function g(x, θ) = ⟨x, f(θ)⟩.Source: §E.1 · p. 26, Table 3 · p. 26

E.2 Comparison with Projected Robust Subspace Wasserstein

• PRW searches for a subspace of dimension at least 2, where the projected Wasserstein-2 distance has no closed-form solution.The experiment compares DSW projections with PRW subspaces on MNIST.Source: §E.2 · pp. 27–28
• DSW-10 achieved a better Wasserstein-2 score than PRW with a 5-dimensional subspace while running 30 times faster.For PRW with a 50-dimensional subspace, the score improved 32.25% while computation time increased 10 times.Source: §E.2 · p. 28, Table 4 · p. 28

E.3 Joint contrastive inference

• In joint contrastive inference, DSW outperformed SW at the same projection count, and DSW-1000 performed best across all reported metrics.The metrics compare joint latent-observed distributions, their marginals in Wasserstein-2 distance, and reconstruction ability.Source: §E.3 · p. 29, Figure 13 · p. 29
• For nonlinear sliced distances in joint contrastive inference, circular DGSW outperformed the other compared nonlinear sliced-based distances.Figure 14 presents the nonlinear-distance comparisons.Source: §E.3 · p. 29, Figure 14 · p. 30
• With 10 projections, DSW and DGSW reconstructed MNIST digits more correctly than SW and GSW, which could not recreate them.At higher projection counts, visual comparisons were difficult, but reconstruction-error plots still favored DSW and DGSW over the other sliced-based distances.Source: §E.3 · p. 29, Figure 13 · p. 29, Figure 15 · p. 30

F Experiment Settings

• Evaluation computes discrete Wasserstein distance by linear programming on empirical distributions of 10,000 samples from each target distribution.The experiments use the L2 ground metric; GSW and DGSW use r = 1000 for the circular function.Source: F Experiment Settings · p. 30
• The MNIST generator and joint-contrastive encoder-decoder use fully connected layers, with the encoder ending at a 32-dimensional latent representation.The generator maps 32-dimensional z through layers of 100, 200, 400, and 784 units; the encoder maps 28×28 inputs through 400, 200, 100, and 32 units.Source: F Experiment Settings · p. 31
• CelebA, CIFAR10, and LSUN use a transposed-convolution generator and convolutional discriminator, while all models use Adam with learning rate 0.0005 and betas (0.5, 0.999).Batch size is 512 for MNIST, CelebA, and CIFAR10, and 4096 for LSUN; training lasts 200, 50, 100, and 20 epochs, respectively.Source: F Experiment Settings · pp. 30–31