PaperlayerView exact revision on arXiv

Source-linked AI summary

From optimal transport to generative modeling: the VEGAN cookbook

Olivier Bousquet, Sylvain Gelly, Ilya Tolstikhin, Carl-Johann Simon-Gabriel, Bernhard Schoelkopf

arXiv:1705.07642v1stat.ML

TL;DR

The paper asks how unsupervised generative models can be analyzed through optimal transport between the data distribution and a latent-variable model. It reformulates transport with probabilistic encoders, relaxes the constraints into POT, and shows links to AAE, VAE, and WGAN. The main results identify POT with AAE for squared Euclidean cost, relate POT to WGAN for the 1-Wasserstein distance, and explain a source of VAE-like blurriness.

  • Problem

    Unsupervised generative modeling needs a principled formulation connecting the unknown data distribution P_X with a latent-variable model distribution P_G.

  • Method

    The paper rewrites primal optimal transport with constrained probabilistic encoders and relaxes the constraints into a POT objective optimized by stochastic gradient descent.

  • Results

    POT coincides with the AAE objective for squared Euclidean cost and shares the 1-Wasserstein target with WGAN through primal and dual formulations.

  • Takeaways & Limitations

    The analysis theoretically justifies AAE, clarifies VAE blurriness, and establishes a connection between POT and WGAN.

  • Takeaways & Limitations

    Imperfect enforcement of the aggregated-posterior constraint can restrict latent-space gradients and encourage blurry averages or medians.

Abstract

from arXiv · show

We study unsupervised generative modeling in terms of the optimal transport (OT) problem between true (but unknown) data distribution $P_X$ and the latent variable model distribution $P_G$. We show that the OT problem can be equivalently written in terms of probabilistic encoders, which are constrained to match the posterior and prior distributions over the latent space. When relaxed, this constrained optimization problem leads to a penalized optimal transport (POT) objective, which can be efficiently minimized using stochastic gradient descent by sampling from $P_X$ and $P_G$. We show that POT for the 2-Wasserstein distance coincides with the objective heuristically employed in adversarial auto-encoders (AAE) (Makhzani et al., 2016), which provides the first theoretical justification for AAEs known to the authors. We also compare POT to other popular techniques like variational auto-encoders (VAE) (Kingma and Welling, 2014). Our theoretical results include (a) a better understanding of the commonly observed blurriness of images generated by VAEs, and (b) establishing duality between Wasserstein GAN (Arjovsky and Bottou, 2017) and POT for the 1-Wasserstein distance.

1 Introduction

The paper frames unsupervised generative modeling as optimal transport between the unknown data distribution and a latent-variable model. It derives a penalized formulation linking optimal transport to AAE, VAE blurriness, and WGAN.

  • Optimal transport offers a weaker topology than f-divergences, which can provide no useful gradients when data and model distributions lie on low-dimensional, largely disjoint manifolds.The paper argues that optimal-transport objectives may therefore support more stable training.
  • The paper minimizes the optimal transport cost between the true data distribution P_X and the latent-variable model distribution P_G.
  • The primal transport problem is equivalently formulated using probabilistic encoders Q(Z|X), making the latent space and encoder role explicit.
  • Relaxing encoder constraints by penalization yields the POT objective, which can be minimized with stochastic gradient descent by sampling from P_X and P_G.
  • For squared Euclidean cost, POT coincides with the AAE objective and theoretically supports AAE as an approximate minimizer of the 2-Wasserstein distance, while VAE-like methods produce blurry outputs.
  • For Euclidean cost, POT and WGAN both minimize the 1-Wasserstein distance but use primal and dual formulations, respectively, producing different algorithmic behavior.

2 Notations and preliminaries

This section introduces optimal transport, implicit latent-variable generators, and the VAE, GAN, and AAE approaches that the paper compares. These methods differ in divergences, parameterizations, and how latent codes generate observations.

  • Optimal transport: Optimal transport defines a distance by minimizing expected cost over joint distributions whose marginals are P and Q.
  • Optimal transport: Wasserstein distances arise when the transport cost is a power of a metric, with W_1 also admitting a Kantorovich–Rubinstein dual formulation.
  • Implicit generative models: An implicit generative model samples a latent code Z from a fixed prior P_Z and maps it through G to an observation in X.
  • GANs: GANs minimize a lower bound on Jensen–Shannon divergence, whereas WGAN minimizes a lower bound on the 1-Wasserstein distance.
  • VAEs: VAEs use a random decoder and restrict encoders to Gaussian distributions in the original implementation, thereby optimizing an upper bound on negative log-likelihood.
  • AAEs: AAEs replace the VAE KL regularizer with a GAN-based divergence between the aggregated posterior Q_Z and prior P_Z.The aggregated posterior is obtained by sampling X from P_X and then Z from Q(Z|X).

3 Minimizing the primal of optimal transport

The paper reformulates primal optimal transport for generative modeling through probabilistic encoders, whose aggregated posterior must match the latent prior. Relaxing this constraint yields a penalized objective that supports stochastic optimization and recovers known adversarial-autoencoder objectives in important cases.

  • 3.1 Reparametrization of the couplings: Optimal transport couplings can be factored into an encoder Q(Z|X) followed by the model's generator PG(Y|Z).This reparametrization expresses transport through latent codes while preserving the relevant joint-distribution structure.
  • 3.1 Reparametrization of the couplings: The encoder is constrained so its aggregated posterior QZ matches the prior PZ, unlike in VAE- and marginal-log-likelihood-based formulations.This constraint defines the feasible encoder family used in the optimal-transport formulation.
  • 3.1 Reparametrization of the couplings: For deterministic generators, the original transport problem equals transport between PX and PZ under the induced cost cg(x,z).Theorem 1 establishes Wc(PX, PG) = WcG(PX, PZ) when PG(Y|Z) is generated by Y = G(Z).
  • 3.2 Relaxing the constraints: The constrained optimization is relaxed by adding a convex penalty that vanishes exactly when QZ equals PZ.Increasing the penalty parameter reduces the constraint violation and approaches the feasible region.
  • 3.2 Relaxing the constraints: An adversarial approximation of the latent-distribution penalty produces the POT objective, which can be optimized with SGD by alternating encoder-decoder and discriminator updates.The discriminator separates latent codes sampled from PZ and QZ; the adversarial approximation becomes tight in the nonparametric limit.
  • 3.2 Relaxing the constraints: For Gaussian decoders, the relaxed transport objective matches the relevant formulation up to an additive d·σ2 constant, while DPOT provides its adversarial approximation.This connects the relaxed optimal-transport formulation to the Gaussian-decoder setting used in comparisons with VAEs.

4 Implications: relations to AAE, VAEs, and GANs

The paper connects penalized optimal transport to AAE, VAE, AVB, and WGAN, explaining output blurriness and contrasting primal and dual optimization. It also identifies how regularization and approximate constraints affect learned gradients and generators.

  • Relation to VAE and AVB: The POT solution G† is independent of σ2 and also minimizes the 2-Wasserstein distance to the corresponding noiseless implicit model.By contrast, the VAE and AVB solution G* depends on σ2.
  • Regularization effects: When the AAE penalty λ approaches zero, the relaxation becomes loose and AAE approaches an unregularized autoencoder; as λ approaches infinity, it converges toward the constrained problem.Because λ = 2σ2, these regimes correspond to σ2 approaching zero or infinity.
  • Relation to VAE and AVB: VAE, AVB, and related marginal-likelihood methods average input points whenever encoder supports overlap, producing blurry outputs with Gaussian decoders.The paper attributes blurriness to the combination of Gaussian decoders and the KL-based objective, rather than to the reconstruction cost alone.
  • Relation to WGAN: For Euclidean cost, POT and WGAN minimize the same 1-Wasserstein objective through primal and dual formulations, respectively.The two formulations permit gradient computation at optimal encoders or witness functions, although practical approximations can behave differently.
  • Optimization caveats: Approximate enforcement of the POT posterior constraint can restrict gradients to part of latent space or map several inputs to one code, encouraging averaging and blurriness.These effects arise when the aggregated posterior fails to match the prior.

5 Conclusion

The work proposes fitting generative models by minimizing optimal transport costs and establishes theoretical links among unsupervised probabilistic modeling techniques. Its empirical relevance is strongest for the 2-Wasserstein case, while other cost functions remain future work.

  • The method fits generative models by minimizing any optimal transport cost.
  • The paper establishes novel links between popular unsupervised probabilistic modeling techniques.
  • Strong empirical performance is reported for the special case of the 2-Wasserstein distance.
  • Experiments with other cost functions c are beyond the scope of the work and left for future studies.

A Further details on VAEs and GANs

This section relates VAE, AAE, and GAN objectives through inequalities and optimization conditions. It emphasizes that restricted encoder families may not match the model posterior, while deriving bounds among the associated divergences.

  • The VAE-related discussion uses KL-divergence and marginal log-likelihood for latent-variable models PG.
  • For unrestricted conditional distributions, the infimum is achieved when Q(Z|X) equals PG(Z|X).
  • For restricted classes Q, the optimization yields only an inequality because Q(Z|X) may not match P(Z|X) for every X.
  • DAAE(PX, PG) is bounded above by DAVB(PX, PG) for any distributions PX and PG.
  • Under certain assumptions, the paper also links DAAE(PX, PG) to DVAE(PX, PG).

B.1 Proof of Theorem 1

The proof develops a relationship between two sets of joint distributions used in the optimal transport formulations. Equality holds when the decoder produces deterministic outputs from the latent variable.

  • The lemma establishes that PX,Y is contained in P(PX, PG).
  • The two sets coincide when PG(Y|Z = z) is a Dirac distribution for every z ∈ Z2.
  • The proof derives the identity using the fact that Y is a deterministic function of Z and properties of conditional independence.

B.2 Random decoders PG(Y |Z)

This section analyzes random decoders whose conditional outputs have specified means and marginal variances. The stated proof follows the earlier theorem, while a related necessary condition is left as a conjecture.

  • The random decoder has conditional mean G(z) ∈ Rd and marginal variances σ2_1, …, σ2_d ≥ 0 for every z ∈ Z.
  • The proof is stated to be similar to the proof of Theorem 1 and is given in Section B.2.
  • The paper conjectures that the stated condition is necessary, but does not use necessity in the remainder of the paper.
  • The argument concludes using equation (17) and PX,Z defined by independent draws X ∼ PX and Z ∼ PZ.

B.3 Proof of Proposition 2

The proof shows that G† is independent of σ2 in the zero-variance case, while the positive-variance case generally depends on σ2. A Gaussian example identifies the standard normal as a global minimum.

  • G† does not depend on the variance σ2.
  • When σ2 = 0, PG(Y|Z) becomes Dirac, and G† also minimizes Wc(PX, P0G).
  • When σ2 > 0, G* generally depends on σ2.
  • For PX = PZ = N(0, 1) and 0 < σ2 < 1, the Gaussian construction yields a global minimum at the standard normal Gaussian N(0, 1).

B.4 Proof of Proposition 3

The proof constructs a discrete two-point setting and perturbs optimal functions locally to produce sharply different gradient directions while controlling the objective change.

  • The proof considers PX supported on {x0, x1} and PZ supported on {0, 1}, with y0 = G(0) and y1 = G(1).
  • A local modification around y0 preserves the Lipschitz constant while making the gradients of the resulting fε differ arbitrarily from those of f*.
  • For smooth functions, placing y0 and y1 close together forces the gradients of fF,0 at those points to be very close despite opposite gradients of f*.
  • Shifting the function by ε can swap one gradient direction while changing the criterion by less than ε.
Loading 1705.07642v1…