PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixed-curvature Variational Autoencoders

Ondrej Skopek, Octavian-Eugen Ganea, Gary Bécigneul

arXiv:1911.08411v2cs.LGstat.ML

TL;DR

Some data have non-Euclidean latent structure, while existing generative-model approaches use geometry-specific spaces. This paper introduces a VAE whose latent space is a product of constant-curvature manifolds, and reports competitive performance across datasets while recovering the Euclidean VAE as component curvatures approach zero.

  • Problem

    Some data have non-Euclidean latent structure, while geometry-specific representations have been developed for particular data types.

    Source: §1 · p. 1

  • Method

    The Mixed-curvature VAE uses a product of constant-curvature spaces as its latent space, with component curvature fixed or learnable.

    Source: §1 · p. 1, §2.3 · p. 4

  • Results

    The approach is competitive across various datasets and outperforms current benchmarks on synthetic tree and image-reconstruction tasks for some latent dimensions.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The approach generalizes the Euclidean VAE and recovers it when all component curvatures approach zero.

    Source: §5 · p. 10

Abstract

from arXiv · show

Euclidean geometry has historically been the typical "workhorse" for machine learning applications due to its power and simplicity. However, it has recently been shown that geometric spaces with constant non-zero curvature improve representations and performance on a variety of data types and downstream tasks. Consequently, generative models like Variational Autoencoders (VAEs) have been successfully generalized to elliptical and hyperbolic latent spaces. While these approaches work well on data with particular kinds of biases e.g. tree-like data for a hyperbolic VAE, there exists no generic approach unifying and leveraging all three models. We develop a Mixed-curvature Variational Autoencoder, an efficient way to train a VAE whose latent space is a product of constant curvature Riemannian manifolds, where the per-component curvature is fixed or learnable. This generalizes the Euclidean VAE to curved latent spaces and recovers it when curvatures of all latent space components go to 0.

1 INTRODUCTION

Euclidean spaces are useful for generative modeling, but some data have non-Euclidean latent structure that can benefit from geometry-matched representations. The paper introduces a VAE over products of constant-curvature spaces and reports benchmark improvements on several tasks for some latent dimensions.

Motivation

  • Some data have non-Euclidean latent structure, and changing latent geometry can improve representations for specific data types such as trees and scale-free networks.The introduction also notes that interpretable images are often viewed as lying on a lower-dimensional natural image manifold.Source: §1 · p. 1

Motivation and gap

  • Prior work used spherical, hyperbolic, and product spaces, but choosing each component's dimensionality and curvature remained an open question.VAEs had also been extended to spherical and hyperbolic latent spaces.Source: §1 · p. 1

Contribution

  • The Mixed-curvature VAE generalizes VAEs to products of constant-curvature spaces, yielding a more flexible latent manifold while maintaining efficient optimization.The introduction describes this approach as offering a better reduction in dimensionality.Source: §1 · p. 1

Empirical contribution

  • The models outperform current benchmarks on a synthetic tree dataset and on image reconstruction for MNIST, Omniglot, and CIFAR at some latent dimensions.The contribution includes a framework for products of constant-curvature spaces and generalized Gaussian-like priors.Source: §1 · p. 2

2 GEOMETRY AND PROBABILITY IN RIEMANNIAN MANIFOLDS

The paper develops tools for working across positive-, zero-, and negative-curvature manifolds, including projected models and operations that converge to Euclidean counterparts. It then combines component spaces and develops distributions for VAE priors and posteriors, focusing on Wrapped Normal distributions.

2 GEOMETRY AND PROBABILITY IN RIEMANNIAN MANIFOLDS

  • Constant-curvature spaces are classified by curvature sign: the hypersphere, Euclidean space, and hyperboloid represent positive, zero, and negative curvature, respectively.Sectional curvature describes the curvature of a two-dimensional plane spanned by tangent-space vectors.Source: §2 · p. 2

2.1 STEREOGRAPHICALLY PROJECTED SPACES

  • Hyperboloid and hypersphere coordinates diverge as curvature approaches zero, so the paper uses stereographic projections to obtain the Poincaré ball and projected sphere, whose models converge appropriately toward flat space.The projections preserve angles and map the distinguished point to the origin in the projected space.Source: §2.1 · pp. 2–3

2.2 GYROVECTOR SPACES

  • Gyrovector-space operations provide curvature-aware analogues of vector operations, and the resulting gyrospace distances converge to Euclidean distance as curvature approaches zero.The paper summarizes operations for the original and projected spaces in Tables 1 and 2.Source: §2.2 · p. 3, Table 1 · p. 4, Table 2 · p. 4

2.3 PRODUCTS OF SPACES

  • The latent space is a product of constant-curvature components, each with its own model, dimensionality, and curvature; the resulting product manifold has non-constant curvature.Representations and operations are applied componentwise.Source: §2.3 · p. 4

2.4 PROBABILITY DISTRIBUTIONS ON RIEMANNIAN MANIFOLDS

  • The paper compares wrapped, restricted, and maximum-entropy approaches to manifold distributions, then focuses on Wrapped Normals for their computational efficiency.Wrapped Normals are efficient to sample from and evaluate, while Riemannian Normals are more expensive to sample because available methods use rejection sampling.Source: §2.4 · p. 5

2.4.1 WRAPPED NORMAL DISTRIBUTION

  • A Wrapped Normal maps a Gaussian sample from a tangent space onto a manifold using parallel transport and the exponential map, and converges to a Gaussian as curvature approaches zero.The paper extends the sampling procedure to arbitrary curvature and states that the distribution can be applied to all introduced manifolds.Source: §2.4 · p. 5, §2.4.1 · pp. 5–6

3 VARIATIONAL AUTOENCODERS IN PRODUCTS SPACES

The VAE adapts its mean parametrization and prior and posterior distributions to the selected manifold, using Wrapped Normals across spaces. Curvature may be fixed or learned, and a staged universal-curvature procedure approximates component selection and curvature learning.

3 VARIATIONAL AUTOENCODERS IN PRODUCTS SPACES

  • For manifold-valued latents, the VAE changes mean and covariance parametrization and chooses prior and posterior distributions for the selected space; its mean uses the exponential map.The paper describes Wrapped Normal families as an available choice across its spaces.Source: §3 · p. 6

3.1 LEARNING CURVATURE

  • Curvature can be fixed in advance or learned by differentiating the ELBO with respect to curvature and optimizing it alongside the VAE maps.For fixed-curvature experiments, values are selected from [0.25, 1.0] for stability; learning curvature directly is described as badly conditioned.Source: §3.1 · pp. 6–7
  • The universal-curvature procedure initializes 2-dimensional Euclidean components, assigns groups to hyperbolic, spherical, and Euclidean types, then learns non-Euclidean curvatures.It is an approximation that never uses components larger than dimension 2, but the authors report satisfactory empirical performance.Source: §3.1 · p. 7
  • To avoid divergence near zero curvature, the universal approach uses projected spheres for positive curvature and Poincaré balls for negative curvature.The paper denotes this universal approach as U_n in its experiments.Source: §3.1 · p. 7

4 EXPERIMENTS

The experiments compare mixed-curvature VAEs with fixed-curvature and Euclidean baselines across four datasets and multiple latent dimensions. Universal-curvature models generally outperform corresponding Euclidean baselines at lower dimensions, while other results vary by dataset and covariance setting.

4 EXPERIMENTS

  • The experiments used BDP, dynamically binarized MNIST and Omniglot, and CIFAR-10, evaluating marginal log-likelihood with importance sampling across repeated runs.Models were run at least three times; CIFAR-10 used 50 importance samples because of memory constraints, versus 500 for the other datasets.Source: §4 · p. 7

Dynamically-binarized MNIST reconstruction

  • On MNIST with spherical covariance, several approaches beat the Euclidean baseline, especially Riemannian Normal Poincaré-ball components and product VAEs; universal models performed similarly to the baseline.The vMF component underperformed Wrapped Normal except in a product such as E2×H2×(vMF S2).Source: Table 3 · p. 8, §4 · p. 9

Binary diffusion process

  • On BDP, Riemannian Normal VAEs performed well, learned curvature was worse for that model, and learned-curvature products outperformed single-component VAEs.The universal-curvature product beat other approaches except the Riemannian Normal baseline, though some models were within its margin of error; learned curvatures stayed approximately between −0.1 and +0.1.Source: §4 · pp. 8–9, Table 3 · p. 8

Dynamically-binarized MNIST reconstruction

  • With diagonal covariance on MNIST, the Riemannian Normal Poincaré-ball VAE remained best at dimension 6; at dimension 12, fixed-curvature products and learned-curvature single components performed better, respectively.Fixed-curvature products were marginally better than their learnable counterparts at dimension 6; universal VAEs were around Euclidean-baseline performance and learned negative curvatures.Source: Table 4 · p. 9, §4 · p. 9

Dynamically-binarized Omniglot reconstruction

  • On Omniglot, several models were within the best baseline’s margin of error, but run-to-run variance prevented a clear conclusion; hyperbolic VAEs seemed somewhat better than spherical VAEs.The best baseline was the Poincaré VAE; near-baseline models included (S2)3, D2 × E2 × P2, and U6.Source: §4 · pp. 9–10, Table 5 · p. 10

CIFAR-10 reconstruction

  • On CIFAR-10 at dimension 6, almost all non-Euclidean models outperformed the Euclidean baseline; learnable curvatures converged approximately between −0.15 and +0.15.The fixed hyperboloid and learnable hypersphere were especially strong performers.Source: Table 5 · p. 10, §4 · p. 10

Summary

  • Universal-curvature VAEs generally outperform corresponding Euclidean baselines at lower latent dimensions and retain most competitive performance, with minor losses, as dimensionality rises.The reported advantage is strongest at lower dimensions; at dimension 12, several models are only within the Euclidean baseline’s margin of error.Source: §4 · pp. 9–10

5 CONCLUSION

The conclusion presents a VAE framework with product latent spaces of constant curvature, including learnable component curvatures, and says it recovers the Euclidean VAE when all curvatures approach zero. The appendix introduces Riemannian tools used to describe geometry and move between points and tangent spaces.

5 CONCLUSION

  • The framework generalizes VAEs to products of constant-curvature spaces with learnable curvature, is competitive on various datasets, and recovers the Euclidean VAE as all component curvatures approach 0.The paper derives operations for these spaces and extends probability distribution families to them.Source: §5 · p. 10

A.1 RIEMANNIAN GEOMETRY

  • A Riemannian manifold pairs a smooth manifold with a metric that defines scalar products and norms on its tangent spaces.The tangent space at a point collects the directions and speeds of motion through that point.Source: §A.1 · p. 15
  • Integrating local metric contributions yields a manifold measure and curve lengths, whose minimizing geodesics induce global distances.The exponential map moves from a point along a tangent vector, and the logarithmic map is its inverse.Source: §A.1 · p. 15
  • Parallel transport moves tangent vectors along geodesics to connect tangent spaces while preserving their parallel relation.The paper describes it as an isomorphism between the source and destination tangent spaces.Source: §A.1 · p. 15

A.2 BRIEF COMPARISON OF CONSTANT CURVATURE SPACE MODELS

The appendix compares constant-curvature models by their optimization behavior and their behavior near zero curvature. It also gives Euclidean-space operations as a simple reference case.

A.2 BRIEF COMPARISON OF CONSTANT CURVATURE SPACE MODELS

  • Hyperboloids and hyperspheres have fewer optimization instabilities than their projected counterparts, but their point norms diverge as curvature approaches 0.Crossing curvature 0 between hyperboloid and sphere is problematic in practice; projected models also have conditioning issues, including boundary-related numerical instability in the Poincaré ball.Source: §A.2 · pp. 15–16
  • The convergence argument assumes points remain fixed as curvature changes; the authors conjecture that boundary-near points may explain some projected-model instabilities.They use different projected and unprojected models across experiments to compare their performance.Source: §A.2 · p. 16

A.3.1 EUCLIDEAN SPACE

  • In Euclidean space, the exponential map is x + v, and parallel transport can be taken as the identity because Normal distributions can be sampled directly.The section introduces the Euclidean distance function and invokes the Pythagorean theorem to derive its form.Source: §A.3.1 · p. 16

A.4.1 HYPERBOLOID

The hyperboloid section describes its geometric operations and a projection procedure, while noting that the model's points diverge as curvature approaches zero. This makes the hyperboloid unsuitable for learning sign-agnostic curvatures.

A.4.1 HYPERBOLOID

  • Hyperboloid operations use hyperbolic trigonometric functions, including sinh, cosh, and tanh, rather than their Euclidean counterparts.The text also invokes cosh^2(α) − sinh^2(α) = 1.Source: §A.4.1 · p. 16
  • The hyperboloid projection normalizes an ambient vector, while its coordinate constraint lets one recover a missing coordinate from the others.The section notes this is useful for projecting points from the tangent space at μ0 onto the manifold.Source: §A.4.1 · p. 16
  • As negative curvature approaches 0, hyperboloid points diverge, making this model unsuitable for learning sign-agnostic curvatures.The text identifies the hyperboloid origin as a point that diverges.Source: §A.4.1 · p. 17
  • The logarithmic map is derived as the inverse of the exponential map, and the section also gives a special parallel-transport form for source vector μ0.A generic manifold formula for parallel transport is introduced as well.Source: §A.4.1 · pp. 17–18

A.4.2 POINCAR´E BALL

The Poincaré-ball section relates its distance to gyrospace distance and describes its limiting relation to Euclidean distance. It also notes that the metric varies with position and connects the model to stereographic projection.

A.4.2 POINCAR´E BALL

  • The Poincaré distance equals gyrospace distance, and the Poincaré gyrospace distance converges to Euclidean distance up to a constant as K approaches 0 from below.The distance equivalence is stated for K < 0 and pairs of points in the Poincaré ball.Source: §A.4.2 · pp. 19–20
  • Unlike in the hyperboloid, the Poincaré-ball inner product at a point changes with that point.Source: §A.4.2 · p. 20

A.5.1 HYPERSPHERE

This appendix develops geometric operations for the hypersphere and describes how its geometry behaves as curvature approaches zero. It also notes a limitation of the spherical model for learning curvature sign agnostically.

A.5.1 HYPERSPHERE

  • The sphere's radius satisfies R^2 = 1/K, and this divergence makes the model unsuitable for learning sign-agnostic curvatures.Source: §A.5.1 · p. 21
  • Ambient vectors can be projected onto the hypersphere by normalization, and a missing coordinate can be computed from the sphere constraint.The coordinate relation is stated as ⟨x, x⟩ = 1/K; the missing-coordinate construction is useful for projecting tangent-space points at µ0 onto the manifold.Source: §A.5.1 · p. 21
  • As K approaches 0 from above, points on the sphere diverge while the sphere grows flatter away from the coordinate origin.The north pole is given as an example of a point that diverges.Source: §A.5.1 · p. 21
  • The section derives spherical exponential and logarithmic maps and parallel transport, using Euclidean trigonometric functions in the spherical formulas.Source: §A.5.1 · pp. 21–22

A.5.2 PROJECTED HYPERSPHERE

This section presents geometric operations for the projected hypersphere and establishes limiting relationships with Euclidean geometry. It also identifies a limitation of its stereographic projection and a position-dependent inner product.

A.5.2 PROJECTED HYPERSPHERE

  • The stereographic projection is not a homeomorphism between the sphere and R^n because it is undefined at the point opposite µ0.Adding a point at infinity and mapping it to ρK(µ0) would make it a homeomorphism.Source: §A.5.2 · p. 23
  • As K approaches 0 from above, the projected spherical distance converges to Euclidean distance up to a constant.The spherical projected distance also equals the gyrospace distance for positive K.Source: §A.5.2 · pp. 23–24
  • The section derives projected-sphere exponential maps and parallel transport, including a special case at the origin.Source: §A.5.2 · pp. 24–25
  • The projected sphere's inner product ⟨·,·⟩x changes with x, unlike the hypersphere's.Source: §A.5.2 · p. 25

A.6 MISCELLANEOUS PROPERTIES

The appendix shows that several operations in constant-curvature spaces approach their Euclidean counterparts as curvature tends to zero. It also covers wrapped-normal densities across the considered spaces and situates the framework among related geometric VAE work.

A.6 MISCELLANEOUS PROPERTIES

  • As K tends to zero, Möbius addition, logarithmic and exponential maps, and gyration converge to their Euclidean counterparts or the identity.The stated limits apply to the Poincaré ball or projected sphere; the addition result is noted to work from both sides of zero curvature.Source: §A.6 · pp. 25–28

B.1 WRAPPED NORMAL DISTRIBUTIONS

  • The appendix gives wrapped-normal probability-density results for hyperbolic, spherical, Poincaré-ball, and projected-sphere spaces.The cited prior work established a related hyperbolic result for K = 1, while this treatment does not assume unit radius.Source: §B.1 · pp. 28, 30, 32, 33

C RELATED WORK

  • The paper unifies geometric VAE approaches in a product-of-spaces framework, while generalizing Normal distributions on manifolds remains non-trivial.Prior spherical approaches use von Mises-Fischer distributions, while cited hyperbolic approaches use Wrapped Normal distributions.Source: C RELATED WORK · p. 34

D EXTENDED FUTURE WORK

The authors identify graph-data evaluation and run-to-run variability as directions for further work. They also report optimization difficulties in spherical spaces and suggest more flexible variational distributions and adversarial training as future extensions.

D EXTENDED FUTURE WORK

  • The authors propose testing mixed-curvature VAEs on graph tasks such as social-network link prediction and investigating relatively large run-to-run variance.They note that variance also affects Euclidean VAEs, though less flagrantly.Source: D EXTENDED FUTURE WORK · p. 34
  • Spherical models are reported as harder to optimize and more prone to divergence; subdividing high-dimensional spaces helps in some cases but is not a conclusive rule.Some subdivided or product-space configurations also diverged frequently.Source: D EXTENDED FUTURE WORK · p. 35
  • The authors suggest normalizing flows for greater latent-distribution flexibility at added computational cost, and extending the geometric models to GANs or adversarially trained autoencoders.Source: D EXTENDED FUTURE WORK · p. 35

E EXTENDED RESULTS

The extended results cover learned curvature, reconstruction quality, latent-space samples and visualizations, and tabulated results across datasets and latent-space configurations.

E EXTENDED RESULTS

  • Figure 1 tracks learned curvature across epochs, with standard deviation, for a dim-6 diagonal-covariance model on MNIST.The figure reports the curvature trajectories rather than a numerical benchmark comparison.Source: Figure 1 · p. 35
  • Tables 6 and 7 summarize mean and standard deviation for dim-6 spherical-covariance results on BDP and MNIST.The captions identify the datasets and covariance setting but provide no cell values.Source: Table 6 · p. 36, Table 7 · p. 37
  • Tables 8–10 summarize diagonal-covariance MNIST results at latent dimensions 6, 12, and 72.The supplied captions do not expose the table values.Source: Table 8 · p. 38, Table 9 · p. 39, Table 10 · p. 40
  • Figure 2 qualitatively compares reconstruction quality from selected well-performing models on MNIST test digits.The caption does not identify a winning model.Source: Figure 2 · p. 38
  • Figures 3 and 4 show MNIST latent-space samples and visualizations, respectively, including a grid search around zero in one component and several model configurations.Figure 4 depicts E2×H2×S2, E6, H6, and S6, using projections for curved spaces and PCA where applicable.Source: Figure 3 · p. 41, Figure 4 · p. 42
  • Tables 11–13 report diagonal-covariance results for Omniglot at dimensions 6 and 72, and CIFAR at dimension 6.Table 13 notes that nan standard deviations indicate an unstable repeated experiment without a meaningful spread estimate.Source: Table 11 · p. 43, Table 12 · p. 44, Table 13 · p. 44
Loading 1911.08411v2…