PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperspherical Variational Auto-Encoders

Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, Jakub M. Tomczak

arXiv:1804.00891v3stat.MLcs.LG

TL;DR

Gaussian VAE parameterizations can mismatch hyperspherical data and produce instability because Gaussian mass and KL regularization conflict with non-Euclidean latent structure. The paper replaces normal latent distributions with von Mises-Fisher distributions and evaluates S-VAEs against N-VAEs. S-VAEs recover hyperspherical structure more effectively and improve citation-network link prediction, but high-dimensional hyperspherical models may become unstable.

  • Problem

    Gaussian priors and posteriors can mismatch hyperspherical latent structure, producing instability and origin-directed clustering.

  • Method

    The paper replaces normal latent distributions with von Mises-Fisher distributions and extends rejection-sampling reparameterization to train concentration.

  • Results

    S-VAEs outperform N-VAEs in recovering hyperspherical latent structure and significantly improve citation-network link prediction with VGAE.

  • Takeaways & Limitations

    Hyperspherical latent spaces provide a model suited to directional and manifold-structured data, avoiding Gaussian origin pull and directional bias.

  • Takeaways & Limitations

    Manifold discovery is difficult, and hyperspherical models may show unstable behavior in high dimensions, even for m > 20.

Abstract

from arXiv · show

The Variational Auto-Encoder (VAE) is one of the most used unsupervised machine learning models. But although the default choice of a Gaussian distribution for both the prior and posterior represents a mathematically convenient distribution often leading to competitive results, we show that this parameterization fails to model data with a latent hyperspherical structure. To address this issue we propose using a von Mises-Fisher (vMF) distribution instead, leading to a hyperspherical latent space. Through a series of experiments we show how such a hyperspherical VAE, or $\mathcal{S}$-VAE, is more suitable for capturing data with a hyperspherical latent structure, while outperforming a normal, $\mathcal{N}$-VAE, in low dimensions on other data types. Code at http://github.com/nicola-decao/s-vae-tf and https://github.com/nicola-decao/s-vae-pytorch

1 INTRODUCTION

The paper argues that Gaussian latent distributions can fail on hyperspherical data and proposes vMF-based VAEs to model such structure. Experiments compare S-VAEs with N-VAEs across synthetic, MNIST, and citation-network settings.

  • Motivation: Gaussian priors can destabilize learning when data has a latent circle because they concentrate near the origin while KL divergence reconciles incompatible spaces.A two-hidden-unit autoencoder discovers the circle, whereas a normal VAE becomes highly unstable.
  • Motivation: Directional data and normalized text or image representations motivate models that explicitly account for intrinsically spherical structure.Examples include protein structure and observed wind directions.
  • Approach: The proposed S-VAE replaces the normal distribution with a von Mises-Fisher distribution to generate latent representations.Training the concentration parameter requires an extended reparameterization trick combined with rejection sampling.
  • Implementation: The paper provides code implementations for the proposed models.The introduction lists TensorFlow and PyTorch repositories.
  • Experiments: S-VAEs outperform N-VAEs in recovering hyperspherical latent structure and improve link prediction with a Variational Graph AutoEncoder.The experiments also compare both models on unsupervised and semi-supervised MNIST tasks.

2 VARIATIONAL AUTO-ENCODERS

The VAE uses an inference network and approximate posterior to optimize a tractable variational objective, but Euclidean Gaussian spaces can mismatch manifold-structured data. The paper motivates hyperspherical spaces as alternatives that avoid origin bias and better reflect directional geometry.

  • 2.1 FORMULATION: A VAE introduces an inference network qψ(z|x; θ) that outputs an approximate posterior distribution for each data point.This makes inference scalable to larger datasets while approximating the true posterior.
  • 2.1 FORMULATION: The original VAE defines both prior and posterior as normal distributions and estimates the ELBO with Monte Carlo samples generated through reparameterization.Samples are expressed as z = h(θ, ε, x), with noise independent of θ.
  • 2.2 THE LIMITATIONS OF A GAUSSIAN DISTRIBUTION PRIOR: In low dimensions, Gaussian origin gravity pulls cluster centers together, while a uniform prior over the entire hyperplane is not well defined.This can encourage central clustering when data contains multiple classes.
  • 2.2 THE LIMITATIONS OF A GAUSSIAN DISTRIBUTION PRIOR: In high dimensions, Gaussian mass concentrates near a hyperspherical shell, motivating comparison with posteriors defined directly on hyperspheres.The discussion also notes that the Gaussian L2-based definition suffers from the curse of dimensionality.
  • 2.3 BEYOND THE HYPERPLANE: A uniform hyperspherical prior avoids pull toward the origin, allows clusters to spread without directional bias, and makes cosine similarity more meaningful in higher dimensions.This requires abandoning the hyperplanar latent-space assumption.
  • 2.3 BEYOND THE HYPERPLANE: An encoder cannot provide a globally invertible mapping when the data manifold and latent Euclidean space have incompatible dimensions or topology.When D > M, smooth embedding is possible for sufficiently large D, but random latent points may still fall outside the embedded manifold.
  • 2.3 BEYOND THE HYPERPLANE: Forcing a manifold into a posterior supported across all Euclidean latent space can leave regions empty or create instability and discontinuity.Increasing the KL term may push the encoder to occupy the whole latent space, affecting convergence.
  • 2.3 BEYOND THE HYPERPLANE: The paper identifies discovering the data manifold as difficult and presents manifold-based VAE architectures as an initial research direction.Directly specifying a latent space homeomorphic to the data manifold is described as the best option in principle.

3 REPLACING GAUSSIAN WITH VON MISES-FISHER

The paper replaces Gaussian latent distributions with a von Mises-Fisher distribution to represent hyperspherical latent spaces, while extending reparameterization for trainable concentration. It also describes vMF sampling and the geometric behavior of hyperspheres in high dimensions.

  • 3.1 VON MISES-FISHER DISTRIBUTION: The vMF distribution models unit-vector latents using a mean direction µ and concentration κ, with κ = 0 yielding a uniform distribution.
  • 3.2 HYPERSPHERICAL PRIOR: A uniform hyperspherical prior removes directional pull toward the origin, allowing latent clusters to spread across the surface.
  • 3.2 HYPERSPHERICAL PRIOR: The vMF KL term is independent of µ, so µ is optimized only through reconstruction, while κ requires a derived gradient because of the modified Bessel function.
  • 3.3 SAMPLING PROCEDURE: vMF samples are generated by univariate acceptance-rejection sampling followed by an orthogonal transformation mapping the reference direction to µ.
  • 3.4 N-TRANSFORMATION REPARAMETERIZATION TRICK: Rejection-sampling reparameterization is extended with additional parameter-dependent transformations so the vMF concentration can be trained.
  • 3.5 BEHAVIOR IN HIGH DIMENSIONS: The hypersphere surface area decreases toward zero as dimensionality grows, creating a potential source of unstable behavior for hyperspherical models above moderate dimensions.

4 RELATED WORK

Related work includes more flexible posterior and prior constructions, non-Euclidean latent-space methods, and applications of vMF models. The paper distinguishes its approach by combining a hyperspherical latent space with a non-informative prior and learnable concentration.

  • Extending the VAE: Normalizing flows increase posterior flexibility through sequential invertible transformations and are described as orthogonal to the proposed hyperspherical approach.
  • Extending the VAE: Prior-focused alternatives include mixtures of Gaussians, VampPrior, and non-parametric stick-breaking processes, whereas this work uses a non-informative prior.
  • Extending the VAE: A prior vMF sentence-generation model kept concentration κ constant rather than reparameterizing and learning it.
  • Non-Euclidean Latent Space: Other non-Euclidean methods use Riemannian-manifold particles or hyperbolic embeddings, illustrating alternatives to Euclidean latent representations.
  • Hyperspherical Data: vMF models have been applied to clustering, computer vision, natural language processing, and topic modeling.
  • Hyperspherical Data: Spherical constraints have also been associated with regularization and improved clusterability in deep auto-encoders.

5 EXPERIMENTS

Experiments compare S-VAEs with N-VAEs across synthetic hyperspherical recovery, MNIST reconstruction and semi-supervised learning, and citation-network link prediction. S-VAEs recover hyperspherical structure particularly well and generally show strongest advantages in low-dimensional or naturally spherical settings, while performance can weaken in high dimensions or on more complex graphs.

  • Recovering hyperspherical latent representations: The S-VAE almost perfectly recovers the original hyperspherical dataset structure while its posterior samples closely match the prior.By contrast, Gaussian VAEs either concentrate samples at the origin or require a weakened KL term that makes prior-based generation problematic.
  • Evaluation of expressiveness: On MNIST, the S-VAE outperforms the N-VAE in negative reconstruction error across all dimensions and has an LL edge at d = 2, 5, 10.At d = 20, the two models have comparable log-likelihood; increasing dimensionality also makes the Gaussian posterior more expressive than the vMF posterior.
  • Evaluation of expressiveness: S-VAE latent spaces distribute MNIST digits across the available space with continuity, whereas N-VAE representations concentrate mass around the center.The uniform hyperspherical prior removes the Gaussian prior’s origin pull and yields more separable clusters in S2 than in R2.
  • Semi-supervised learning: For semi-supervised MNIST classification, the S-VAE outperforms the N-VAE for M1 in all dimensions up to d = 40, with larger gains when fewer labels are observed.In M1+M2, pure S-VAE or S+N-VAE models generally perform best; S-VAE performance drops when dim z2 = 50, while S+N-VAE reaches 96.7±.1 accuracy.
  • Link prediction on graphs: The S-VAE outperforms the N-VGAE baseline on two of three citation-network datasets by a significant margin.Constraining embeddings to a hypersphere mitigates zero-center instability in dot-product link prediction and improves latent-cluster separation, but the S-VAE converges to a lower score on the larger, more complex Pubmed dataset.

6 CONCLUSION

The S-VAE is presented as a first step toward hyperspherical latent representations, with advantages for hyperspherical data and competitive or improved performance in lower dimensions. The conclusion also identifies higher-dimensional performance and more flexible hyperspherical posteriors as future directions.

  • S-VAEs have a clear advantage over N-VAEs for data residing on a known hyperspherical manifold.
  • In lower dimensions, S-VAEs are competitive with or surpass N-VAEs for data without an obvious hyperspherical latent representation.
  • Changing only the prior and posterior distributions acts as a simple drop-in replacement for improving semi-supervised separability and citation-graph link prediction.
  • Higher-dimensional S-VAE performance remains a future research target, with dynamically learned hypersphere radii proposed as one possible direction.
  • The vMF sampling procedure uses a univariate acceptance-rejection step, while the S2 case can be sampled directly without rejection.
  • Sampling complexity increases with concentration κ but decreases as dimensionality increases.

B KL DIVERGENCE DERIVATION

This section introduces the KL divergence between a von Mises-Fisher distribution and a uniform distribution on the hypersphere, alongside an exponential-integral computation.

  • The derivation considers the KL divergence between q(z|µ, κ), a von Mises-Fisher distribution, and a uniform hyperspherical distribution.
  • The supplied derivation passages provide the target divergence and computational hint but do not state the complete closed-form expression.
  • The derivation notes that Iexp can be used in the resulting calculation.

C PROOF OF LEMMA 2

The proof establishes a transformation-based expectation identity for samples generated from a parameterized univariate variable and an auxiliary random variable.

  • The lemma applies to any measurable function f with ε distributed according to π1(ε|θ)=s(ε)g(h(ε,θ)|θ).
  • An auxiliary variable v is sampled from π2(v), and a parameter-dependent transformation T maps (ω,v;θ) to z.
  • When ω follows g(ω|θ), the transformed variable z follows the target distribution q(z|θ).
  • The proof rewrites the expression using the change of variables ω=h(ε,θ), then applies the corresponding transformation z=T(ω,v;θ).

D REPARAMETRIZATION GRADIENT DERIVATION

This derivation computes gradients of expectations under the reparameterized sampling scheme, separating the reconstruction gradient from a correction term associated with the sampling transformation.

  • The expectation gradient is expressed over ε and v after substituting z=T(h(ε,θ),v;θ).
  • The resulting expression separates a reparameterization term, grep, from a correction term, gcor.
  • The correction term is formed from parameter derivatives of log g and the Jacobian of h.
  • For the VAE objective, the derivation uses f(z)=pφ(x|z) and optimizes unbiased Monte Carlo gradient estimates.
  • The reparameterization gradient is the reconstruction-loss gradient with respect to θ and can be handled by automatic differentiation.
  • Because g and h do not depend on µ, the correction term with respect to µ is zero; subsequent correction calculations concern κ.
  • The appendix also states that hyperspherical surface area reaches a maximum at m=7 and decreases toward zero at high dimensionality.

F.2 EXPERIMENT 5.3

The experiments evaluate hyperspherical and normal latent-space variants across image and graph models using matched architectures and task-specific training settings.

  • Architecture and Hyperparameters: The M1 experiment compares N-VAE and S-VAE using 5-nearest-neighbor classification, with Euclidean distance for N-VAE and geodesic distance for S-VAE.Performance is evaluated with N = [100, 600, 1000] observed labels.
  • Architecture and Hyperparameters: The stacked M1+M2 model uses single-hidden-layer MLPs with 500 units, ReLU activations, and latent dimensions z1 and z2 varied over [5, 10, 50].Training uses early stopping with a 50-epoch validation look-ahead.
  • Architecture and Hyperparameters: The VGAE comparison retains the original architecture while replacing the normal latent-space generation with a von Mises-Fisher distribution.Models are trained for 200 epochs on Cora and Citeseer and 400 epochs on Pubmed using Adam.
  • Architecture and Hyperparameters: Citation-network dataset statistics and selected hyperparameter settings are reported in Tables 6 and 7, respectively.The final hyperparameters were selected by grid search on validation splits.

G VISUALIZATION OF SAMPLES AND LATENT SPACES

The visualizations compare random samples, latent manifolds, and conditional handwriting generations from normal and hyperspherical VAEs. They emphasize differences in latent-space organization and generated style variation.

  • Latent Spaces: In the two-dimensional MNIST manifold, N-VAE representations have a clear center with digits spread around it, whereas S-VAE digits occupy the entire space.The S-VAE manifold also shows continuity from left to right.
  • Conditional Generation: Figure 10 conditionally generates MNIST handwriting styles using M1+M2 with dim(z1) = 50, dim(z2) = 50, and S+N.Each row fixes z2 inferred from a test image while varying the class label y across columns.
Loading 1804.00891v3…