PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Space Oddity: on the Curvature of Deep Generative Models

Georgios Arvanitidis, Lars Kai Hansen, Søren Hauberg

arXiv:1710.11379v3stat.ML

TL;DR

Deep generative models distort Euclidean latent-space geometry because nonlinear generators map latent points onto curved data manifolds. The paper characterizes this distortion with a stochastic Riemannian metric and shows that geometry-aware distances, interpolations, distributions, sampling, and clustering improve, while proposing a variance architecture with better estimates. The formalism is demonstrated on VAEs and is presented as extensible to other generative models.

  • Problem

    Nonlinear generators make Euclidean latent-space distances and interpolations misleading, while existing generator variance estimates can be poor, especially away from observed data.

  • Method

    The paper models generators as surfaces and derives a stochastic Riemannian metric from their mean and variance functions, adding a variance network that extrapolates precision toward zero.

  • Results

    Geometry-aware distances produce smoother interpolations, better-aligned clustering, meaningful latent distributions, and higher-quality samples than conventional Euclidean geometry in the reported experiments.

  • Takeaways & Limitations

    The latent space can reveal data-manifold geometry and support more relevant distances, uncertainty estimates, clustering, interpolation, and sampling.

  • Takeaways & Limitations

    The analysis assumes sufficiently smooth generator functions, and its extension to more sophisticated generative models is presented as future scope.

Abstract

from arXiv · show

Deep generative models provide a systematic way to learn nonlinear data distributions, through a set of latent variables and a nonlinear "generator" function that maps latent points into the input space. The nonlinearity of the generator imply that the latent space gives a distorted view of the input space. Under mild conditions, we show that this distortion can be characterized by a stochastic Riemannian metric, and demonstrate that distances and interpolants are significantly improved under this metric. This in turn improves probability distributions, sampling algorithms and clustering in the latent space. Our geometric analysis further reveals that current generators provide poor variance estimates and we propose a new generator architecture with vastly improved variance estimates. Results are demonstrated on convolutional and fully connected variational autoencoders, but the formalism easily generalize to other deep generative models.

1 INTRODUCTION

Deep generative models efficiently represent data manifolds with low-dimensional latent variables, but nonlinear generators distort latent-space distances and interpolants. The paper introduces a geometric treatment of this distortion using a stochastic Riemannian metric.

  • Low-dimensional latent variables and flexible neural generators efficiently represent useful distributions over underlying data manifolds.
  • MNIST latent representations can make same-class points appear farther apart than opposite-class points until the latent-space distortion is corrected.
  • Latent distances lack physical units and depend on network specifics, motivating infinitesimal input-space distances governed locally by the generator Jacobian.
  • Shortest curves on the generator surface generally differ from straight latent-space lines assumed by the Euclidean metric.
  • The paper develops a stochastic Riemannian metric, improves distances and interpolations, and examines latent distributions, clustering, and variance estimation.

2 THE VARIATIONAL AUTOENCODERS ACTING AS THE GENERATOR

A VAE uses an encoder to represent observations in a latent space and a decoder to reconstruct them. Its stochastic generator models reconstruction uncertainty, while training maximizes an evidence lower bound.

  • A VAE combines an encoder that learns latent codes with a decoder that reconstructs input data from those codes.
  • The generator mean maps latent variables to a data-space surface, while its variance function captures reconstruction error or uncertainty.
  • Because the exact posterior is intractable, the VAE approximates it with a Gaussian variational distribution produced by an inference network.
  • The decoder’s smoothness depends on the activation functions composing its linear maps and nonlinearities.
  • The model parameters are optimized by maximizing the ELBO using gradient-based optimization and the reparameterization trick.

3 SURFACES AS THE FOUNDATION OF GENERATIVE MODELS

A smooth generator maps a low-dimensional latent space to a surface in the input space. The generator Jacobian induces a locally varying metric, allowing surface lengths, distances, and shortest curves to be computed in latent coordinates.

  • A smooth deterministic generator x = f(z) represents a low-dimensional latent parametrization of a surface embedded in the input space.
  • The Jacobian provides local input-space directions, while det(J⊺J) measures the volume of an infinitesimal region.
  • Mapping a latent curve through the generator measures its length in input space rather than with an unprincipled latent-space metric.
  • The matrix J⊺J is symmetric positive definite and acts as a local Mahalanobis distance, defining a smoothly varying Riemannian inner product.
  • Shortest surface curves define distances between latent points and satisfy differential-geometric ordinary differential equations solved numerically.

4 THE GEOMETRY OF STOCHASTIC GENERATORS

The stochastic generator induces an expected Riemannian metric that captures distortion from both mean and variance, enabling geometry-aware distances while exposing poor variance extrapolation. An RBF-based precision model makes variance large outside data support and improves the resulting geometric volume measure.

  • The stochastic metric: The stochastic generator induces a stochastic Riemannian metric whose expected value can approximate the underlying metric in high data dimensions.The expected metric combines Jacobian terms from the mean and variance functions.
  • The stochastic metric: The variance contribution enlarges distances in uncertain latent regions, causing shortest paths to avoid them without additional learning.The metric is computed directly from the generator.
  • Variance-function geometry: Standard variance networks can produce arbitrary, nonrepresentative estimates away from training data, including inconsistent high and low variance regions.Constant unit variance is also common in publicly available VAE implementations.
  • Variance-function geometry: The proposed architecture models inverse variance with an RBF network whose precision extrapolates toward zero, ensuring larger variance outside the data support.Positive weights ensure positive precision, while centers and bandwidths define the radial basis functions.
  • Variance-function geometry: The proposed variance model captures data trends in the geometric volume measure det(M_z), unlike the standard variance model.The volume measure represents the volume of an infinitesimal input-space area.

5 EMPIRICAL RESULTS

Experiments evaluate Riemannian geometry against Euclidean geometry for distances, interpolations, latent distributions, sampling, and random walks. Across these tests, the geometric approach better reflects data structure and produces smoother or more manifold-consistent behavior.

  • Experimental setup: The experiments first train a VAE and then apply its induced Riemannian metric to compare geometric and Euclidean methods.The evaluation covers clustering, interpolation, probability distributions, sampling, and random walks.
  • 5.1 Meaningful distances: Riemannian distances are more useful than Euclidean distances for k-means clustering of MNIST latent representations.Euclidean k-means finds little structure because the latent points roughly follow a unit Gaussian, whereas the Riemannian distance better reflects the underlying organization.
  • 5.2 Meaningful interpolations: Riemannian interpolation produces smoother changes in generated images than Euclidean interpolation when transitioning between MNIST digit classes.The Euclidean interpolation changes abruptly between classes, while the Riemannian interpolant changes more smoothly.
  • 5.3 Latent probability distributions: Riemannian LAND mixtures reveal underlying MNIST clusters and generate higher-quality samples than Euclidean normal mixtures.The Euclidean model produces several blurry samples in regions where the generator was not trained, while pairwise Riemannian distances also reveal the clusters.
  • 5.4 Random walks: Geometrically informed random walks stay on the data manifold for much longer runs than Euclidean random walks.In the unrestricted Brownian-motion experiment, the Euclidean walk moves freely, whereas the Riemannian walk remains within the data support.

6 RELATED WORK

The paper situates its contribution among generative-model, neural-network-geometry, and latent-variable Riemannian-geometry research. Its metric is derived directly from the generator rather than learned separately.

  • Generative models: The framework is developed for VAEs but extends to related generative models with flexible approximate posteriors and explicit generators, including GANs.The authors present VAEs as the main case while identifying broader applicability to related models.
  • Geometry in neural networks: Prior neural-network geometry work uses Jacobians to study smoothness, local generalization, regularization, and invariances.The paper connects its geometric treatment to earlier Jacobian-based analyses of learned representations and functions.
  • Riemannian geometry: Unlike approaches that explicitly learn a metric, this paper derives its Riemannian metric from the generator without extra learning once the generator is available.The comparison also distinguishes the approach from Gaussian-process latent-variable models, whose computational complexity raises practical concerns.

7 DISCUSSION AND FURTHER EXTENSIONS

The discussion frames latent space as a compressed, flattened representation whose underlying data-manifold geometry can be more complex. The proposed geometric view improves uncertainty estimates and several latent-space operations, while extending naturally to richer models.

  • Discussion: Latent space is a compressed and flattened version of the data manifold, but the manifold’s actual geometry can be more complex than it appears.The paper uses this observation to motivate proper geometries for generative models.
  • Discussion: The proposed variance network provides meaningful uncertainty estimates while regularizing the geometry.The geometric analysis identifies shortcomings in traditional variance approximations and motivates the new architecture.
  • Discussion: Geometrically informed distances improve alignment with ground-truth labels, interpolation smoothness, mixture-component structure, and random-walk manifold adherence.These consequences are reported for k-means, interpolants, LAND mixtures, and latent-space random walks.
  • Further extensions: The analysis extends to sophisticated generative models whose latent spaces may have more flexible nonlinear structures.The authors identify incorporating geometry during generative-model learning as an open question.

A THE DERIVATION OF THE GEODESIC DIFFERENTIAL EQUATION

The appendix derives the geodesic differential equation by reformulating shortest-path optimization through curve length and energy, then expressing the resulting system in matrix and tensor form.

  • Geodesic optimization: Shortest paths on a Riemannian manifold are obtained by optimizing a path functional, with curve energy providing an alternative formulation.The derivation moves from shortest-path length to an equivalent energy-minimization problem.
  • Matrix formulation: The derivation expands the metric inner product along a curve and uses vectorization and Kronecker products to express the terms explicitly.The appendix explains that parenthesized indices select vector or matrix elements, while vec stacks matrix columns.
  • Differential equation: The final result is a system of second-order ordinary differential equations for the geodesic curve.The appendix separately relates this system to the classical geodesic equation and its Christoffel-symbol representation.
  • Coefficient correspondence: The coefficient matrix used in the derived ODE system is not itself the matrix of true Christoffel symbols, although symmetry yields a relation between them.The appendix notes that the coefficient matrix can be nonsymmetric while the corresponding Christoffel symbols retain the relevant symmetry.

B THE DERIVATION OF THE RIEMANNIAN METRIC

The stochastic generator induces a random latent-space metric through its Jacobian. Taking its expectation separates deterministic mean-function geometry from variance-function contributions.

  • The stochastic generator is f(z) = µ(z) + σ(z) ⊙ϵ, combining mean and variance functions with random noise.
  • Its Jacobian decomposes into a mean-function term A and a noise-dependent variance term B.
  • The induced latent-space metric is M_z = J_z^⊺J_z, with randomness arising from ϵ.
  • The cross terms vanish because the noise has zero expectation, leaving the expected metric determined by the mean and variance components.

C INFLUENCE OF VARIANCE ON THE MARGINAL LIKELIHOOD

The experiment compares two variance functions while holding the VAE encoder and decoder mean function fixed. The proposed RBF variance model achieves higher test marginal likelihood by assigning high variance away from encoded data.

  • The VAE experiment trains standard and RBF variance functions after fixing the encoder and decoder mean function, isolating variance effects.
  • -50.34 mean log-marginal likelihood was achieved by the proposed RBF model versus -68.25 for the standard variance function, using S = 10000 samples.
  • The standard variance network behaves arbitrarily where no encoded data exist, whereas the RBF model assigns large variance there.
  • The two models produce similar marginal likelihoods in high-density regions but differ significantly elsewhere.
  • The RBF model gives mixture components high variance in data-sparse regions, assigning minimal density there and attaining higher marginal likelihood elsewhere.

D IMPLEMENTATION DETAILS FOR THE EXPERIMENTS

The experiments use geometry-aware VAE training, RBF variance models, and numerical geodesic or manifold-sampling procedures. Implementation choices include shared layers, regularization, positive variance constraints, and dataset-specific architectures.

  • Training procedure: Geometry-aware VAE training first optimizes µφ, σφ, and µθ while keeping σψ fixed, then trains σψ using the proposed procedure.
  • Experiments 5.1–5.3: The fully connected experiments use an RBF βψ model with 64 centers, parameter a = 2, and L2 regularization equal to 1e−5.
  • Experiments 5.1–5.3: After training, geodesics are computed by solving Eq. 7 numerically, and the LAND mixture model is fitted as described in prior work.
  • Experiment 5.4: The convolutional experiment also uses an RBF model with 64 centers and a = 2 for βψ.
  • Experiment 5.4: The convolutional variance network expands an initial RBF-generated variance image through deconvolution layers to produce per-pixel, per-channel variances.
  • Experiment 5.4: Convolutional and deconvolutional filters are clipped to R+ during training to ensure positive variance.
  • Manifold sampling: Brownian-motion sampling starts from z, uses stepsize s and Ns steps, and updates positions using eigendecompositions of the metric tensor M(·).
Loading 1710.11379v3…