Source-linked AI summary
The Riemannian Geometry of Deep Generative Models
Hang Shao, Abhishek Kumar, P. Thomas Fletcher
TL;DR
Deep generative models define nonlinear manifolds whose geometry and intrinsic distances require characterization. The paper develops geodesic and parallel-translation algorithms, finding surprisingly little curvature in VAE manifolds learned from real images, so latent-space straight lines closely approximate geodesics.
Problem
The geometry of low-dimensional manifolds underlying high-dimensional observations generated by deep models remains to be characterized, despite the manifold hypothesis’ importance.
Method
The paper uses the generator’s induced Riemannian metric to compute geodesics and parallel-translate tangent vectors for manifold interpolation and analogies.
Results
VAE manifolds learned from CelebA and SVHN images have surprisingly little curvature, making latent-space straight lines relatively close to manifold geodesics.
Takeaways & Limitations
Latent-space traversal can produce visually plausible changes because curvilinear distances in the data metric are roughly preserved.
Abstract
from arXiv · showhide
Deep generative models learn a mapping from a low dimensional latent space to a high-dimensional data space. Under certain regularity conditions, these models parameterize nonlinear manifolds in the data space. In this paper, we investigate the Riemannian geometry of these generated manifolds. First, we develop efficient algorithms for computing geodesic curves, which provide an intrinsic notion of distance between points on the manifold. Second, we develop an algorithm for parallel translation of a tangent vector along a path on the manifold. We show how parallel translation can be used to generate analogies, i.e., to transport a change in one data point into a semantically similar change of another data point. Our experiments on real image data show that the manifolds learned by deep generative models, while nonlinear, are surprisingly close to zero curvature. The practical implication is that linear paths in the latent space closely approximate geodesics on the generated manifold. However, further investigation into this phenomenon is warranted, to identify if there are other architectures or datasets where curvature plays a more prominent role. We believe that exploring the Riemannian geometry of deep generative models, using the tools developed in this paper, will be an important step in understanding the high-dimensional, nonlinear spaces these models learn.
1. Introduction
The paper studies the Riemannian geometry of manifolds learned by deep generative models, whose generators map low-dimensional latent coordinates into high-dimensional data space. It introduces methods for geodesic paths and parallel translation to measure intrinsic distances and construct analogies on these manifolds.
- Motivation: The manifold hypothesis motivates modeling high-dimensional sensory observations as concentrated around lower-dimensional manifolds.This hypothesis underlies prior work in unsupervised and semi-supervised learning.
- Motivation: Deep generative models, including VAEs, GANs, and PixelCNN, learn mappings from lower-dimensional latent spaces to high-dimensional observations and generate high-quality images.Their learned generators can therefore be viewed as approximating data manifolds.
- Problem: The paper addresses the previously unexplored metric properties of manifolds generated by deep generative models.Earlier work used learned tangent directions to estimate smooth invariances for classification, but did not examine the manifolds’ metric structure.
- Contributions: The authors propose an algorithm to compute geodesic paths that interpolate between generated data points with minimal change while remaining on the manifold.The arclength of a minimal geodesic path provides an intrinsic distance between points on the generated manifold.
- Contributions: Parallel translation continuously moves a tangent vector along a path with minimal change while keeping it tangent, enabling analogies between manifold points.An analogy transports the change from a to b, represented by a geodesic segment, and applies it to a third point c.
2. Deep Generative Models as Manifolds
Deep generative models map a low-dimensional latent space into a high-dimensional data space, and under rank and smoothness conditions their image forms a smooth, d-dimensional immersed manifold. The model’s Jacobian maps latent tangent vectors to manifold tangent vectors, while the induced Riemannian metric supplies inner products on those tangent spaces.
- Latent-to-data mapping: A deep generative model maps low-dimensional latent coordinates Z ⊆ R^d into a high-dimensional data space X ⊆ R^D, typically with d ≪ D.The mapping is written g : Z → X.
- Manifold conditions: Under suitable conditions, the image of g is a smooth, d-dimensional immersed manifold whose latent linear coordinates become curvilinear coordinates on the manifold.The image is smooth when the Jacobian has rank d at every z ∈ Z.
- Manifold conditions: Sufficient conditions for the manifold property are a smooth monotonic activation function and maximal-rank weight matrices, although the latter must be checked after training.Maximal rank of every weight matrix is sufficient but not necessary; the more general Jacobian rank condition is infeasible to verify at every input.
- Riemannian structure: The Jacobian J_g(z) linearly maps tangent vectors from the latent tangent space T_zZ to the manifold tangent space T_g(z)M and is computed through backpropagation.In practice, J_g(z) is the d × D partial derivative matrix of g.
- Riemannian structure: The induced Riemannian metric defines an inner product between tangent vectors in each tangent space, providing the geometric structure used on the generated manifold.Curvature measures how the metric deviates from being Euclidean; zero curvature does not imply that the manifold is linear.
3. Riemannian Geometry Computations
This section develops three algorithms for Riemannian computations on manifolds represented by deep generative networks: geodesic interpolation, parallel translation, and geodesic shooting. It defines the metric in latent coordinates and presents numerical methods that avoid expensive Christoffel-symbol calculations.
- Overview: The section develops algorithms for geodesic interpolation, parallel translation, and geodesic shooting on manifolds represented by deep generative networks.These procedures operate on objects defined in latent coordinate space, with corresponding counterparts on the generated manifold.
- Metric and geodesics: The latent-space Riemannian metric is G(z) = Jg(z)T Jg(z), and tangent-vector inner products are computed as uT G(z)v.The metric is a symmetric, positive definite matrix field defined at each latent coordinate z.
- Metric and geodesics: Geodesics locally minimize arc length by minimizing an energy functional, yielding constant-speed parameterizations.The geodesic equation follows from varying this energy functional and can be solved by numerical integration.
- Geodesic path computation: Christoffel-symbol integration requires second derivatives of the generator and inversion of G, so discrete energy optimization avoids these expensive calculations.The discrete geodesic method fixes endpoints and applies gradient descent to the intermediate curve coordinates.
- Geodesic path computation: For models with an encoder h satisfying h(g(z)) = z, the Jacobian Jh can replace the generator Jacobian in the modified geodesic gradient.This direction has the same initial movement and fixed points as gradient descent while being faster to compute.
- Parallel translation and geodesic shooting: Parallel translation repeatedly moves a tangent vector between successive points and corrects it into each tangent space using SVD-based projection and rescaling.Geodesic shooting uses parallel translation to update velocity after each small step, producing a geodesic from an initial point and velocity.
4. Experiments
Experiments evaluate geodesic interpolation, means, distance embeddings, and analogies on a synthetic hyperbolic paraboloid and real CelebA and SVHN images. Across these studies, geodesic results are often close to linear latent-space results, while geodesic paths can be shorter and group similar images slightly more tightly.
- Synthetic manifold: On a VAE-learned hyperbolic paraboloid, the geodesic curve is about 35% shorter in arclength than the linear curve, despite tracing a longer latent-space path.The experiment trains a two-dimensional-latent VAE on 50k sampled manifold points and compares paths between two mapped points.
- Real-image interpolation: For CelebA and SVHN, geodesic interpolation produces shorter manifold arclengths than linear interpolation, but the difference is less pronounced than in the synthetic experiment.Images are sampled along linear and geodesic curves, with points equally spaced in latent space.
- Fréchet means: For four 100-image CelebA groups defined by hair-color and mouth-state attributes, Fréchet means look similar to linear latent-space means, with subtle differences.The Fréchet mean minimizes summed squared geodesic distances and is optimized by gradient descent in latent coordinates.
- Distance structure: Geodesic distances group similar images slightly closer together than linear distances, and their MDS embedding appears slightly more tightly concentrated.The MDS analysis compares embeddings based on geodesic and linear distance matrices.
- Analogies: Linear analogies are visually quite close to geodesic analogies, with subtle differences, when transferring hair and mouth-state changes between CelebA images.The analogy uses geodesic means for blond-hair groups with closed and open mouths, while the linear answer is y = g(zb −za + zc).
5. Conclusion
The paper introduces methods for exploring the Riemannian geometry of manifolds learned by deep generative models and finds surprisingly little curvature in manifolds representing real image data. Consequently, straight latent-space lines are relatively close to manifold geodesics, potentially explaining visually plausible latent traversals.
- The paper introduces methods for exploring the Riemannian geometry of manifolds learned by deep generative models.
- Experiments show that manifolds representing real image data have surprisingly little curvature.
- Consequently, straight lines in latent space are relatively close to geodesic curves on the manifold.
- This near-geodesic behavior may explain why latent-space traversal produces visually plausible changes in generated data.Curvilinear distances in the original data metric are roughly preserved.
- The experiments were limited to a single type of deep generative model.