PaperlayerView exact revision on arXiv

Source-linked AI summary

Metrics for Deep Generative Models

Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, Patrick van der Smagt

arXiv:1711.01204v2stat.MLcs.LG

TL;DR

Euclidean distance in latent space can misrepresent similarity because separated observation-space manifolds may be placed next to each other in latent space. The paper uses generator-induced Riemannian geometry to measure shortest paths, and reports applying the resulting metric as an alternative that accounts for the underlying manifold.

  • Problem

    Euclidean latent distances may not reflect data similarity because separated observation-space manifolds can be placed next to each other in latent space.

    Source: §2 · p. 2

  • Method

    The method treats latent space as a Riemannian manifold and approximates shortest paths using a parameterized curve whose length accounts for generator-induced distortions.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    The Riemannian distance metric was successfully applied as an alternative metric accounting for the underlying manifold, with SVD used to produce shorter manifold distances than Euclidean distance.

    Source: §5 · p. 8

  • Takeaways & Limitations

    For two-dimensional latent spaces, the magnification factor visualizes the generative model’s distortion of infinitesimal areas, and the metric can support robot path planning from learned skills.

    Source: §1 · p. 2, §5 · p. 8

    Future work includes making the distance metric easier to use and applying it to models with dynamics.

    Source: §5 · p. 8

Abstract

from arXiv · show

Neural samplers such as variational autoencoders (VAEs) or generative adversarial networks (GANs) approximate distributions by transforming samples from a simple random source---the latent space---to samples from a more complex distribution represented by a dataset. While the manifold hypothesis implies that the density induced by a dataset contains large regions of low density, the training criterions of VAEs and GANs will make the latent space densely covered. Consequently points that are separated by low-density regions in observation space will be pushed together in latent space, making stationary distances poor proxies for similarity. We transfer ideas from Riemannian geometry to this setting, letting the distance between two points be the shortest path on a Riemannian manifold induced by the transformation. The method yields a principled distance measure, provides a tool for visual inspection of deep generative models, and an alternative to linear interpolation in latent space. In addition, it can be applied for robot movement generalization using previously learned skills. The method is evaluated on a synthetic dataset with known ground truth; on a simulated robot arm dataset; on human motion capture data; and on a generative model of handwritten digits.

1 Introduction

High-dimensional distance measures are difficult to choose or learn, while neural samplers create latent spaces where Euclidean distance can miss the structure of the generated data. This work transfers Riemannian shortest-path distances to neural samplers and describes applications including robot path planning.

1 Introduction

  • Distance measures in high-dimensional spaces require careful metric choice, and distances can become less meaningful as dimensionality increases.Prior metric-learning approaches include linear, nonlinear, and recurrent transformations [Xing et al., 2003, Weinberger et al., 2006, Davis et al., 2007, Kulis et al., 2013, Salakhutdinov and Hinton, 2007, Bayer et al., 2012].Source: §1 · p. 1
  • GANs and VAEs [Kingma and Welling, 2013, Rezende et al., 2014] are two distinct neural-sampling approaches for probabilistic modeling.Prior manifold work includes geodesic interpolation and Principal Geodesic Analysis [Noakes et al., 1989, Crouch and Leite, 1995, Fletcher et al., 2004].Source: §1 · p. 1
  • The method transfers Riemannian shortest-path distances to neural samplers, parameterizes the paths, and evaluates them across high-dimensional datasets.The approach follows Tosi et al. [2014].Source: §1 · p. 2
  • In robotics, the learned Riemannian metric can support path planning along data manifolds using previously learned skills.The motivation includes natural motion exploration from expert demonstrations.Source: §1 · p. 2

2 Using Riemannian geometry in generative latent variable models

Euclidean distance in latent space can fail to reflect similarity in observation space because separated data manifolds may be placed next to each other in latent space. The section motivates an IWAE-based model and Riemannian geometry to account for changes induced by the generator.

2 Using Riemannian geometry in generative latent variable models

  • The section frames the goal as measuring similarity between generated data points, rather than assuming latent and observation-space distances are proportional.The proportional-distance assumption is identified as a fallacy for these latent-variable models.Source: §2 · p. 2
  • Separated observation-space manifolds may lie adjacent in latent space, where likelihood discontinuities compensate for the lack of low-density regions.Consequently, Euclidean latent distances do not reliably track observation-space distances.Source: §2 · p. 2

2.1 Importance-weighted autoencoder

  • The approach uses importance-weighted autoencoders (IWAEs) for nonlinear latent-variable modeling and Riemannian geometry to define distances that account for likelihood changes.IWAEs use importance sampling for a tighter lower bound and are presented as a basis for an accurate generative model [Burda et al., 2015, Cremer et al., 2017].Source: §2 · p. 2, §2.1 · pp. 2–3

2.2 Riemannian geometry

  • A Riemannian manifold assigns a metric tensor to each point, and the generator’s Jacobian induces the metric tensor G = J^T J.This tensor measures curve length while accounting for transformation into observation space.Source: §2.2 · p. 3

3 Approximating the geodesic

The method approximates a shortest path by optimizing a parameterized latent-space curve according to its length under the generator-induced metric. It enforces curve endpoints and smooths the metric tensor through SVD to reduce manifold distances relative to Euclidean distance.

3 Approximating the geodesic

  • Geodesics are approximated by optimizing a neural-network-parametrized latent curve to minimize length measured with generator-induced distortions.The curve length is approximated at equidistant samples, with each summand interpreted as the induced velocity.Source: §3 · p. 3

3.1 Dealing with boundary constraints

  • The predicted curve is shifted and rescaled to satisfy its specified start and end points, simplifying optimization to minimizing its length.The endpoints are z0 and z1.Source: §3 · p. 3, §3.1 · p. 3

3.2 Smoothing the metric tensor

  • The smoothing penalty is intended to make the geodesic follow the data manifold by ensuring manifold distance is smaller than Euclidean distance.The penalty is applied as post-processing of the constrained optimization.Source: §3.2 · pp. 3–4
  • SVD-based smoothing nonlinearly rescales singular values, weakening smaller components and reducing manifold distance relative to Euclidean distance.A higher λ_s increases the difference between Euclidean interpolation and the path along the manifold.Source: §3.2 · p. 4

4 Experiments

The experiments evaluate geodesic interpolation and distance measures across pendulum, MNIST, robot-arm, and human-motion data. They also examine training choices and visualizations of the learned metric.

4.1 Training

  • The IWAE uses a Gaussian prior and diagonal-Gaussian inference model, with Bernoulli or Gaussian likelihoods depending on the experiment.The likelihood parameters are functions of latent variables represented by neural networks.Source: §4.1 · p. 4

4.2 Visualization

  • For two-dimensional latent spaces, the magnification factor visualizes how mapping points from the Riemannian manifold to Euclidean space changes infinitesimal volume.The evaluation uses it to visualize metric tensors.Source: §4.2 · p. 4

4 Experiments

  • Geodesics produced more uniform pendulum rotation, smoother MNIST transitions, outperformed Euclidean robot interpolation, and generated natural, smooth walking movements.The comparisons are reported in the pendulum, MNIST, robot-arm, and human-motion evaluations, respectively.Source: §4.3 · p. 5, §4.4 · p. 6, §4.5 · p. 7

4.3 Pendulum

  • For pendulum data, geodesic and data-manifold distances fit one another and vary linearly with observation-space angle; large λs was needed to obtain paths along the data manifold.The comparison used 100 randomly selected pairs with rotation-angle differences in (0, 180] degrees.Source: §4.3 · p. 5

4.4 MNIST

  • On MNIST, geodesic distances separate classes when Euclidean distances do not; Euclidean interpolation crosses four classes versus two for geodesic interpolation.The Euclidean path also produces less smooth reconstruction transitions than the geodesic path.Source: §4.4 · p. 6

4.5 Robot arm

  • For robot motions, geodesic interpolation outperforms Euclidean interpolation without explicitly requiring task-space constraints, and the approach can extend to higher-dimensional joint angles.The comparison is shown using end-effector trajectories in a two-dimensional Cartesian space.Source: §4.5 · p. 7, Figure 9 · p. 7

4.6 Human motion

  • For human motion, the geodesic follows the data manifold to produce natural, smooth walking, while Euclidean interpolation crosses high-magnification areas associated with large movement jumps.In those areas, body poses change little in other parts of the Euclidean path.Source: §4.6 · pp. 7–8

5 Conclusion and future work

Latent-space distances can misrepresent observation-space similarity; the conclusion presents a SVD-smoothed Riemannian metric that accounts for the manifold and supports smoother interpolation.

5 Conclusion and future work

  • In the human-motion samples, one latent-space circle corresponds to two walking steps, ending at the initial pose.Source: Figure 13 · p. 8
  • The Riemannian distance metric accounts for the underlying manifold, while SVD smoothing helps produce shorter manifold paths than Euclidean distances.The metric can also be used for smoother latent-space interpolations.Source: §5 · p. 8
  • For two-dimensional latent spaces, the magnification factor visualizes the magnitude of generative-model distortion of infinitesimal areas.Source: §5 · p. 8
  • Future work includes making the distance metric easier to use and applying it to models with dynamics.Source: §5 · p. 8

A Details of the training procedure

The training procedure pre-trains the curve network on random parametric curves to address local minima, then validates results using both maximum velocity and path length.

A Details of the training procedure

  • Training validation uses maximum velocity and a penalized path length to avoid solutions with narrow velocity spikes but low overall length.The penalty adds λφ times maximum velocity to path length.Source: A Details of the training procedure · p. 10
  • To address local minima, the curve network is pre-trained on random Bézier curves, and the model with the lowest validation value is selected.Each candidate curve network is fitted to a separately sampled curve before optimization of the target loss.Source: A Details of the training procedure · p. 10

B Gradients of piecewise linear activation functions

The path-length gradient requires differentiating the generator Jacobian, which makes gradient-based optimization unavailable with piecewise-linear units.

B Gradients of piecewise linear activation functions

C Gradients of sigmoid, tanh and softplus activation functions

The appendix describes Jacobian calculations for sigmoid, tanh, and softplus activations, and states that the Jacobian derivative is straightforward.

C Gradients of sigmoid, tanh and softplus activation functions

D Experiment setups

The section documents experiment setups across geodesic neural networks and four datasets, alongside shared optimizer and architecture details.

D Experiment setups

Related papers

Loading 1711.01204v2…