PaperlayerView exact revision on arXiv

Source-linked AI summary

Explorations in Homeomorphic Variational Auto-Encoding

Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, Taco S. Cohen

arXiv:1807.04689v1stat.MLcs.LG

TL;DR

Non-trivial data-manifold topology can make a blob-like latent space unsuitable for continuous, invertible encoding. The paper develops a VAE with Lie-group latent variables, including SO(3), and reports that matching latent and data topology supports continuous, well-behaved representations in its experiments.

  • Problem

    A continuous invertible encoder cannot map a non-trivial data manifold to a blob-like latent region, while a flat higher-dimensional embedding introduces directions that may leave the data manifold.

    Source: §1 · p. 1

  • Method

    The paper extends the reparameterization trick to compact connected Lie groups and develops an SO(3) VAE with a homeomorphic encoder and group-action decoder.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5

  • Results

    Experiments show that topology-matched manifold latents yield continuous, well-behaved latent spaces, and only the correctly structured manifold latent fully solves the more complicated experiment.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Matching the latent-variable topology to the latent data manifold is important for preserving its structure and obtaining a well-behaved latent space.

    Source: §7 · p. 8

    The approach assumes a priori knowledge of the observed data's latent structure; learning arbitrary unknown manifolds remains future work.

    Source: §7 · p. 8

Abstract

from arXiv · show

The manifold hypothesis states that many kinds of high-dimensional data are concentrated near a low-dimensional manifold. If the topology of this data manifold is non-trivial, a continuous encoder network cannot embed it in a one-to-one manner without creating holes of low density in the latent space. This is at odds with the Gaussian prior assumption typically made in Variational Auto-Encoders (VAEs), because the density of a Gaussian concentrates near a blob-like manifold. In this paper we investigate the use of manifold-valued latent variables. Specifically, we focus on the important case of continuously differentiable symmetry groups (Lie groups), such as the group of 3D rotations $\operatorname{SO}(3)$. We show how a VAE with $\operatorname{SO}(3)$-valued latent variables can be constructed, by extending the reparameterization trick to compact connected Lie groups. Our experiments show that choosing manifold-valued latent variables that match the topology of the latent data manifold, is crucial to preserve the topological structure and learn a well-behaved latent space.

1. Introduction

The paper argues that topology-mismatched latent spaces can disrupt continuous, invertible representations, and develops a Lie-group VAE with experiments favoring topology-matched latents. Its approach combines SO(3) reparameterization with a homeomorphic encoder and a group-action decoder.

1. Introduction

  • A non-trivial data-manifold topology prevents a continuous invertible map to a blob-like latent region, while a higher-dimensional flat embedding adds directions that may leave the data manifold.A Gaussian prior concentrates near a blob-like region, which motivates matching latent and data-manifold topology.Source: §1 · p. 1
  • The paper extends reparameterization to compact connected Lie groups, working out SO(3) and the absolute-continuity requirement for its induced measure.The construction also addresses how an encoder can learn a homeomorphism to the posterior mean parameter.Source: §1 · p. 2
  • Only a manifold-valued latent variable with the required topology fully solves the more complicated synthetic experiment; standard Gaussian models show discontinuities in latent trajectories.The experiments use embedded SO(3) and images of rotations of a colored cube.Source: §1 · p. 2
  • The three contributions are an SO(3) reparameterization trick, a homeomorphic encoder for the mean parameter, and a decoder that uses group action to respect group structure.Source: §1 · p. 2

2. Preliminary Concepts

This section introduces the VAE setup and the Lie-group geometry used later, culminating in SO(3) as a compact, connected rotation group that is not homeomorphic to Euclidean space.

2.1. Variational Auto-Encoders

  • Neural-network marginal likelihoods are generally intractable, so VAEs use variational inference, an inference network, and Monte Carlo estimates with the reparameterization trick.The inference network outputs an approximate posterior for each observation, and the objective is the ELBO.Source: §2.1 · p. 2

2.2. Lie Groups and Lie Algebras

  • A Lie group is a smooth manifold with a group product; its Lie algebra is the tangent space at the identity, and the exponential map sends algebra elements to group elements.For compact connected Lie groups such as SO(3), the exponential map is surjective.Source: §2.2 · pp. 2–3

2.3. The group SO(3)

  • SO(3) consists of 3D rotation matrices with R⊤R = I and det(R) = 1, and it is not homeomorphic to R^N.A full rotation around a fixed axis illustrates the topological distinction from Euclidean space.Source: §2.3 · p. 3
  • The SO(3) Lie algebra is represented by skew-symmetric 3×3 matrices and is identified with R^3; its exponential map uses the Rodrigues rotation formula.The map is surjective but not injective.Source: §2.3 · p. 3

3. Reparameterizing SO(3)

The SO(3) reparameterization maps samples from a distribution near the origin in the Lie algebra onto the group, then shifts their location by left multiplication. The paper also addresses the induced density and KL-divergence calculations.

3. Reparameterizing SO(3)

  • The SO(3) sampling path draws an algebra sample, applies the exponential map, and left-multiplies by Rµ to set the distribution's location.This contrasts with the classic Euclidean case, where the exponential step is the identity and location changes by translation.Source: §3 · p. 3, Figure 2.1 · p. 4
  • The measure pushed from R^3 through the exponential map is absolutely continuous with respect to normalized Haar measure on SO(3).This establishes the existence of a density for the constructed group measure.Source: §3 · p. 3
  • The principal-branch logarithm maps group elements back near the origin, and the resulting density may be singular at the identity while still integrating to 1.Source: §3 · p. 3
  • Left invariance of Haar measure permits computing the entropy using the centered distribution, with Monte Carlo samples; for a uniform prior, the cross-entropy simplifies.The entropy expression depends only on Lie-algebra samples, and one sample suffices with mini-batches.Source: §3 · pp. 3–4

4. Encoder and Decoder networks

The section maps inputs to SO(3)-valued posterior means through a topology-preserving encoder and decodes group actions through transformed Fourier modes. The representation-based decoder can encode pose meaningfully and may support disentangling content and pose.

4. Encoder and Decoder networks

  • The networks map inputs to reparameterization parameters Rµ and σ, and map group-valued latent elements back to output predictions.Source: §4 · p. 4

4.1. Homeomorphic Encoder

  • Assuming the data manifold is homeomorphic to SO(3), the encoder learns its inverse on that manifold to preserve its topology.Source: §4.1 · p. 4
  • The encoder factors encµ as π∘f, choosing Y so SO(3) embeds in it and π restricts to an inverse on that embedding.With X and Y Euclidean spaces, the paper states that a continuous f can be approximated by neural networks.Source: §4.1 · pp. 4–5

4.2. Group Action Decoder

  • A spherical signal decoder is an alternative, but parameterizing and projecting its function is highly non-trivial.The proposed decoder can also disentangle content and pose when the encoder generates the Fourier modes; the paper leaves this extension for future work.Source: §4.2 · p. 5
  • Wigner-D matrices transform Fourier modes linearly, so the decoder can apply a group action before image generation while preserving group composition.The modes are fed to an image generative network; Figure 4.1 depicts the equivalent spatial rotation and representation action.Source: §4.2 · p. 5, Figure 4.1 · p. 5

5. Related Work

Related work explores more flexible latent distributions, including non-Gaussian priors and reparameterizable distributions, while group-based distribution work did not focus on reparameterization. Other threads address equivariance and scene rendering, with the latter assuming ground-truth poses.

5. Related Work

  • Prior work explores learned mixture and non-parametric priors, discrete latents, and reparameterized continuous distributions, including hyperspherical von Mises-Fisher latents (Nalisnick & Smyth, 2017; Jang et al., 2017; Maddison et al., 2017; Naesseth et al., 2017; Figurnov et al., 2018).Source: §5 · p. 5
  • Normalizing flows apply sequential invertible transformations to reparameterizable densities, and their framework could be investigated for SO(3) in future work (Rezende & Mohamed, 2015).Source: §5 · p. 6
  • Prior studies define distributions on homogeneous spaces, and harmonic exponential families provide a powerful family there, but these works did not focus on reparameterizable distributions (Cohen & Welling, 2015).Source: §5 · p. 6
  • Scene rendering from multiple poses has been explored, but the cited approach assumes ground-truth poses rather than unsupervised pose learning.Source: §5 · p. 6
  • Equivariant networks operate on spherical and SO(3)-valued data, while quotienting over group orbits can reduce task complexity (Cohen et al., 2018a; Cohen & Welling, 2016; Cohen & Welling, 2017; Worrall et al., 2017; Weiler et al., 2018; Cohen et al., 2018b).Source: §5 · p. 6

6. Experiments

The experiments compare latent-manifold topology, decoder architecture, and SO(3) mean parameterizations on synthetic rotation data and cube images. Across both tasks, S2 × S2 yields continuous encodings, while the sphere-cube results also favor the group action decoder.

6.1. Toy experiment

  • The toy data embed uniformly sampled SO(3) rotations as a 3D submanifold of R64, and models use a learned group-action decoder.The study compares SO(3) models with a 3D Normal mapped by ZYZ-Euler angles and an S3 von Mises-Fisher mapped through unit quaternions.Source: §6.1 · p. 7
  • In the toy experiment, S2 × S2 strongly outperforms competing non-variational autoencoders in reconstruction and is the only model without latent-space discontinuities.The toy results also identify S2 × S2 as the only method learning a continuous latent space along the encoded S1 trajectory.Source: §6.1 · p. 7, Table 1 · p. 7
  • The 3D Normal and SO(3) algebra mean models perform worst; both represent SO(3) as R3 at an intermediate step.The authors report that this flat-space representation is associated with poorer reconstruction and log-likelihood performance and a poorly structured latent space.Source: §6.1 · p. 7

6.2. Sphere-Cube

  • For sphere-cube images, higher-dimensional Normal VAEs achieve competitive metrics but learn unstructured embeddings, whereas the SO(3) S2 × S2 model learns a nearly perfect encoding.The 10-dimensional Normal learns disconnected patches of the data manifold.Source: §6.2 · p. 8
  • On sphere-cube data, only the continuous S2 × S2 encoding achieves low log-likelihood and reconstruction losses relative to other mean parameterizations.Table 3 compares SO(3) mean parameterizations and decoder variants, with both regularizers used.Source: §6.2 · p. 8
  • The group action decoder yields significantly higher performance than the MLP decoder on the sphere-cube task.The authors suggest that the group action decoder encourages structure in the latent space.Source: §6.2 · p. 8

7. Discussion & Conclusion

The paper extends the reparameterization trick to compact connected Lie groups and studies SO(3) as a specific case. Its experiments emphasize topology-matched latent variables, group-action decoding, and careful choice of the posterior mean’s embedding space.

7. Discussion & Conclusion

  • The work extends the reparameterization trick to compact connected Lie groups and details an implementation for SO(3).Experiments support matching latent-variable topology to the data manifold to obtain a continuous, well-behaved latent space.Source: §7 · p. 8
  • The authors report improved latent-space structure with a group action decoder and stress careful selection of the posterior mean’s embedding space.They also identify SO(3) and other known manifold-valued latents as possible additions for model-based reinforcement learning and computer vision.Source: §7 · p. 8
  • The approach assumes a priori knowledge of the observed data’s latent structure; learning arbitrary unknown manifolds is left to future work.Source: §7 · p. 8

A. Additional Figures

The appendix includes visualizations of S1 trajectories and derivations concerning SO(3) pushforward measures and entropy computation. The derivations state an absolute-continuity result and a condition for truncating an entropy sum.

A. Additional Figures

  • Figure A.1 shows reconstructions of an S1 trajectory, with R64 elements mapped to 3D by principal component analysis.Source: Figure A.1 · p. 11
  • Figure A.2 measures encoder variation along an S1 trajectory using the norm of successive latent-code differences.The figure refers readers to Section 6.1 for details.Source: Figure A.2 · p. 11

B. Pushforward Measure SO(3)

  • An absolutely continuous measure on R3 pushed forward through the SO(3) exponential map is absolutely continuous with respect to normalized Haar measure.Source: B. Pushforward Measure SO(3) · p. 11

C. Entropy computation

  • The entropy computation uses samples from the Lie algebra, and its infinite sum can be truncated when the density decays quickly enough, as for exponentially decaying distributions such as the Normal.Source: C. Entropy computation · p. 13

D. Mean parameterization

The mean parameterization must permit a continuous right inverse so the encoder can represent the data manifold homeomorphically. Several maps to SO(3) fail this requirement, while the S2 × S2 construction provides a suitable right inverse.

D. Mean parameterization

  • The algebra parameterization uses the exponential map, but its log-map inverses are discontinuous along a full fixed-axis rotation in SO(3).Thus this parameterization does not provide a continuous inverse globally.Source: D. Mean parameterization · p. 14
  • The quaternion parameterization uses the two-to-one projection from S3 to SO(3), but no embedding right inverse exists.The obstruction follows from recognizing S3 as a non-trivial principal bundle over SO(3).Source: D. Mean parameterization · p. 14
  • The S2 × S1 axis-and-angle map has no continuous right inverse because its π-angle fiber identifies μ and −μ, leaving a hemisphere boundary open.A continuous right inverse would have a closed image, but this intersection is not closed.Source: D. Mean parameterization · p. 14
  • The S2 × S2 construction uses the first two rows of an SO(3) matrix to give a continuous injection with a right inverse.The third row is recoverable as the vector product of the first two.Source: D. Mean parameterization · p. 14

E. Continuity Metric

The continuity metric flags paths with unusually large jumps, while the group-action decoder represents spherical-signal rotations through Fourier modes. Regularizers are added because a suitable mean parameterization and decoder alone may still yield a discontinuous latent space.

E. Continuity Metric

  • A path is discontinuous when its maximum local length exceeds γ times the α-th-percentile local length; the practical score is the fraction of discontinuous paths.The implementation uses 1,000 paths, γ = 10, and α = 90 (90th percentile).Source: E. Continuity Metric · pp. 14–15

F. Group Action

  • The decoder's group action makes its content latent code represent Fourier coefficients of a virtual signal on the sphere.Rotations of the signal correspond to linear transformations of its Fourier modes, represented by Wigner-D matrices.Source: F. Group Action · p. 15

G. Regularizers

  • Even with an appropriate mean parameterization and group-action decoder, the latent space can remain discontinuous, motivating two additional loss regularizers.Their effectiveness is evaluated by an ablation in Table 4.Source: G. Regularizers · p. 15

G.1. Equivariance regularizer

The equivariance regularizer enforces agreement between subgroup rotations of the latent pose and planar pixel rotations, under stated geometric conditions. A separate continuity regularizer penalizes encoding differences for nearby image pairs.

G.1. Equivariance regularizer

  • The equivariance regularizer enforces agreement between latent S1 actions and planar pixel rotations, with exactness requiring a centered object, action on all pixels, and an orthographic camera.For off-center objects, rotation may use a learned center; non-orthographic projection makes the relation approximate.Source: §G.1 · pp. 15–16, Figure G.1 · p. 16
  • S1 equivariance aligns rotations within each orbit but not different orbits, reducing the remaining alignment problem to SO(3)/S1, homeomorphic to S2.Table 4 describes an ablation of regularizers in a model using SO(3) latents, the group-action decoder, and the S2S2 mean.Source: §G.1 · p. 16, Table 4 · p. 16

G.2. Continuity regularizer

  • The continuity regularizer penalizes mean-squared Frobenius-norm differences between encoded rotation matrices for image pairs that are nearby under the manifold metric.This uses nearby pairs rather than only unsupervised i.i.d. samples.Source: §G.2 · p. 16

Related papers

Loading 1807.04689v1…