Neural Manifold Ordinary Differential Equations

• Continuous Normalizing Flows use Neural ODE dynamics and require only the trace of the dynamics’ Jacobian for continuous change of variables.Source: §1 · p. 2
• Earlier manifold flows do not generalize to arbitrary manifolds, while diffeomorphic flows require the data and latent distributions to have equivalent topology.Source: §1 · p. 2
• Neural Manifold ODEs generalize Neural ODEs; dynamic charts enable local Euclidean integration, smooth transitions, forward integration, and adjoint-based gradient computation.Source: §1 · p. 2
• The method outperforms existing manifold normalizing flows on their specific domains.Source: §1 · p. 2

• Projection-based flows [16] require a manifold-to-Euclidean diffeomorphism, whereas integrating local dynamics constructs a global diffeomorphism without mapping the entire manifold to Euclidean space.Source: §2 · p. 2
• On hyperbolic space, the method derived from vector fields respects manifold geometry and substantially improves on the flows in [2].Source: §2 · p. 3
• On spheres, the method is general, yields globally defined densities, and remains tractable for higher dimensions; it markedly outperforms the constructions in [39].Source: §2 · p. 3

3.1 Differential Geometry

• Tangent spaces are vector spaces at manifold points; pushforwards generalize Euclidean Jacobians and relate tangent spaces at different points.Source: §3.1 · p. 3
• Tangent spaces provide the derivation used to define manifold ODEs and can serve as chart domains for the dynamic chart method.Source: §3.1 · p. 3

3.2 Riemannian Geometry

• A Riemannian metric on tangent spaces induces manifold distances; the exponential map acts as a chart when restricted to a local neighborhood of zero.Source: §3.2 · p. 4

3.3 Manifold Ordinary Differential Equations

• A manifold ODE relates a curve on M to a vector field, and local solutions are guaranteed under sufficient conditions on that field.Source: §3.3 · p. 4

4 Neural Manifold Ordinary Differential Equations

• Neural Manifold ODE dynamics are parameterized by a neural network, while forward and backward computations use different strategies.Forward computation requires explicit manifold methods; backward computation can be defined through a Euclidean-space ODE.Source: §4 · p. 4

4.1 Forward Mode Integration

• Projection solvers embed the manifold in Euclidean space and project after each step, requiring a level-set representation; implicit solvers use local charts but can be difficult to define.Projection can be ill-defined for some points or manifolds, while implicit methods can apply to any manifold without a level-set representation.Source: §4.1 · pp. 4–5

4.2 Backward Mode Adjoint Gradient Computation

• The manifold adjoint method analytically computes derivatives instead of differentiating through the solver, assuming the manifold is embedded in Euclidean space.The adjoint state is a(t) = Dz(t)L, and its gradients can cover the time bounds, initial condition, and network weights.Source: §4.2 · p. 5

5 Dynamic Chart Method

• Local chart solves connected by transitions form the dynamic chart forward pass, mapping to Euclidean space, solving an equivalent ODE, and returning to the manifold.The propositions establish that the chart-mapped local solution satisfies the manifold ODE and that finitely many charts suffice to integrate to the end time.Source: §5 · p. 6
• The method avoids repeated Lie and Riemannian exponential-map evaluations by defining dynamics directly on the tangent space.This shifts computational cost to the chart derivative and supports the paper's hyperbolic-space construction.Source: §5 · p. 7
• Chart-based dynamics avoid catastrophic gradient failure when solver errors could push gradients outside measure-zero tangent spaces in a higher-dimensional ambient space.The authors attribute this to Neural ODE block errors not causing gradients to leave their domain and chart transitions introducing only precision error.Source: §5 · p. 7
• The method can use advanced Euclidean ODE solvers and directly incorporate Neural ODE improvements without additional manifold constructions.The paper contrasts this with cutting-edge ODE solvers that remain restricted to Euclidean space.Source: §5 · p. 7

6 Manifold Continuous Normalizing Flows

• MCNFs compute probability change by applying continuous change of variables on Neural ODE blocks and using smooth chart transitions.The value is integrated through the ODE, with the change in log probability supplied.Source: §6 · p. 7
• The required determinant of the exponential-map derivative can be evaluated analytically for the hyperboloid and sphere.Source: §6 · p. 7
• Because MCNFs require only local dynamics, their construction generalizes to arbitrary manifolds.Chart domains can exclude conjugate points, avoiding diffeomorphism issues such as antipodal points on the sphere.Source: §6 · p. 7

7.1 Density Estimation

• Density estimation trains flows on H2 and S2, selected for efficient computation and ease of visualization.The broader experiment evaluates density estimation and variational inference on these two spaces.Source: §7.1 · p. 7, §7 · p. 7
• MCNFs fit complex, discontinuous, and multimodal target densities on H2 and S2, significantly improving over manifold-flow baselines.Baseline methods struggle to reduce probability mass in low-density areas; Figure 3 visualizes H2 density estimation.Source: §7.1 · p. 8, Figure 3 · p. 8

7.2 Variational Inference

• In variational inference, continuous flows outperform the hyperbolic baselines; hyperbolic models tend to outperform Euclidean models at low dimensions but not consistently at high dimensions.Experiments use Binarized Omniglot and MNIST; Table 1 reports average negative test log likelihood and standard deviation over five trials.Source: §7.2 · p. 8, Table 1 · p. 8

8 Conclusion

• Figure 4 visualizes density estimation on S2 using a two-dimensional Mollweide projection.Source: Figure 4 · p. 8
• Neural Manifold ODEs provide a framework for continuous-time neural networks on manifolds, augmented with a dynamic chart method.The method addresses numerical and computational cost issues and integrates with modern Neural ODE theory.Source: §8 · pp. 8–9
• MCNFs outperform existing manifold normalizing flow baselines on density estimation and variational inference, and require only local manifold structure.The paper anticipates applications to topologically nontrivial data in lattice quantum field theory, motion estimation, and protein structure prediction.Source: §8 · p. 9

9 Broader Impact

• The authors cannot determine the eventual impact of applications in physics, robotics, and biology, and do not expect deepfake-related harms because the work targets data topology rather than hyper-realistic vision or audio.Source: §9 · p. 9

A.1 Dynamic Chart Method

• The dynamic-chart appendix states that a locally integrated Euclidean solution maps to a valid manifold ODE solution, and that finitely many charts cover the curve.Source: §A.1 · p. 12

A.2 Derivation of Gradient for Adjoint State

• For an embedded manifold, the adjoint state propagates gradients backward through the derivative of the flow map, following the chain rule.The appendix presents this as the manifold analogue of the adjoint derivation.Source: §A.2 · p. 12

B.1 Background

• A continuous normalizing flow maps a base sample to a target sample by solving an ODE, with log density evolving by the instantaneous change-of-variables formula.Source: §B.1 · p. 13

B.2 Manifold Continuous Normalizing Flows

• MCNFs integrate dynamics in local charts and compose the resulting updates, including the Riemannian probability update induced by each chart.The chart determinant shorthand is not generally an ordinary determinant, such as when mapping from R^n to R^d for d > n.Source: §B.2 · pp. 13–14

B.3 Base Distributions

• The hyperbolic base is a wrapped normal sampled in a tangent space, transported to its mean, and mapped onto the manifold; the spherical base uses a von Mises-Fisher distribution because transport between conjugate points is undefined.Source: §B.3 · p. 14

B.4.1 Analytic Derivations

• Hyperbolic operations use Lorentz geometry; stereographic projection visualizes H2 on the Poincaré ball, while the log-map log determinant negates the exponential-map value and parallel transport has log determinant zero.Source: §B.4.1 · p. 15

B.4.2 Numerical Stability

• Numerical stability measures clamp arccosh inputs above 1 and use the limiting value for sinh-division terms near zero.Source: §B.4.2 · p. 15

B.5.1 Analytic Derivations

• Table 3 collects sphere-operation formulas; the section also specifies Mollweide coordinates and exponential, logarithmic, and parallel-transport log determinants.Source: Table 3 · p. 16, §B.5.1 · p. 16

B.5.2 Numerical Stability

• Near zero, sin(x)/x is specially handled; the sphere log-map derivative is derived explicitly because higher-order derivatives can be numerically unstable.As r = x^T y approaches 1, the derivative limit is I − xx^T.Source: §B.5.2 · p. 16

B.6 Backpropagation

• MCNF gradients use the Euclidean adjoint equation for the negative log likelihood, while hyperbolic and spherical log-determinant terms are directly differentiable.Hyperbolic VAE training differentiates the ELBO using reparameterization.Source: §B.6 · p. 17

B.7.1 Construction

• General dynamics are neural-network outputs projected onto the manifold tangent space; hyperbolic dynamics use a single exp₀ ◦ ODE ◦ log₀ chart.Source: §B.7.1 · p. 17

Spherical Space.

• Spherical dynamics use projection rather than a global diffeomorphism and require the log-map derivative and injectivity radius π.Source: Spherical Space · p. 17

B.7.2 Existence of a Solution

• Picard–Lindelöf conditions are supported by Lipschitz neural-network dynamics and a compactly bounded chart domain.Source: §B.7.2 · p. 18

C.1 Data

• Density targets include wrapped normals, Gaussian mixtures, and checkerboards on hyperbolic and spherical spaces; variational inference uses dynamically binarized MNIST and resized Omniglot.Source: §C.1 · p. 18

C.2 Density Estimation

• MCNF achieved better density-estimation results than spherical baseline [39], frequently with over an order of magnitude fewer samples.Source: §C.2 · pp. 18–19

C.3 Variational Inference

• For variational inference, latent dimensions range from 2 to 6, and continuous flows replace discrete flows with neural ODE or MCNF blocks.Source: §C.3 · p. 19

D Dynamic Chart Method

• The dynamic chart method supports learning arbitrary manifold densities with conjugate points, including on S².Source: D Dynamic Chart Method · p. 19

D.1 Choosing Charts

• Uniform time segments move exp-map injectivity balls along anchor points, enabling antipodal mass transport, though local dynamics can still reach chart edges and destabilize solving.A local Lipschitz bound can keep solutions inside the chart domain; most experiments remained stable without it.Source: §D.1 · pp. 19–20

D.2 Expressivity and Generality of MCNF

• With 16 charts, MCNF learns high probability at the sphere’s antipode, unlike the one-chart model; Figure 6 shows the chart-count comparison.Source: Figure 6 · p. 20, §D.2 · p. 20

E Generated MNIST Samples

• A two-dimensional hyperbolic MCNF generates MNIST-like digits, and latent interpolation yields semantic transitions such as 4-to-7 producing 9-like intermediates.Figure 7 shows real, random-generated, and grid-positioned samples.Source: E Generated MNIST Samples · p. 21, Figure 7 · p. 21