Source-linked AI summary
Riemannian Motion Generation: A Unified Framework for Human Motion Representation and Generation via Riemannian Flow Matching
Fangran Miao, Jian Huang, Ting Li
TL;DR
Human motion generation commonly uses Euclidean representations despite the structured non-Euclidean geometry of valid motions. RMG models motion on a product manifold and learns its dynamics with Riemannian flow matching. It achieves strong, balanced results across HumanML3D and MotionMillion, while remaining limited in modalities, horizons, body configurations, and some generation settings.
Problem
Existing motion-generation systems often operate in Euclidean spaces even though valid motions follow structured non-Euclidean geometry, leaving representation geometry less systematically addressed.
Method
RMG factorizes motion into natural manifold factors and uses geodesic interpolation, tangent-space vector-field learning, and manifold-preserving integration within a product-manifold framework.
Results
RMG shows strong and balanced performance on HumanML3D and generalizes to MotionMillion, surpassing strong baselines across the reported benchmark results.
Takeaways & Limitations
The experiments support representation geometry as a core design principle, with translation and rotation factors sufficient in practice while extra factors do not necessarily improve quality.
Takeaways & Limitations
The framework has not been tested for autoregressive generation, richer conditioning modalities, longer than 10-second horizons, hands or face, motion editing, or temporal in-painting.
Abstract
from arXiv · showhide
Human motion generation is often learned in Euclidean spaces, although valid motions follow structured non-Euclidean geometry. We present Riemannian Motion Generation (RMG), a unified framework that represents motion on a product manifold and learns dynamics via Riemannian flow matching. RMG factorizes motion into several manifold factors, yielding a scale-free representation with intrinsic normalization, and uses geodesic interpolation, tangent-space supervision, and manifold-preserving ODE integration for training and sampling. On HumanML3D, RMG achieves state-of-the-art FID in the HumanML3D format (0.043) and ranks first on all reported metrics under the MotionStreamer format. On MotionMillion, it also surpasses strong baselines (FID 5.6, R@1 0.86). Ablations show that the compact $\mathscr{T}+\mathscr{R}$ (translation + rotations) representation is the most stable and effective, highlighting geometry-aware modeling as a practical and scalable route to high-fidelity motion generation.
1 Introduction
Human motion generation has advanced through architectural innovations, but motion representation geometry remains underexplored despite affecting optimization, sampling, and physical validity. RMG addresses this gap by modeling motion through natural manifold factors and Riemannian flow matching.
- Motivation: Conditional motion generation supports signals including text, action labels, audio, music, and scene context.High-fidelity synthesis is relevant to human-computer interaction, embodied AI, and augmented-reality content creation.
- Motivation: Recent systems increasingly use diffusion or autoregressive architectures, while the geometry of motion representation receives less systematic attention.Representation geometry directly affects optimization difficulty, sampling stability, and physical plausibility.
- Motivation: Common encodings use redundant Euclidean coordinates even though articulated poses have intrinsic degrees of freedom on the order of 3J.These representations occupy a higher-dimensional ambient space than the lower-dimensional manifold containing physically valid motions.
- Method: RMG decomposes motion into factors with natural geometries and models them on a product manifold using Riemannian flow matching.This integrates geometric consistency into both representation and generative dynamics.
- Results: RMG uses a compact, natural-manifold representation and matches or exceeds strong HumanML3D baselines across quality, alignment, and diversity metrics.It also surpasses all baselines on MotionMillion, supporting scalability and generalization.
2 Related Works
Human motion generation has progressed from regression and latent-variable models toward conditional architectures using joint embeddings, transformers, and diffusion. Riemannian modeling supplies the geometric foundation for flow matching, replacing Euclidean paths with manifold-consistent trajectories.
- Human Motion Generation: Human motion generation has evolved from regression-based action-to-motion mappings to generative models including VAEs, GANs, normalizing flows, and diffusion models.These models support complex, multimodal motion distributions and more realistic, diverse synthesis.
- Human Motion Generation: Conditional generative models synthesize motion from contextual signals using joint embeddings, transformers, and diffusion processes.This progression has produced higher-fidelity and more controllable motion sequences.
- Human Motion Generation: Human motion generation remains challenging because movement is highly articulated and nonlinear while outputs must align semantically with conditioning signals.Evaluation commonly combines objective metrics and user studies covering naturalness, diversity, and consistency.
- Riemannian Manifold: A Riemannian manifold is a smooth space equipped with a metric that defines tangent-space inner products and geodesic distances.The metric supports geometric measurements needed for modeling non-Euclidean data.
- Riemannian Manifold: The Riemannian metric induces a volume form used to integrate and define probability distributions on the manifold.This enables statistical and generative modeling on non-Euclidean domains.
- General Flow Matching: Flow matching learns a time-dependent velocity field that transports a source distribution p0 to a target distribution p1.Riemannian extensions replace Euclidean linear paths with geodesics and define target velocities in tangent spaces.
3 Methodology
RMG represents human motion on a product manifold and learns geometry-consistent dynamics through Riemannian flow matching. Its compact factorization, geodesic supervision, tangent-space projection, and manifold-preserving integration provide a unified representation and generation pipeline.
- Motion representation: Global translation captures trajectory in R3, while unit-quaternion rotations lie on the product manifold (S3)^J.Quaternions encode global orientation and local joint rotations relative to a canonical reference pose.
- Motion representation: The local pose factor uses Kendall pre-shape space, which removes global translation and scale to retain relative skeletal configuration.Centering and Frobenius normalization produce a translation-invariant, scale-free representation.
- Motion representation: RMG factorizes motion into global translation, global orientation, per-joint rotations, local pose, and optional temporal differences on their natural manifolds.The resulting product-manifold view covers prior representations while making each factor’s geometry explicit.
- Representation design: The compact translation-plus-rotation representation omits redundant pre-shape and temporal-difference factors while retaining the factors argued sufficient for articulated motion.The paper states that this choice is supported by empirical ablations and theoretical analysis.
- Prior distribution: RMG uses a rest pose with zero translation and identity rotations as the prior reference, so prior samples correspond to plausible static poses.The reference is specified factor-wise for translation, rotations, and optional pre-shape.
- Training and inference: Riemannian flow matching interpolates prior and data samples along geodesics, supervises tangent velocities, projects predictions into tangent spaces, and integrates the manifold ODE with exponential-map updates.Factor-wise exponential and logarithm maps handle Euclidean translation and manifold-valued rotation or pre-shape factors; the Euler update preserves manifold constraints.
4 Experiments
Experiments evaluate RMG on HumanML3D and MotionMillion using standard motion-generation metrics, then isolate representation and temporal-difference effects through controlled ablations. RMG achieves strong performance across formats and datasets, while T + R is the most stable representation under guidance.
- HumanML3D experiments use 14,616 motions and 44,970 descriptions, while MotionMillion provides 1 million clips and 4 million descriptions for pre-training and generalization evaluation.
- The evaluation measures motion quality with FID, diversity with Diversity and MultiModality Distance, and condition matching with R Precision.
- HumanML3D results: 0.043 FID is achieved on HumanML3D, slightly surpassing MoMask at 0.045, while R@1 reaches 0.525 and diversity metrics remain high.
- HumanML3D results: Under MotionStreamer, RMG ranks first on all reported metrics, including FID = 5.835 and R@1 = 0.710.
- MotionMillion results: With guidance scale 2.0 on MotionMillion, RMG attains FID 5.6 versus 10.3 for MotionMillion-7B; at scale 3.0, R@1 increases from 0.81 to 0.86.
- Ablation studies: Across controlled ablations, T + R is most stable: its FID reaches about 0.043 at ω = 6.5, whereas temporal-difference variants degrade as guidance increases.
5 Conclusion
RMG models human motion on a product manifold with geometry-consistent generation dynamics, achieving effective and scalable performance while exposing representation and modeling boundaries.
- RMG represents motion on the product manifold R3 × (S3)J and uses geodesic interpolation, tangent-space learning, and manifold-preserving integration.
- Translation and rotation factors are sufficient in practice, while adding extra factors does not necessarily improve generation quality.
- The framework uses Euclidean space R3 for translation, hypersphere S3 for quaternions, and pre-shape space for normalized relative skeleton configurations.
- The study does not explore the more involved shape space obtained by quotienting pre-shape space by the rotation group.
- Redundant coordinates introduce at least d − m flat Hessian directions when an m-dimensional quantity is represented in Rd.
- These redundant directions can produce ill-posed optimization and unstable training, whereas intrinsic compact representations remove redundant degrees of freedom.
B.2 Statistical advantage of Riemannian Flow Matching
Riemannian Flow Matching has a sharper asymptotic error rate than Euclidean Flow Matching when data lie on a low-dimensional manifold. The improvement follows from estimating the velocity field on the intrinsic state space rather than the ambient space.
- n^(-2s/(2s+d)) is the Riemannian Flow Matching error rate, compared with n^(-2s/(2s+D)) for Euclidean Flow Matching.The intrinsic dimension d is smaller than the ambient dimension D.
- The flow matching objective reduces velocity-field estimation to nonparametric regression on the manifold where samples live.An intrinsic formulation uses an s-smooth class of tangent vector fields on the d-dimensional manifold.
- The generated terminal distribution inherits the intrinsic-dimensional rate under standard flow stability assumptions.Its Wasserstein error is bounded by a quantity scaling as n^(-2s/(2s+d)).
- Because d < D, the manifold formulation yields a strictly sharper asymptotic error estimate than the ambient Euclidean formulation.The proposition frames this as a statistical advantage when the data distribution is supported on a low-dimensional manifold embedded in a higher-dimensional space.
C.1 Full Results
The paper reports full results for HumanML3D in two formats and for MotionMillion, with repeated confidence-interval evaluation for HumanML3D and single-run evaluation for MotionMillion.
- Full HumanML3D results are reported in both the original H3D format and the MotionStreamer format.These results correspond to Tables 4 and 5.
- Full MotionMillion results are reported in Table 6.
- HumanML3D experiments use 20 independent runs with means and 95% confidence intervals computed using the t-distribution.
- MotionMillion evaluation is limited to a single run because of dataset scale and compute-budget constraints.
C.2 Training Stability
MotionMillion training shows a smooth loss decrease and generally stable gradient norms, with one early spike motivating warm-up learning rates and gradient clipping.
- The MotionMillion training loss decreases smoothly and steadily throughout training.The corresponding training-loss curves are shown in Figure 4.
- The gradient norm remains stable apart from one spike at the beginning of training.
- The early gradient spike motivates a warm-up learning-rate strategy and one gradient-clipping threshold for the full training process.
- Both the loss and gradient norm curves remain stable at the end of training.
D.1 Model Architecture
RMG uses Diffusion Transformer backbones with dataset-specific text conditioning, while MotionMillion processes text and motion as a unified token stream. Gradient norm curves are reported for MotionMillion.
- D.1 Model Architecture: RMG uses a Diffusion Transformer backbone, with the text encoder varying across datasets.For HumanML3D, text features are fused with time embeddings through an MLP before conditioning the transformer.
- D.1 Model Architecture: HumanML3D conditioning uses Qwen3-Embedding-0.6B to produce a 1024-dimensional text feature vector.The fused text and time representation serves as the conditioning input to the Diffusion Transformer.
- D.1 Model Architecture: MotionMillion uses Qwen3-1.7B features with a single-stream multi-modal Diffusion Transformer that concatenates text and motion tokens.The concatenated sequence is processed as a unified input stream.
D.2 Training Details
The appendix reports training hyperparameters by model scale and describes conversion functions that map RMG representations into the target dataset formats.
- D.2 Training Details: Training hyperparameters are listed for different model scales in Table 7.Effective batch size equals per-device batch size multiplied by device count and gradient accumulation steps.
- D.2 Training Details: RMG implements separate conversion functions for HumanML3D and MotionMillion formats.These functions first map the representation to rotation-based or joint-based motion representations.
- D.2 Training Details: The converted rotation-based or joint-based representation is processed using the target format’s existing preprocessing pipeline.Figure 6 summarizes the conversion functions between motion representations.
E Limitations
The authors identify scope limits involving conditioning, sequence length, body configuration, and downstream motion-editing tasks. They also note risks from misuse, inherited bias, and deployment without safeguards.
- E Limitations: The representation has not been tested in autoregressive generation, and music, video, and interactive human-object interaction remain unexplored.These limitations constrain the evaluated conditioning modalities and generation settings.
- E Limitations: Generation is limited to 10 seconds or 300 frames, while longer horizons may require more data and computational resources.The framework also excludes richer body configurations such as hands and face.
- E Limitations: Motion editing and temporal in-painting are not studied, and generated motion systems may pose risks through deceptive behavior, bias, or unsafe deployment.The authors recommend dataset curation, synthetic-content disclosure, and application-level safety mechanisms.