PaperlayerView exact revision on arXiv

Source-linked AI summary

GENMO: A GENeralist Model for Human MOtion

Jiefeng Li, Jinkun Cao, Haotian Zhang, Davis Rempe, Jan Kautz, Umar Iqbal, Ye Yuan

arXiv:2505.01425v1cs.GRcs.AIcs.CVcs.LGcs.RO

TL;DR

Motion modeling traditionally separates estimation from generation, limiting shared learning between accurate reconstruction and diverse synthesis. GENMO unifies them by treating estimation as constrained generation within a dual-mode diffusion framework, and experiments show state-of-the-art performance across both task families. The framework also supports flexible multimodal, variable-length motion generation while reducing reliance on 3D motion-capture data.

  • Problem

    Existing motion estimation and generation systems use separate models despite sharing temporal and kinematic representations, limiting knowledge transfer between accurate reconstruction and diverse synthesis.

  • Method

    GENMO combines estimation and generation in one diffusion framework using constrained motion generation, dual-mode training, estimation-guided learning, and flexible multimodal conditioning.

  • Results

    GENMO achieves state-of-the-art performance across motion estimation and generation benchmarks while handling multiple human-motion tasks within one model.

  • Takeaways & Limitations

    GENMO supports variable-length motion generation with text, videos, music, 2D keypoints, and 3D keyframes while reducing reliance on 3D motion-capture data.

  • Takeaways & Limitations

    The current model supports only full-body motion and does not integrate camera-parameter estimation, which the authors identify as future work.

Abstract

from arXiv · show

Human motion modeling traditionally separates motion generation and estimation into distinct tasks with specialized models. Motion generation models focus on creating diverse, realistic motions from inputs like text, audio, or keyframes, while motion estimation models aim to reconstruct accurate motion trajectories from observations like videos. Despite sharing underlying representations of temporal dynamics and kinematics, this separation limits knowledge transfer between tasks and requires maintaining separate models. We present GENMO, a unified Generalist Model for Human Motion that bridges motion estimation and generation in a single framework. Our key insight is to reformulate motion estimation as constrained motion generation, where the output motion must precisely satisfy observed conditioning signals. Leveraging the synergy between regression and diffusion, GENMO achieves accurate global motion estimation while enabling diverse motion generation. We also introduce an estimation-guided training objective that exploits in-the-wild videos with 2D annotations and text descriptions to enhance generative diversity. Furthermore, our novel architecture handles variable-length motions and mixed multimodal conditions (text, audio, video) at different time intervals, offering flexible control. This unified approach creates synergistic benefits: generative priors improve estimated motions under challenging conditions like occlusions, while diverse video data enhances generation capabilities. Extensive experiments demonstrate GENMO's effectiveness as a generalist framework that successfully handles multiple human motion tasks within a single model.

1. Introduction

GENMO unifies motion estimation and generation by treating estimation as constrained generation, combining precise conditioning with diverse multimodal synthesis. Its dual-mode training and flexible architecture support variable-length motions across multiple inputs, with state-of-the-art results across estimation and generation tasks.

  • Motivation: GENMO unifies motion estimation and generation in one framework by reformulating estimation as constrained motion generation.The generated motion must adhere to observed signals while retaining generative flexibility.
  • Motivation: Generative priors improve estimation under challenging conditions, while diverse video data enriches generative motion distributions.The paper identifies bidirectional benefits between the two tasks, including improved plausibility under occlusions and greater generative diversity.
  • Method: GENMO uses dual-mode training that combines estimation-focused prediction with diffusion training for rich conditional motion distributions.Estimation mode targets maximum-likelihood motion from conditioning signals, while generation mode learns diverse outputs from noisy motions and sampled timesteps.
  • Method: The architecture supports variable-length motions and arbitrary combinations of multimodal conditions without complex post-processing.Supported inputs include videos, music, text, 2D keypoints, and 3D keyframes.
  • Results: GENMO achieves state-of-the-art performance across global and local motion estimation, music-to-dance generation, text-to-motion generation, and motion-inbetweening.The evaluation covers both estimation and generation tasks within one generalist framework.

2. Related Work

Prior work largely treats motion generation and estimation as separate problems, often relying on limited data, fixed-length generation, or expensive optimization. GENMO addresses these limitations with unified multimodal modeling, arbitrary-length generation, and direct use of in-the-wild video annotations.

  • Motion Generation: Most multimodal motion-generation methods support diverse inputs but do not jointly perform precise motion estimation.Video-conditioned generative methods may loosely imitate video content rather than match it accurately.
  • Long-Sequence Generation: Existing long-sequence methods commonly stitch separately generated fixed-length motions using ad-hoc post-processing.GENMO instead uses a diffusion architecture for seamless arbitrary-length generation with multiple modalities.
  • Data: Limited size and diversity in motion datasets motivate using in-the-wild videos with 2D annotations without explicit 3D reconstruction.The paper presents this as reducing reliance on noisy data while enhancing robustness and diversity.
  • Motion Estimation: Global motion estimation is inherently ill-posed, so prior methods use generative priors and SLAM, often with expensive optimization or separate post-processing.Feed-forward approaches improve speed, while GENMO jointly models estimation and generation in one diffusion framework.
  • Motion Estimation: GENMO extends feed-forward estimation by using shared representations and generative priors to produce more plausible estimates.Its unified framework models generation and estimation together rather than maintaining separate task-specific systems.

3. Generalist Model for Human Motion

GENMO unifies motion estimation and generation by conditioning variable-length motion synthesis on mixed multimodal inputs, including video, music, text, and 2D signals. Its architecture and training combine flexible conditioning, diffusion generation, regression-style estimation, and geometric constraints.

  • Unified formulation: GENMO represents both estimation and generation as conditional synthesis of an arbitrarily long motion sequence from condition signals and masks.Conditions can include video, camera motion, 2D skeletons, music, 2D bounding boxes, and text.
  • Motion representation: The motion representation jointly encodes egocentric and camera-space motion, including local pose parameters, root motion, and camera pose.Each pose combines global motion, local motion, and camera pose information.
  • Architecture: GENMO processes noisy motion with multimodal conditions through additive fusion, RoPE-based Transformer blocks, and specialized injection mechanisms.Frame-aligned modalities use temporal masking, while text uses multi-text attention for inputs assigned to user-specified time windows.
  • Architecture: Binary masks restrict each text prompt to its designated time window, while subsequent RoPE-based processing produces smooth motion across window boundaries.This design addresses the lack of frame alignment in text conditioning without requiring text insertion at arbitrary motion positions.
  • Variable-length inference: Relative positional embeddings and sliding-window attention allow arbitrary-length motion generation in a single diffusion forward pass.The model can incorporate multiple text inputs across different temporal spans while limiting each token’s attention to a local neighborhood.
  • Dual-mode training: GENMO combines standard diffusion training with estimation-mode regression using pure Gaussian noise and geometric regularization, including 2D reprojection when only 2D annotations are available.The paper reports that video-conditioned predictions are more deterministic than text-conditioned predictions, motivating the distinction between estimation and generation.

4. Experiments

GENMO is evaluated with one jointly trained checkpoint across motion estimation, motion generation, and motion in-betweening tasks. Results show strong estimation performance, diverse music-to-dance generation, improved text-to-motion generation from 2D training, and benefits from both estimation and generation objectives.

  • Global Motion Estimation: GENMO consistently outperforms specialized methods for global human motion estimation, including 202.1 mm W-MPJPE on EMDB versus 222.4 mm for TRAM.The comparison uses identical SLAM systems and backbone features for both methods.
  • Local Motion Estimation: GENMO surpasses existing approaches across most local motion-estimation metrics and remains robust under severe occlusions and truncations.Removing 2D-only training causes performance degradation, supporting the estimation-guided 2D training objective.
  • Music-to-Dance Generation: GENMO improves music-to-dance diversity, physical plausibility, and motion-music correlation, outperforming the specialized variant on Divk, Divm, PFC, and BAS.Its higher FID than the music-only variant is attributed to training on more heterogeneous motion data across tasks and domains.
  • Text-to-Motion Generation: GENMO improves motion fidelity and text-prompt correspondence over MDM across HumanML3D and Motion-X, while 2D training substantially improves both benchmarks.These results support using in-the-wild 2D data for text-conditioned motion generation.
  • Evaluation Limitation: GENMO’s HumanML3D comparison is limited by conversion between HumanML3D and SMPL representations, which introduces distribution shifts affecting evaluation metrics.The reported mismatch alters bone lengths, joint angles, and joint velocities.
  • Motion In-betweening: GENMO achieves superior motion in-betweening performance when evaluated with either 2 or 5 sampled keyframes.The experiments use HumanML3D and Motion-X test sets and overwrite noisy motion with keyframe poses before each denoising step.
  • Ablation Study: Ablations show that removing estimation training degrades global motion estimation, while removing generation training reduces performance relative to the unified model.The results indicate complementary benefits from estimation objectives and generative priors.
  • Inference Analysis: Motion-estimation performance remains relatively stable across denoising-step counts, with optimal estimation at 5 inference steps and single-step denoising producing video-consistent motion.Text-to-motion generation is more sensitive to the number of inference steps.

5. Conclusion

GENMO unifies human motion estimation and generation in one framework, supporting flexible multimodal control and state-of-the-art performance. Its current scope is limited by reliance on off-the-shelf camera estimation and full-body-only motion support.

  • GENMO bridges motion estimation and generation while supporting variable-length motion and control through text, videos, music, 2D keypoints, and 3D keyframes.
  • GENMO achieves state-of-the-art performance on motion estimation and generation benchmarks while reducing reliance on 3D motion capture data.
  • GENMO handles multiple human motion tasks within one framework and achieves superior results compared with task-specific models.
  • The current system relies on off-the-shelf SLAM methods to obtain camera parameters for videos.
  • The model currently supports only full-body motion, with facial expressions and hand articulation left for future work.

A. Implementation Details

GENMO combines a Transformer-based diffusion architecture with mixed motion datasets and dual-mode training. Its training and inference pipeline uses multimodal conditioning, robustness augmentation, and variable-length inference support.

  • Model Architecture: GENMO uses 16 layers, each containing a ROPE-based Transformer block followed by a multi-text injection block.Each attention unit has 8 heads, and the MLP width is dmlp = 1024.
  • Training Datasets: Training combines motion estimation datasets, music-to-dance data, and text-to-motion datasets, including AMASS, BEDLAM, Human3.6M, 3DPW, AIST++, HumanML3D, and Motion-X.HumanML3D motions are converted to SMPL parameters using inverse kinematics.
  • Training Datasets: BEDLAM and Human3.6M use video frames and 2D keypoints as conditions with global 3D motions as targets, while 3DPW uses pseudo-clean global trajectories and local-pose losses.Motion-X training uses 2D keypoints and text because its 3D ground truth is noisy.
  • Condition Processing: Video conditions use a frozen TRAM encoder, music uses the EDGE encoder, and 2D keypoints are normalized using bounding boxes and focal length.
  • Training Details: Training augments 2D keypoints with random masking and Gaussian noise, masks input conditions, uses N = 120 sequences, and supports variable sequence lengths during inference.The model is trained from scratch for 500 epochs with AdamW.

B. Evaluation Settings for Music-to-Dance Generation

GENMO’s music-to-dance evaluation uses one shared checkpoint on AIST++ and measures motion quality, diversity, physical plausibility, and synchronization with music.

  • Evaluation on AIST++ uses the same one-in-all GENMO checkpoint employed across the other tasks.
  • The evaluation covers motion quality, generation diversity, physical plausibility, and motion-music correlation.
  • Motion quality and generation diversity are assessed with FID and average feature distance using kinetic and geometric features.
  • Physical plausibility is measured with MPJPE, PA-MPJPE, and the Physical Foot Contact score.
  • Motion-music correlation is measured with Beat Alignment Score, based on temporal distances between kinematic and musical beats.

C. Evaluation Settings for Text-to-Motion Generation

Text-to-motion evaluation uses established HumanML3D protocols alongside additional keypoint-based metrics, while Motion-X evaluation uses independently implemented encoders. The SMPL-to-HumanML3D conversion introduces motion-quality degradation.

  • HumanML3D evaluation converts GENMO’s raw motion representation into HumanML3D format using recovered SMPL parameters and the neutral-gender SMPL model.
  • The SMPL-to-HumanML3D conversion can degrade motion quality because mismatched bone lengths introduce artifacts such as foot skating.
  • Additional HumanML3D evaluation reports FID and Diversity using kinetic and geometric features computed from 24 keypoints.
  • Motion-X evaluation uses independently implemented text and motion encoders because the original encoder implementation details were undisclosed.
  • Motion-X text prompts are encoded with a pretrained CLIP language model and tokenizer into 512-dimensional embeddings, with encoder weights frozen during evaluation.
  • Table 9 benchmarks human motion generation and evaluates motion quality on the 3DPW-XOCC dataset.

D. Evaluation Settings for Motion In-betweening

GENMO evaluates motion in-betweening by enforcing desired poses at specified keyframes during diffusion denoising. Because its representation lacks global root information, evaluation overwrites only local body poses and global root orientation.

  • D. Evaluation Settings for Motion In-betweening: Desired keyframe poses overwrite the noisy motion before each denoising step during motion in-betweening evaluation.This follows the methodology used by prior diffusion-based approaches.
  • D. Evaluation Settings for Motion In-betweening: The same one-in-all GENMO checkpoint is used for motion in-betweening and all other evaluated tasks.
  • D. Evaluation Settings for Motion In-betweening: Because the feature representation lacks global root information, only local body poses and global root orientation are overwritten at keyframes.

E. Evaluation on Occlusion-Specific Benchmark

GENMO is evaluated on 3DPW-XOCC, which tests motion estimation under extreme occlusion and truncation. It outperforms prior methods, and ablations show that generative tasks improve robustness in these conditions.

  • E. Evaluation on Occlusion-Specific Benchmark: 3DPW-XOCC evaluates 3D human pose estimation under extreme occlusion and truncation using random occlusion patches and frame truncations.
  • E. Evaluation on Occlusion-Specific Benchmark: GENMO outperforms state-of-the-art human motion estimation methods, including methods explicitly designed for occlusions.
  • E. Evaluation on Occlusion-Specific Benchmark: Removing generative tasks from training worsens estimation performance compared with the complete GENMO model.The ablation supports the contribution of generative priors to plausibility and accuracy under visually challenging conditions.

F.1. Generative Priors for Estimation

GENMO differs from prior generative-prior approaches by unifying motion estimation and generation in one model. It aims to retain both diverse generation and precise estimation rather than sacrificing one capability during fine-tuning.

  • F.1. Generative Priors for Estimation: Prior approaches repurpose generative models for estimation tasks such as depth, surface normals, and semantic segmentation.
  • F.1. Generative Priors for Estimation: GENMO integrates motion generation and estimation within a single coherent framework.
  • F.1. Generative Priors for Estimation: Unlike prior fine-tuning approaches, GENMO maintains stochastic diversity for generation and deterministic precision for estimation.
Loading 2505.01425v1…