PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling

Yiheng Li, Zhuo Li, Ruibing Hou, Yingjie Chen, Hong Chang, Hao Liu, Shiguang Shan

arXiv:2605.29488v2cs.CVcs.AI

TL;DR

Human motion generation lacks large-scale multimodal data and scalable frameworks for arbitrary conditioning combinations. AnyMo addresses this with OmniHuMo and a unified masked-modeling framework, achieving high-quality, diverse synthesis with flexible multimodal control.

  • Problem

    Human motion generation lacks large-scale semantically aligned multimodal data and scalable architectures supporting arbitrary combinations of conditioning modalities.

  • Method

    The paper introduces OmniHuMo and AnyMo, combining multimodal encoders, a residual FSQ-based motion tokenizer, and a scalable masked transformer for arbitrary conditioning.

  • Results

    AnyMo achieves competitive performance across diverse motion-generation tasks, with high-quality synthesis and flexible control across text, speech, music, and trajectories.

  • Takeaways & Limitations

    OmniHuMo and AnyMo provide a foundation for controllable multimodal human-motion generation from arbitrary combinations of text, speech, music, and trajectories.

  • Takeaways & Limitations

    OmniHuMo lacks finger-joint annotations, and audio-aligned data covers only a limited portion of the dataset.

Abstract

from arXiv · show

Conditional human motion generation remains a fundamental challenge in computer vision and robotics. Despite significant progress, current methods are often constrained by fixed modality configurations and task-specific architectures, leaving cross-modal interactions and the scaling laws of multimodal-conditioned synthesis largely underexplored. A key bottleneck is the scarcity of large-scale modality-aligned motion data, limiting generalization across diverse control signals. In this work, we introduce OmniHuMo, a large-scale, high-quality dataset comprising over 5,000 hours of motion and 3.2 million sequences with precisely aligned multimodal annotations (e.g., text, speech, music, and trajectory). Leveraging OmniHuMo, we propose AnyMo, a unified multimodal framework combining a Residual FSQ-based motion tokenizer with a scalable masked modeling transformer, enabling high-quality motion synthesis under arbitrary modality combinations. Extensive experiments show that AnyMo achieves high-fidelity synthesis while offering flexible control over both spatial and stylistic attributes.

1 Introduction

The paper identifies scarce large-scale multimodal-aligned motion data and limited scalable architectures as bottlenecks to precise controllability and robust cross-modal generalization. It introduces OmniHuMo and AnyMo to address these needs with aligned multimodal data and arbitrary-modality motion synthesis.

  • Challenges: Precise controllability and robust cross-modal generalization remain challenging because large-scale multimodal-aligned motion data are scarce.Existing optical motion-capture datasets provide high-fidelity sequences but are costly and labor-intensive, limiting scale and diversity.
  • Design Requirements: The paper argues that robust generalization and flexible controllability require both large-scale multimodal data and scalable architectures supporting arbitrary modality combinations.These requirements motivate a semantically aligned motion dataset and a framework for synthesis under arbitrary combinations of input modalities.
  • OmniHuMo: Over 5,000 hours and 3.2M+ sequences make OmniHuMo a large-scale dataset with textual annotations for all sequences and speech or music annotations for a ∼500-hour subset.The dataset is constructed through an efficient automatic-labeling pipeline applied to web-scale videos.
  • AnyMo: AnyMo is a scalable masked modeling framework that generates motion under arbitrary combinations of conditioning signals.Its architecture includes an R-FSQ-based motion tokenizer using residual quantization to address information loss in Finite Scalar Quantization.
  • Results: Extensive experiments show competitive performance across diverse motion-generation tasks, demonstrating AnyMo’s effectiveness and cross-modal versatility.The paper presents OmniHuMo as integrating text, audio, and visual modalities.

2 Related Work

Related work has progressed from single-modality motion generation toward multimodal control, while generative masked transformers provide an efficient non-autoregressive paradigm for motion synthesis. However, limited well-aligned multimodal motion data remains a key constraint on existing approaches.

  • Motion Generation with Diverse Modalities: Motion generation has expanded from text, audio-conditioned gesture, and dance synthesis to methods integrating multiple modalities for greater controllability.The cited single-modality tasks include text-driven motion generation, audio-conditioned gesture, and dance synthesis.
  • Motion Generation with Diverse Modalities: Scarce well-aligned multimodal motion data leads most existing methods to use a shared backbone for multiple single-modality tasks.
  • Generative Masked Transformers: Generative masked modeling formulates motion synthesis as a non-autoregressive “mask-and-in-between” task, balancing generation quality and efficiency.The paradigm originated with BERT’s masked modeling for language and has been extended to other domains.

3 OmniHuMo Dataset

OmniHuMo is a large-scale omni-modal human motion dataset built from diverse online videos through an automated five-stage annotation pipeline. It contains over 5,000 hours and more than 3.2 million sequences, with textual captions for every sequence and temporally aligned audio for a subset.

  • Dataset Scale: Over 5,000 hours of human motion data and more than 3.2 million motion sequences comprise OmniHuMo.The dataset scales motion data collection beyond conventional motion datasets.
  • Multimodal Annotations: Each sequence has 1–3 textual captions, while approximately 500 hours additionally contain temporally aligned audio annotations.Uneven modality coverage reflects the heterogeneous video sources.
  • Dataset Utility: OmniHuMo balances large-scale motion diversity with high-quality multimodal annotations, supporting general motion synthesis and audio-driven generation.Its aligned modalities include text, speech, and music-related signals derived from the source videos.
  • Dataset Diversity: The dataset spans indoor performances and choreographed dance as well as outdoor sports and daily events, with most sequences lasting 2–10 seconds.These durations support modeling atomic actions and rapid motion transitions.

4 Method

AnyMo is a unified framework for 3D motion generation from arbitrary combinations of text, music, speech, and trajectory inputs. It combines hierarchical Residual FSQ motion tokenization, modality-specific encoders, and scalable parallel masked Transformer reconstruction, trained through a staged multimodal curriculum.

  • Framework: AnyMo generates 3D motion from arbitrary combinations of text, music, speech, and trajectory conditions using a scalable masked modeling framework.The model is trained on OmniHuMo and represents motion sequences X ∈ R^T×D.
  • Framework: The architecture combines a Residual FSQ tokenizer with a LLaMA-based masked Transformer that reconstructs masked motion tokens.The tokenizer discretizes continuous motion into hierarchical tokens, while the Transformer performs masked-token reconstruction.
  • Motion Tokenizer: Residual FSQ uses V + 1 hierarchical quantization stages to produce coarse-to-fine token streams, with early levels encoding global patterns and higher levels refining dynamics.Each stage applies FSQ to the current residual, and the resulting ordered discrete sequences form the motion representation.
  • Condition Encoders: Modality-specific encoders process text, speech, music, and trajectories before their features condition the shared motion-generation model.T5-XL encodes text, WavTokenizer handles speech and music, and a lightweight convolutional encoder captures local trajectory dependencies.
  • Masked Modeling: The masked Transformer masks aligned temporal positions across all residual streams and predicts each stream in parallel with independent embedding and prediction heads.A bidirectional backbone supports global context reasoning beyond conventional autoregressive generation.
  • Training Curriculum: AnyMo uses staged training that progresses from text-to-motion pre-training to multimodal alignment and joint fine-tuning to address limited audio–motion synchronization.Only about one-tenth of OmniHuMo, approximately 500 hours, contains high-quality audio–motion alignment; final training uses disproportional sampling and modality augmentation.

5 Experiments

Experiments evaluate AnyMo across OmniHuMo and HumanML3D, covering reconstruction, text-, speech-, and music-conditioned generation. Results show benefits from larger data, masked and parallel token modeling, model scaling for text, and multimodal structural cues, while audio scaling is non-monotonic.

  • Experimental Setup: Experiments use OmniHuMo and HumanML3D, with modality-specific OmniHuMo splits for text, speech, and music evaluation.OmniHuMo-Text uses 50K test and 10K validation samples; speech and music use 8K test and 2K validation samples.
  • Motion Tokenizer: Tokenizer reconstruction consistently improves as training data scale increases, demonstrating the importance of large-scale data for accurate motion representations.The tokenizer is trained on OmniHuMo and evaluated on HumanML3D.
  • Motion Token Modeling: Masked modeling outperforms autoregressive modeling, while parallel modeling further improves generation quality through bidirectional context and simultaneous decoding.These strategy comparisons are conducted on HumanML3D.
  • Text-driven Motion Generation: Text-driven generation improves consistently as AnyMo scales from 111M to 3B parameters, indicating stable gains under the tested data and training regime.The evaluation uses the OmniHuMo-Text test split.
  • Multimodal and Audio-driven Generation: Multimodal conditioning generally improves generation, with trajectory cues strengthening realism, alignment, trajectory accuracy, and beat alignment, whereas audio-driven scaling is non-monotonic.AnyMo-775M achieves the best FID for audio-driven generation; additional modalities do not consistently increase motion diversity.

6 Conclusion … B Details of OmniHuMo Construction

The paper introduces OmniHuMo as a large-scale multimodal motion dataset and AnyMo as a unified framework for controllable synthesis from arbitrary modality combinations. The appendix details related work, dataset construction, experimental materials, tokenizer ablations, visualizations, and limitations.

  • 6 Conclusion: OmniHuMo is presented as the first large-scale human motion dataset with rich multimodal annotations, supporting multimodal motion modeling.The conclusion identifies the dataset as a foundation for multimodal motion modeling.
  • 6 Conclusion: AnyMo provides controllable motion generation from arbitrary combinations of text, speech, music, and trajectories.The framework is described as unified and capable of flexible multimodal control.
  • Appendix: The appendix covers related work, OmniHuMo construction, experimental setup, R-FSQ tokenizer ablations, visualizations, and limitations.These topics are organized across Appendix Sections A–F.
  • A Related Work: Motion synthesis quality depends heavily on training-data scale and diversity, while AMASS and HumanML3D face high acquisition costs and limited variety.The related-work discussion contrasts traditional optical motion-capture datasets with newer in-the-wild video extraction efforts.
  • B Details of OmniHuMo Construction: 3.2M motion sequences were distilled from over 200M raw videos using strict filtering criteria for raw videos and 3D motion annotations.The construction section describes these procedures as part of the data-processing pipeline.
  • B Details of OmniHuMo Construction: Figure S1 illustrates the filter operators used in the OmniHuMo data-processing pipeline.The figure accompanies the dataset filtering procedures.

B.1 Video Filtering. … C Experimental Setup

OmniHuMo is built through multi-stage filtering, 3D reconstruction, motion captioning, and visualization to provide high-quality, multimodally aligned motion data. The supplied passages describe these data-construction procedures but provide no details for C Experimental Setup.

  • B.1 Video Filtering.: Videos are filtered using normalized bitrate, luminance, quality, and motion criteria to remove low-quality or unsuitable content.The thresholds are bitrate ≥500, luminance [10, 210], CLIP score ≥4.0, DOVER score ≥0.25, and motion score [3.5, 350].
  • B.2 Human 3D Annotation: GVHMR reconstructs 3D human motion in gravity-view world coordinates from bounding boxes, keypoints, frames, and camera rotations.It predicts SMPL root translation, body pose, root rotation, and shape parameters.
  • B.2 Human 3D Annotation: DROID-SLAM estimates camera extrinsics while RF-DETR detection boxes mask dynamic human regions to improve camera-estimation stability and accuracy.The masking strategy replaces unstable SAM2 masks during fast motion.
  • B.3 Motion Caption Annotation: Qwen3-VL-32B generates 1–3 fine-grained captions per motion sequence, each limited to 30 words and focused on observable actions and poses.Descriptions exclude clothing, facial attributes, background, camera motion, and other irrelevant or unobservable details.
  • B.3 Motion Caption Annotation: Dance and sports categories additionally receive explicit activity labels alongside their descriptions.Examples include “lat pulldown” and “Latin dance.”
  • B.4 Visualization Examples: Visualization examples show that OmniHuMo covers diverse motion patterns with strong multimodal alignment, supporting its use for large-scale motion modeling.The examples are presented in Figures S3, S4, and S5.

C.1 Implementation Details · C.2 Evaluation Metrics.

C.1 describes the distributed data and model-training setup for AnyMo, including scalable tokenizer and transformer configurations. C.2 evaluates reconstruction, text-driven, speech-driven, and music-driven generation with geometry, fidelity, diversity, retrieval, alignment, and synchronization metrics.

  • C.1 Implementation Details: The data pipeline uses separate CPU, L20, and H20 clusters for curation, annotation, and captioning, generating approximately 100k motion sequences daily.Human 2D/3D and audio annotation use 100 L20 GPUs, while motion captioning uses 40 H20 GPUs.
  • C.1 Implementation Details: The motion tokenizer uses four-layer residual FSQ with 2048 codes per layer, fourfold temporal downsampling, AdamW, and 200-epoch training.Optimization starts at 2 × 10^-4, decays by 0.3 at epochs 60 and 140, and uses 16 NVIDIA H20 GPUs.
  • C.1 Implementation Details: AnyMo is LLaMA-based and scales from 111M to 3B parameters, training for 210 epochs on 48 H20 GPUs with per-GPU batch size 16.Training uses RMSNorm, AdamW with 500 warm-up steps, and cosine decay from 2 × 10^-4 to 1 × 10^-5.
  • C.2 Evaluation Metrics.: MPJPE measures motion reconstruction accuracy as the average L2 distance between reconstructed and ground-truth joint positions across frames.The metric is defined as Mean Per Joint Position Error.
  • C.2 Evaluation Metrics.: Text-driven motion generation is evaluated with FID, R-Precision, Div, and MMDist, covering distributional similarity, retrieval accuracy, diversity, and cross-modal alignment.These metrics follow T2M-GPT [60].
  • C.2 Evaluation Metrics.: R-Precision ranks distances to 32 candidate text descriptions—one ground truth and 31 randomly sampled negatives—and reports Top 1/2/3 retrieval accuracy.FID measures Fréchet distance between generated and real motion feature distributions; Div averages pairwise motion-feature distances; MMDist measures motion-text feature distance.
  • C.2 Evaluation Metrics.: Speech-driven gesture generation uses FID, BAS, and Div, with BAS measuring Gaussian-weighted temporal alignment between audio beats and nearest motion beats.FID and Div follow the text-generation definitions, while BAS uses squared temporal distance.
  • C.2 Evaluation Metrics.: Music-driven dance generation uses FID, BAS, and Div; traditional five-metric evaluations include FIDk, Divk, FIDg, Divg, and BAS, but EDGE reports feature-design reliability concerns.The passage states that traditional kinematic and geometric metrics may fail to capture high-level properties.

D Ablation Study

The ablations examine R-FSQ under reconstruction and motion-generation settings, showing that larger codebooks improve quality while additional residual depth benefits reconstruction but can hurt generation beyond four layers. Training-data scaling provides larger gains than codebook scaling alone.

  • Experimental setup: R-FSQ reconstruction uses a 200K OmniHuMo subset and HumanML3D evaluation, while motion generation is trained and evaluated on HumanML3D.The reconstruction and generation ablations use different data configurations.
  • Codebook size on R-FSQ: Larger R-FSQ codebooks consistently improve reconstruction and generation quality, but increasing training data yields more significant gains than scaling codebook size alone.The codebook-size ablation is summarized in Table S1.
  • Number of residual layers on R-FSQ: Increasing R-FSQ residual depth improves reconstruction quality, whereas generation performance degrades when residual layers exceed 4.The degradation is attributed to the increased difficulty of jointly modeling additional token streams.

E Qualitative Results

AnyMo generates motion sequences that closely follow diverse input modalities, with additional visualizations demonstrating this behavior. For trajectory-controlled generation, test-time optimization refines coarse predictions for more precise control.

  • Qualitative visualizations: AnyMo produces motion sequences that closely follow diverse input modalities in additional qualitative visualizations.The examples are presented in Figs. S6, S7, and S8.
  • Trajectory-controlled generation: For trajectory-controlled generation, test-time optimization refines coarse predictions to achieve more precise trajectory control.The strategy follows TLControl [51].

F Limitation and Future works

The paper identifies missing finger-joint annotations and limited audio-aligned data as key limitations of OmniHuMo. Future work should improve the dataset’s diversity and coverage while addressing these gaps.

  • Limitations: OmniHuMo lacks finger-joint annotations because hand regions are often occluded or blurred in internet videos, making reliable hand reconstruction difficult.The paper contrasts hand-region data usability with that of body movements.
  • Limitations: Audio-aligned data represents only a limited portion of OmniHuMo, restricting coverage of audio-conditioned motion.The passage identifies audio alignment as an underrepresented dataset component.
  • Future works: Future work should improve OmniHuMo’s data diversity and coverage to address these annotation and modality-alignment gaps.The passage explicitly proposes improving data diversity and coverage.
Loading 2605.29488v2…