Source-linked AI summary
Go to Zero: Towards Zero-shot Motion Generation with Million-scale Data
Ke Fan, Shunlin Lu, Minyue Dai, Runyi Yu, Lixing Xiao, Zhiyang Dou, Junting Dong, Lizhuang Ma, Jingbo Wang
TL;DR
Text-to-motion generation lacks robust zero-shot generalization because existing datasets and evaluation frameworks are limited. The paper introduces MotionMillion, a scalable 7B model, and MotionMillion-Eval, reporting strong out-of-domain and complex-compositional generalization while focusing on full-body motion and omitting hands and facial expressions.
Problem
Limited training-data scale and the lack of a comprehensive evaluation framework constrain zero-shot generalization in text-to-motion generation.
Method
The paper builds MotionMillion through large-scale video annotation and dataset unification, uses wavelet-preprocessed FSQ tokenization in a scalable architecture, and introduces MotionMillion-Eval.
Results
The 7B MotionMillion model achieves state-of-the-art benchmark performance and demonstrates strong zero-shot capabilities for out-of-domain and complex compositional motions.
Takeaways & Limitations
The work provides a foundation for evaluating and advancing zero-shot human motion generation through large-scale data and scalable modeling.
Takeaways & Limitations
The methodology primarily focuses on full-body motion while omitting hand and facial expressions.
Abstract
from arXiv · showhide
Generating diverse and natural human motion sequences based on textual descriptions constitutes a fundamental and challenging research area within the domains of computer vision, graphics, and robotics. Despite significant advancements in this field, current methodologies often face challenges regarding zero-shot generalization capabilities, largely attributable to the limited size of training datasets. Moreover, the lack of a comprehensive evaluation framework impedes the advancement of this task by failing to identify directions for improvement. In this work, we aim to push text-to-motion into a new era, that is, to achieve the generalization ability of zero-shot. To this end, firstly, we develop an efficient annotation pipeline and introduce MotionMillion-the largest human motion dataset to date, featuring over 2,000 hours and 2 million high-quality motion sequences. Additionally, we propose MotionMillion-Eval, the most comprehensive benchmark for evaluating zero-shot motion generation. Leveraging a scalable architecture, we scale our model to 7B parameters and validate its performance on MotionMillion-Eval. Our results demonstrate strong generalization to out-of-domain and complex compositional motions, marking a significant step toward zero-shot human motion generation. The code is available at https://github.com/VankouF/MotionMillion-Codes.
1. Introduction
The paper attributes weak zero-shot motion-generation generalization to limited data and architecture, and addresses it through scalable data, modeling, and evaluation. MotionMillion, its scalable model, and MotionMillion-Eval together support reported generalization to complex motions.
- Limited dataset scale and model architecture hinder robust zero-shot generalization in text-to-motion generation.
- MotionMillion combines web-scale motion reconstruction, semantically rich captions, multi-stage filtering, and unified annotations across existing datasets.The dataset is described as containing over 2,000 hours of high-quality motion, with the passage continuing beyond the supplied excerpt.
- The model uses Finite Scalar Quantization for motion tokenization and wavelet preprocessing to reduce reconstruction jitter at large data scale.The paper attributes increased FSQ information loss and high-frequency mis-modeling to the extremely large dataset scale.
- MotionMillion-Eval contains 126 prompts across 7 categories and evaluates text-motion alignment, smoothness, and physical feasibility.
- The contributions include a 7B scalable model and a benchmark intended to verify zero-shot capabilities, with reported generalization to out-of-domain complex compositional motions.
2. Related Work
Related work has pursued text-aligned motion generation through larger datasets and larger models, but the cited approaches retain performance or efficiency limitations.
- Text-aligned human-motion generation has benefited from advances in generative models and expanding large-scale datasets.
- Recent studies scale dataset sizes to improve motion-generation quality, while other works enlarge model capacities through pretrained large language models.
- Large-language-model-based approaches have often produced suboptimal performance, while LMM suffers from slow training speed.
3. MotionMillion Construction
MotionMillion combines web-scale video reconstruction with re-aggregated datasets through staged filtering and estimation, producing a large, diverse, smooth motion resource. Its construction targets temporal coherence, robust tracking, and reliable SMPL-based motion recovery.
- Data Sources: MotionMillion combines established datasets with web-scale monocular videos to expand motion diversity and coverage.The integrated sources include MotionX, InterHuman, Inter-X, BABEL, Fitness, PhantomDance, GDance, FineDance, HI4D, TRUMANS, and HumanSC3D.
- Reconstruction Pipeline: The reconstruction pipeline segments videos, detects and tracks people, filters boxes and transitions, estimates SMPL motion, and filters motion artifacts.The six stages enforce scene and detection quality before and after motion estimation.
- Reconstruction Pipeline: A coarse-to-fine detection strategy addresses variable human counts and severe occlusions in online videos.Grounding DINO and SAM2 support detection, tracking, and identity preservation, while confidence-based refinement filters unreliable boxes.
- Reconstruction Pipeline: Transition filtering splits clips when consecutive-frame bounding-box centers shift beyond a threshold, complementing PySceneDetect when backgrounds remain unchanged.This targets sudden subject-position changes that preliminary shot segmentation may miss.
- Reconstruction Pipeline: GVHMR estimates realistic human motion in camera and world space, predicting SMPL parameters including translation, pose, orientation, and shape.The framework uses a Gravity-View coordinate system to reduce pose-estimation ambiguity.
- Dataset Characteristics: Over 2,000 hours and more than 2 million sequences are standardized to 30fps, while quality analysis reports improved smoothness and richer semantic diversity.MotionMillion’s average jerk is significantly improved over MotionX and near HumanML3D; its semantic distribution is richer than existing benchmarks.
4. Architecture
The architecture combines a refined motion representation, FSQ-based tokenization with wavelet processing, and a scalable transformer decoder for text-conditioned motion generation.
- Architecture: The model uses a discrete autoregressive architecture with efficient motion tokenization and scalable motion generation stages.Motion is compressed and encoded with FSQ, while text is encoded separately before autoregressive generation.
- Motion Representations: Each pose represents root velocities and angular velocity together with local joint positions, velocities, and rotations relative to the root space.The representation is defined as xi = { ˙rx, ˙rz, ˙ra, pi, vi, ri}, where N denotes the number of joints.
- Motion Representations: The refined representation avoids inverse kinematics for obtaining SMPL or BVH representations and provides mutual regularization between position and rotation components.Both components derive from the same skeletal structure, and the representation can be losslessly converted to relative rotations akin to SMPL.
- Efficient Motion Tokenization: FSQ normalizes latent vectors, discretizes each dimension into L levels, and forms a structured grid of L^d codes without an explicit codebook.The model is optimized solely with reconstruction loss, removing auxiliary codebook commitment terms used in VQ-VAE.
- Efficient Motion Tokenization: Wavelet transforms are applied before encoding and after decoding to suppress jitter caused by information loss during discrete encoding.The wavelet transform targets deviations in high-frequency motion information introduced by vanilla FSQ.
- Scalable Motion Generation: The transformer uses bidirectional attention among words and causal attention across motion sequences within stacked Hybrid Attention Blocks.The blocks contain RMS-Norm layers, mixed attention, and feedforward modules; a final classification head is trained with cross-entropy loss.
5. Experiments
Experiments evaluate reconstruction quality, model scaling, and zero-shot generation on MotionMillion-Eval. Results show wavelet processing reduces FSQ-induced jitter, while the 7B model generalizes strongly across complex and out-of-domain motions.
- 5.1. FSQ Reconstruction Comparison: Wavelet transformation significantly reduces reconstructed-motion acceleration deviation from ground truth and slightly improves MPJPE over vanilla FSQ.Vanilla FSQ loses high-frequency information during discretized compression, producing severe jitter.
- 5.2. Quantitative Comparison on Different Models: Metrics often fail to fully reflect the difference between the 3B and 7B models despite improved generation ability with increasing scale.The authors identify this as a current dilemma for generative-model evaluation.
- 5.3. Zero-shot Potential Verification: Increasing model size from 1B to 7B significantly improves Text Alignment on MotionMillion-Eval.The evaluation measures Text Alignment, Physical Plausibility, and Motion Smoothness.
- 5.3. Zero-shot Potential Verification: Our 3B model significantly outperforms ScaMo-3B in Text Alignment, Physical Plausibility, and Motion Smoothness.The comparison indicates stronger generalization for the model trained on MotionMillion than for ScaMo trained on MotionUnion.
- 5.3. Zero-shot Potential Verification: Model scaling does not significantly improve Physical Plausibility or Motion Smoothness, which are reported as more dependent on dataset motion quality.Our model is comparable to MDM and T2M-GPT on these dimensions and significantly better than ScaMo-3B.
- 5.4. Qualitive Results: Visualizations show recognition of abstract zombie walking and accurate execution of long, combined motion instructions.These examples cover characteristic postures and complex compositional sequences.
6. Conclusion
The paper addresses limited zero-shot generalization by building a large annotated motion dataset, introducing a dedicated benchmark, and scaling a model to 7B parameters. It reports state-of-the-art benchmark performance and strong zero-shot capabilities.
- 6. Conclusion: The authors attribute existing methods’ limited generalization ability to constrained dataset size.They present this diagnosis as the starting point for their approach.
- 6. Conclusion: The paper introduces an efficient annotation mechanism and the largest annotated human motion dataset.This dataset is intended to support progress toward zero-shot motion generation.
- 6. Conclusion: MotionMillion-Eval is introduced as a dedicated benchmark for assessing zero-shot capabilities.The benchmark complements the dataset and model contributions.
- 6. Conclusion: A scalable architecture enables scaling the model to 7B parameters, achieving state-of-the-art benchmark performance and strong zero-shot capabilities.The conclusion presents this as a first step toward zero-shot applications in motion generation.
7. Overview
The paper provides additional implementation details and directs readers to demonstrations of out-of-domain and complex compositional long-motion generation.
- 7. Overview: Additional details cover data distribution and prompts in Section 8.These implementation details are provided for the paper’s data and prompting procedures.
- 7. Overview: Fine-grained scoring criteria for MotionMillion-Eval are described in Section 9.The scoring criteria supplement the benchmark description.
- 7. Overview: Section 10 contains the 126 prompts used by MotionMillion-Eval.The paper also presents various out-of-domain and complex compositional long motions in a demo video.
8. Data Distribution and Prompts
The paper reports distributions of motion length, velocity, and diversity, and shows prompts used for web-scale captioning and inference-time text rewriting.
- 8. Data Distribution and Prompts: Figure 7 presents the distributions of motion length, motion velocity, and motion diversity.These distributions characterize properties of the motion data.
- 8. Data Distribution and Prompts: Figure 8 shows the prompt used to caption motions in web-scale human videos.The prompt documents the captioning stage of the data pipeline.
- 8. Data Distribution and Prompts: Figure 8 also shows the prompt used for text rewriting during inference.This documents the inference-stage prompting procedure.
9. Scoring Criteria Details of MotionMillion-Eval
MotionMillion-Eval scores generated motions across text alignment, motion smoothness, and physical plausibility using four-level criteria. The benchmark also evaluates data distributions and uses captioning prompts to describe motion content.
- MotionMillion-Eval evaluates three dimensions: Text Alignment, Motion Smoothness, and Physical Plausibility.These dimensions organize the scoring criteria.
- Text Alignment: Text Alignment ranges from fully matching all prompt details to being entirely inconsistent with the prompt.Scores decrease from 4 to 1 as omissions, deviations, and inaccuracies increase.
- Motion Smoothness: Motion Smoothness ranges from fluid, natural transitions to frequent stuttering and abrupt transitions.The scale captures continuity and naturalness of movement.
- Physical Plausibility: Physical Plausibility assesses adherence to physical laws, including object interactions, lighting, shadows, and collisions.The supplied criteria define the highest score as physically realistic motion and interactions.
- Prompt-based assessment: Captioning prompts describe actions from uniformly sampled motion-video frames and require body-part involvement and action-sequence information.The prompts support detailed descriptions of the movements shown.
10. Prompts in MotionMillion-Eval
The prompts for this section are presented in the figures below. The section therefore directs readers to the figures for the complete prompt set.
- The prompts are presented in the figures below.
- The figures contain the complete set of prompts shown for this section.
- Readers should consult the figures to view all prompts.