Source-linked AI summary
MMM: Generative Masked Motion Model
Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, Chen Chen
TL;DR
Text-to-motion systems face a trade-off among fidelity, speed, and editability. MMM combines discrete motion tokenization with conditional masked-token prediction and parallel iterative decoding. It reports strong benchmark quality, editing performance, and substantial speed advantages, while remaining limited by dataset sequence length and single-person motion support.
Problem
Existing text-to-motion methods face a trade-off between real-time performance, high fidelity, and motion editability.
Method
MMM combines a motion tokenizer that produces discrete latent tokens with a conditional masked motion transformer that predicts masked tokens from text and motion context.
Results
MMM outperforms state-of-the-art methods across reported quality metrics, supports motion editing, and is two orders of magnitude faster than raw-motion diffusion models.
Takeaways & Limitations
Parallel iterative decoding provides a single framework for high-fidelity generation, fast inference, and editable motion synthesis.
Takeaways & Limitations
Training datasets contain motions up to 196 frames, limiting fine-grained detail in exceptionally long single descriptions; interactive multi-person motion is unsupported.
Abstract
from arXiv · showhide
Recent advances in text-to-motion generation using diffusion and autoregressive models have shown promising results. However, these models often suffer from a trade-off between real-time performance, high fidelity, and motion editability. To address this gap, we introduce MMM, a novel yet simple motion generation paradigm based on Masked Motion Model. MMM consists of two key components: (1) a motion tokenizer that transforms 3D human motion into a sequence of discrete tokens in latent space, and (2) a conditional masked motion transformer that learns to predict randomly masked motion tokens, conditioned on the pre-computed text tokens. By attending to motion and text tokens in all directions, MMM explicitly captures inherent dependency among motion tokens and semantic mapping between motion and text tokens. During inference, this allows parallel and iterative decoding of multiple motion tokens that are highly consistent with fine-grained text descriptions, therefore simultaneously achieving high-fidelity and high-speed motion generation. In addition, MMM has innate motion editability. By simply placing mask tokens in the place that needs editing, MMM automatically fills the gaps while guaranteeing smooth transitions between editing and non-editing parts. Extensive experiments on the HumanML3D and KIT-ML datasets demonstrate that MMM surpasses current leading methods in generating high-quality motion (evidenced by superior FID scores of 0.08 and 0.429), while offering advanced editing features such as body-part modification, motion in-betweening, and the synthesis of long motion sequences. In addition, MMM is two orders of magnitude faster on a single mid-range GPU than editable motion diffusion models. Our project page is available at \url{https://exitudio.github.io/MMM-page}.
1. Introduction
Text-driven motion generation must balance fidelity, speed, and editability. MMM addresses this trade-off with bidirectional masked-token modeling and parallel decoding.
- Text-to-motion generation is challenging because language and motion data have inherently different distributions.
- Diffusion and autoregressive models improve motion quality but trade off generation speed or editability.
- MMM uses a VQ-VAE-based motion tokenizer and a conditional masked motion model to represent and generate discrete motion tokens.
- Bidirectional attention enables parallel, iterative prediction of multiple motion tokens that remain consistent with text and motion dynamics.
- MMM supports motion editing by placing mask tokens in regions requiring changes while preserving smooth transitions with unchanged regions.
2. Related Work
Prior text-to-motion methods rely on latent alignment, diffusion, or autoregressive decoding, each leaving quality, efficiency, or editability limitations that motivate MMM.
- Latent-space alignment methods use distance losses between text and motion representations but generally produce unsatisfactory motion quality.
- MMM instead uses masked motion modeling to support concurrent prediction of multiple motion tokens.
- Raw-motion diffusion models support editing through inpainting but incur high inference costs because they process redundant motion sequences.
- Autoregressive models improve fidelity by modeling temporal correlations, yet causal sequential decoding limits bidirectional dependency modeling and motion editability.
3. Method
MMM tokenizes motion into a discrete codebook representation and trains a text-conditioned masked transformer to reconstruct missing tokens. Inference progressively refines multiple tokens in parallel.
- Motion Tokenizer: The motion tokenizer quantizes encoder outputs into entries of a learned codebook, producing a discrete latent motion representation.
- Motion Tokenizer: A large codebook preserves fine-grained motion representations, while factorized codes stabilize codebook usage and reduce collapse risk.
- Conditional Masked Motion Model: The transformer receives motion tokens, text embeddings, and special tokens, using self-attention to model relationships among motion tokens.
- Conditional Masked Motion Model: Training randomly replaces r × L motion tokens with [MASK] tokens and reconstructs them from the corrupted sequence and text embedding.
- Inference via Parallel Decoding: Inference begins with an empty masked canvas, then repeatedly remasks low-confidence tokens and predicts the masked subset in parallel.
- Inference via Parallel Decoding: The pretrained length predictor estimates the generated motion length from the input text for iterative decoding.
4. Motion Editing
MMM uses bidirectional masking to edit motion by filling selected gaps, enabling in-betweening, long-sequence synthesis, and upper-body modification.
- Motion In-betweening: Mask tokens can be placed wherever editing is required because bidirectional decoding uses both past and future context.
- Motion In-betweening: Motion in-betweening fills gaps between keyframes without additional training because training already covers random temporal masking patterns.
- Long Sequence Generation: Long sequences are synthesized by generating motion for multiple text prompts and using the trained masked model as a prior.
- Upper Body Editing: Upper-body editing uses separate upper- and lower-body tokenizers, then concatenates their tokens into full-body embeddings.
- Upper Body Editing: Randomly masking lower-body tokens helps the transformer learn whole-body spatial and temporal dependencies for upper-body editing.
5. Experiments
MMM is evaluated on HumanML3D and KIT-ML for text-to-motion quality, inference speed, and editability, outperforming or improving on leading methods across these dimensions.
- Experimental Setup: Experiments use HumanML3D and KIT-ML with standard text-to-motion evaluation protocols and comparisons against state-of-the-art approaches.The evaluation includes alignment, distribution, diversity, speed, and editing measures.
- Quantitative Comparison: MMM consistently achieves the best FID and Multimodal Distance, while remaining competitive on R-Precision and Diversity.On HumanML3D, it excels in Top-1, Top-2, and Top-3 R-Precision, FID, Multimodal Distance, and Diversity.
- Qualitative Comparison: MMM generates more realistic and text-aligned motion for walking, turning, and cartwheel prompts than MDM, MLD, and T2M-GPT.The qualitative comparison reports failures in cartwheel execution, realism, or turning trajectory for the competing methods.
- Inference Speed: MMM is at least two times faster than autoregressive and latent-space diffusion models and two orders of magnitude faster than motion-space diffusion models.The comparison is performed on a single NVIDIA RTX A5000.
- Long Sequence Generation: MMM generates a 10.873-minute motion sequence in 1.658 seconds.This result uses transition motion tokens for long-sequence synthesis.
- Editability: MMM supports motion in-betweening and upper-body editing with realistic motion and smooth transitions, outperforming MDM in both editing tasks.In-betweening generates the middle 50% conditioned on the first and last 25% of a sequence.
6. Ablation Study
The ablation study examines training mask ratios, inference iterations, and codebook design as factors affecting motion quality, speed, and representation fidelity.
- Masking Ratio: Training with either too aggressive masking at α = 0.7 or too mild masking at α = 0 harms motion fidelity and diversity.The masking ratio is sampled uniformly between α and 1 during training.
- Iteration Number: Using 10 inference iterations achieves the lowest FID while maintaining high R-precision and low MM-dist.More iterations increase latency and slightly improve R-precision and MM-dist.
- Codebook Size: Codebook factorization enables a large codebook for high-resolution motion quantization and preserves fine-grained motion representations.It decouples code lookup from code embedding and projects matched codes into a higher-dimensional embedding space.
7. Conclusion
MMM combines discrete motion tokenization with conditional masked-token prediction to enable fast, high-fidelity, iteratively refined motion generation and editing.
- Conclusion: MMM uses a motion tokenizer and conditional masked motion transformer to synthesize motion from textual descriptions.The tokenizer converts 3D motion into discrete latent tokens, while the transformer predicts masked motion tokens from text tokens.
- Conclusion: MMM enables parallel iterative decoding, inherent motion editability, and competitive quality and speed relative to state-of-the-art methods.The paper reports at least two times faster generation than autoregressive models and two orders of magnitude faster generation than raw-motion diffusion models.
Supplementary Material
The supplementary material compares speed, quality, and editability across models and documents additional analyses of MMM’s inference and motion-generation behavior.
- Additional Analyses: The supplementary material includes analyses of codebook reset, word-embedding cross attention, confidence-based masking, mask scheduling, token sampling, and additional editing tasks.It also examines inference speed relative to motion length and long-sequence-related qualitative results.
- Inference Speed, Quality, and Editability: Supplementary Section B compares AITS, R-Precision, FID, and editability for MDM, MotionDiffuse, MLD, T2M-GPT, AttT2M, and MMM.AITS is measured on HumanML3D without model or data loading time, using a single NVIDIA RTX A5000.
- Inference Speed, Quality, and Editability: Motion-space diffusion supports editing but requires nearly three times the generated sequence duration for inference in the reported comparison.MDM takes 28.112 seconds to generate 196 frames, equivalent to a 9.8-second motion sequence.
C. Codebook Reset
Codebook reset is necessary to prevent collapse and preserve motion detail, but the best reset frequency differs between tokenizer reconstruction and text-to-motion generation.
- Codebook collapse causes FID score, R-precision, and Multi-modal Distance to worsen significantly.
- Less frequent resets underutilize the codebook and reduce fine-grained detail, producing worse quality at 60 to 80 reset iterations.
- Stage 1 favors resetting every 40 iterations, whereas the text-to-motion task performs best with resets every 20 iterations.The objectives differ because Stage 1 reconstructs motion without text, while Stage 2 generates motion conditioned on text.
- Increasing cross-attention layers improves R-Precision but worsens FID score.
- Confidence visualization tracks motion-token confidence across 10 generation iterations over 49 temporal tokens.The 49 tokens represent 196 frames, with four frames compressed into each token.
F. Inference Speed Relative to Motion Length
MMM’s inference time scales with motion length and remains faster than the compared approaches for both short and long sequences, supporting efficient long-range synthesis.
- MMM’s total dynamic iterations are proportional to motion-token length, with Tdyn = T · L/M when L ≤ M.The mask count is determined by a decaying schedule over iteration t, maximum iteration T, sequence length L, and maximum input length M.
- 0.018 seconds is sufficient for generating a 40-frame sequence, faster than all compared state-of-the-art approaches.
- MMM is faster than the compared methods for long sequences and also faster for short-sequence generation.Diffusion methods show inference time independent of generated-frame count, while token-based baselines slow down on long sequences.
- A 10.873-minute motion sequence is generated in 1.658 seconds by combining short motions with transition tokens.Transition generation can be completed in a single iteration because transitions are short and decoding is parallel.
- HumanML3D is heavily skewed toward 196-frame samples, with around 1680 such samples and fewer than 200 samples at each shorter length.This distribution may make the reported average speed higher than it would be under equal counts by length.
- Among tested schedules, linear masking achieves the best FID at 15 iterations and the best Top-1 R-precision of 0.519.Square Root performs worst in the reported comparison.
H. Token Sampling Strategies for Parallel Decoding
MMM samples masked motion tokens in parallel using temperature, top-k, or top-p strategies, with performance varying across fidelity and text-alignment metrics.
- Parallel decoding: Masked tokens are decoded in parallel after each iteration, with stochastic sampling based on prediction confidences.The method evaluates temperature, top-k, and top-p sampling for parallel motion-token decoding.
- Sampling strategies: Temperature β = 1 achieves the best overall generation performance measured by FID while maintaining competitive performance on other metrics.
- Sampling strategies: Top-k sampling restricts sampling to the k most probable codebook entries, and using 100% yields the best reported performance.Using 100% is mathematically equivalent to temperature sampling with β = 1.
- Sampling strategies: Top-p sampling selects the smallest set of codebook entries whose cumulative probability exceeds threshold p, and p = 0.1 improves FID but worsens R-precision, MM-Dist, and MModality.
- Temporal motion editing: MMM supports temporal motion outpainting, temporal completion with text, and temporal completion without text.
- Body-part editing: Lower-body mask tokens provide controllable adjustment when combining upper- and lower-body motion from different prompts.This addresses the out-of-distribution issue caused by combining body-part distributions absent from the dataset.
- Long sequence generation: Long sequences are formed by generating prompt-specific motions and transition tokens between them.The transition tokens connect the end of one motion sequence to the start of the next.
J. Limitations
MMM has scope limitations for exceptionally long single textual descriptions and interactive motions involving multiple individuals.
- MMM may struggle to render fine-grained details for exceptionally long single textual descriptions because training datasets contain motions up to 196 frames.The paper proposes segmenting lengthy descriptions into shorter prompts and does not support interactive multi-person motion generation.