Source-linked AI summary
MoMask: Generative Masked Modeling of 3D Human Motions
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, Li Cheng
TL;DR
Text-to-motion generation needs precise motion tokenization and efficient generation beyond conventional VQ and unidirectional or costly diffusion-based decoding. MoMask combines hierarchical residual quantization with masked and residual transformers, achieving state-of-the-art benchmark performance and supporting text-guided inpainting. Its main limitations are relatively limited diversity, requiring target length input, and challenges with fast-changing root motions.
Problem
Existing motion tokenization introduces quantization errors, unidirectional decoding limits context and accumulates errors, and discrete diffusion methods can require hundreds of iterations.
Method
MoMask uses hierarchical residual vector quantization, a text-conditioned Masked Transformer for base tokens, and a Residual Transformer for subsequent-layer tokens.
Results
FID is 0.045 versus 0.141 on HumanML3D and 0.204 versus 0.514 on KIT-ML, while MoMask also supports text-guided motion inpainting.
Takeaways & Limitations
MoMask provides efficient, high-quality text-to-motion generation and supports related motion inpainting tasks without further model fine-tuning.
Takeaways & Limitations
MoMask has relatively limited diversity, requires target length as input, and may struggle with fast-changing root motions such as spinning.
Abstract
from arXiv · showhide
We introduce MoMask, a novel masked modeling framework for text-driven 3D human motion generation. In MoMask, a hierarchical quantization scheme is employed to represent human motion as multi-layer discrete motion tokens with high-fidelity details. Starting at the base layer, with a sequence of motion tokens obtained by vector quantization, the residual tokens of increasing orders are derived and stored at the subsequent layers of the hierarchy. This is consequently followed by two distinct bidirectional transformers. For the base-layer motion tokens, a Masked Transformer is designated to predict randomly masked motion tokens conditioned on text input at training stage. During generation (i.e. inference) stage, starting from an empty sequence, our Masked Transformer iteratively fills up the missing tokens; Subsequently, a Residual Transformer learns to progressively predict the next-layer tokens based on the results from current layer. Extensive experiments demonstrate that MoMask outperforms the state-of-art methods on the text-to-motion generation task, with an FID of 0.045 (vs e.g. 0.141 of T2M-GPT) on the HumanML3D dataset, and 0.228 (vs 0.514) on KIT-ML, respectively. MoMask can also be seamlessly applied in related tasks without further model fine-tuning, such as text-guided temporal inpainting.
1. Introduction
MoMask addresses quantization error, limited unidirectional context, and costly bidirectional decoding in text-to-motion generation with hierarchical residual quantization and two transformers. It achieves state-of-the-art results while generating layered tokens efficiently and supporting motion inpainting.
- Existing VQ-based motion generators suffer quantization errors, unidirectional context limits, and error accumulation, while discrete diffusion methods require hundreds of iterations.These drawbacks motivate a more precise and efficient bidirectional generation framework.
- MoMask uses hierarchical residual vector quantization to progressively reduce quantization errors and represent motions with multi-layer discrete tokens.The base layer performs standard quantization, while later layers capture residual coding errors.
- Its Masked Transformer predicts randomly masked base-layer tokens conditioned on text, while the Residual Transformer progressively predicts tokens in subsequent layers.The two transformers separately handle base-token generation and residual refinement.
- The Masked Transformer generates complete base-layer sequences through iterative parallel prediction, retaining high-confidence tokens and remasking uncertain ones.The masking ratio is scheduled from 0 to 1 during training.
- 15 iterations generate the entire set of layered motion tokens regardless of motion length, and the pipeline also supports text-guided motion inpainting.MoMask is presented as both efficient and applicable to related tasks without further fine-tuning.
- FID is 0.045 versus 0.141 on HumanML3D and 0.204 versus 0.514 on KIT-ML, establishing new state-of-the-art text-to-motion performance.The comparisons use the baseline reported in [49].
2. Related Work
Related work spans neural motion generation, discrete motion tokenization, and generative masked modeling. MoMask combines hierarchical residual tokenization with masked and residual transformers for human motion synthesis.
- Human Motion Generation: Neural motion generation has been conditioned on motion prefixes, action classes, audio, and text, following earlier deterministic approaches that produced averaged and blurry motions.Stochastic models, GANs, temporal VAEs, and transformers were subsequently explored.
- Deep Motion Quantization and RVQ: MoMask introduces generative masked modeling for human motion synthesis alongside residual-based multi-layer tokenization and residual-token prediction.Its approach overview combines a residual quantizer, Masked Transformer, and Residual Transformer.
3. Approach
MoMask represents motions with hierarchical residual tokens and uses bidirectional transformers to generate base tokens before refining them with residual layers. Its inference process iteratively fills masked base tokens, predicts residual tokens, and decodes the complete representation into motion.
- Motion Residual VQ-VAE: MoMask tokenizes each motion into V + 1 ordered code sequences using residual quantization layers.The base sequence carries the most prominent information, while later layers add progressively finer details.
- Motion Residual VQ-VAE: Residual quantization recursively approximates each residual with a code sequence, sums the quantized sequences, and decodes the result into a reconstructed motion.The residual VQ-VAE is trained with motion-reconstruction and latent-embedding losses.
- Motion Residual VQ-VAE: Quantization dropout randomly disables the last 0 to V layers during training so early layers restore motion while later layers learn missing details.After training, each motion is represented by V + 1 discrete token sequences.
- Masked Transformer: The M-Transformer predicts randomly masked base-layer tokens conditioned on text, using a scheduled masking ratio and simultaneous masked-token prediction.During generation, high-confidence predictions remain fixed while low-confidence tokens are remasked and repredicted.
- Residual Transformer: The Residual Transformer predicts tokens in a selected residual layer in parallel from preceding-layer embeddings, text embeddings, and the layer indicator.Its prediction-layer and next-layer token-embedding parameters are shared for more efficient learning.
- Inference: Inference starts with an empty masked base sequence, iteratively generates it in L iterations, progressively predicts residual layers, and decodes all tokens through the RVQ-VAE.Classifier-free guidance is applied to both transformers during inference.
4. Experiments
MoMask is evaluated on HumanML3D and KIT-ML across quality, efficiency, user preference, ablations, and temporal inpainting. It achieves strong benchmark performance, favorable quality-efficiency trade-offs, and supports smooth text-guided inpainting.
- Evaluation setup: HumanML3D and KIT-ML provide the two motion-language benchmarks used for evaluation.HumanML3D contains 14,616 motions and 44,970 descriptions, while KIT-ML contains 3,911 motions and 6,278 descriptions.
- Quantitative comparison: MoMask attains state-of-the-art performance on both datasets, improving FID, R-Precision, and multimodal distance.The reported comparison covers multiple quality and alignment metrics across both benchmarks.
- Efficiency and user study: MoMask positions favorably between generation quality and inference efficiency compared with baseline methods.Inference time is averaged over 100 samples on a single Nvidia2080Ti.
- Efficiency and user study: MoMask is preferred by users most of the time in HumanML3D comparisons and receives 42% preference against ground-truth motions.The study involved 42 qualified users and compared MoMask with MDM, MLD, and T2M-GPT.
- Inference analysis: Inference sweeps identify s = 4 as an accuracy-fidelity trade-off point and L = 10 iterations as sufficient for masked decoding.The sweep varies guidance scale and inference iteration count.
- Component analysis: Residual quantization and quantization dropout improve motion reconstruction and generation, while replacing-and-remasking and residual quantization support faithful generation.These effects are reported in component ablations against prior VQ-based approaches.
- Temporal inpainting: MoMask generates smooth motions for temporal inpainting when the missing region occurs in the prefix, middle, or suffix.The method masks tokens in the selected region and applies the same inference procedure used for generation.
5. Discussion and Conclusion
MoMask delivers faithful and efficient text-driven motion generation while extending to temporal inpainting. The authors identify limited diversity, a target-length requirement, and challenges with fast-changing root motions as limitations.
- Limitations: MoMask's diversity is relatively limited despite strong fidelity and faithfulness in text-to-motion synthesis.The authors identify understanding the cause of this limitation as future work.
- Limitations: MoMask requires the target motion length as input, although text2length sampling could be applied beforehand.This requirement is presented as a limitation of the current generation setup.
- Limitations: Like most VQ-based methods, MoMask may struggle with fast-changing root motions such as spinning.The paper points to supplementary videos for exemplar cases.
- Conclusion: MoMask is described as efficient and flexible, with superior performance and support for temporal inpainting in multiple contexts.The conclusion attributes the framework's capabilities to residual quantization, masked transformers, and residual transformers.