Source-linked AI summary
T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen
TL;DR
Text-to-motion generation needs precise mappings between language and motion, while prior methods can struggle with challenging descriptions. T2M-GPT combines Motion VQ-VAE with GPT over discrete motion codes, using EMA, Code Reset, and training corruption. It achieves competitive results against diffusion-based approaches, including FID 0.116 versus MotionDiffuse's 0.630 on HumanML3D, while larger datasets may improve performance.
Problem
Text-driven motion generation requires learning a precise mapping between different language and motion modalities, and prior methods can struggle with long, complicated descriptions.
Method
T2M-GPT uses a two-stage framework that discretizes motion with VQ-VAE and autoregressively generates code indices from text with GPT, using EMA, Code Reset, and sequence corruption.
Results
FID 0.116 on HumanML3D largely outperforms MotionDiffuse's 0.630 while achieving comparable R-Precision, and the method performs comparably or better than concurrent diffusion-based approaches.
Takeaways & Limitations
The results suggest that the classic VQ-VAE and GPT framework remains highly competitive for human motion generation.
Takeaways & Limitations
For excessively long texts, generated motion might miss details, and some sequences may slightly jitter in leg and hand movements.
Abstract
from arXiv · showhide
In this work, we investigate a simple and must-known conditional generative framework based on Vector Quantised-Variational AutoEncoder (VQ-VAE) and Generative Pre-trained Transformer (GPT) for human motion generation from textural descriptions. We show that a simple CNN-based VQ-VAE with commonly used training recipes (EMA and Code Reset) allows us to obtain high-quality discrete representations. For GPT, we incorporate a simple corruption strategy during the training to alleviate training-testing discrepancy. Despite its simplicity, our T2M-GPT shows better performance than competitive approaches, including recent diffusion-based approaches. For example, on HumanML3D, which is currently the largest dataset, we achieve comparable performance on the consistency between text and generated motion (R-Precision), but with FID 0.116 largely outperforming MotionDiffuse of 0.630. Additionally, we conduct analyses on HumanML3D and observe that the dataset size is a limitation of our approach. Our work suggests that VQ-VAE still remains a competitive approach for human motion generation.
1. Introduction
Text-driven motion generation must bridge language and motion modalities, especially for long and complicated descriptions. T2M-GPT addresses this with a simple two-stage VQ-VAE/GPT framework and reports competitive results against diffusion-based approaches.
- Generating motion from text could reduce the cost and time of motion capture for games, films, and robot animation.
- Long and complicated descriptions remain difficult for prior text-to-motion methods, which can produce motion inconsistent with the text.
- T2M-GPT uses a two-stage pipeline that maps motion sequences to discrete code indices, then trains GPT to generate those indices from pretrained text embeddings.
- EMA and Code Reset address VQ-VAE code collapse, while sequence corruption during GPT training alleviates the training-inference discrepancy.
- FID 0.116 on HumanML3D outperforms MotionDiffuse's 0.630 while achieving comparable R-Precision for text-motion consistency.
- The authors analyze quantization strategies and dataset size, and report state-of-the-art performance on HumanML3D and KIT-ML.
2. Related work
Prior work spans discrete and continuous representations for human motion synthesis, including VQ-VAE, autoencoders, VAEs, and transformer-based methods. Text-driven approaches increasingly target challenging descriptions and large datasets, while diffusion models have recently become prominent.
- VQ-VAE learns reconstruction with discrete representations and can decouple discrete-representation learning from prior learning.
- Naive VQ-VAE training can cause codebook collapse, motivating stop-gradient losses, EMA updates, and Code Reset for codebook optimization.
- Human motion synthesis includes prediction and animator-control approaches based on RNNs, GANs, GCNs, attention, MLPs, and convolutional autoencoders.
- The framework overview contains Motion VQ-VAE and T2M-GPT modules, with an End token stopping autoregressive code-index generation before decoding motion.
- Text-driven methods generate 3D human motion from descriptions, with prior systems addressing short texts, out-of-distribution descriptions, temporal compositions, and joint text-motion tasks.
- HumanML3D introduced a large-scale dataset, while concurrent work incorporated diffusion models into text-to-motion generation.
3. Method
The method uses a two-stage framework: a motion VQ-VAE converts motion into discrete codes, and T2M-GPT autoregressively generates those codes from text. The design combines convolutional encoding and decoding, codebook optimization, and training strategies that improve code utilization and robustness to inference-time errors.
- 3. Method: The framework maps motion sequences to discrete code sequences with a VQ-VAE, then generates code indices conditioned on text using a GPT-like transformer.The decoded code sequence is converted back into motion by the VQ-VAE decoder.
- 3.1. Motion VQ-VAE: Motion is encoded into latent features, quantized by selecting the most similar codebook entry, and decoded into reconstructed motion.Quantization assigns each latent feature to its nearest code according to Euclidean distance.
- 3.1. Motion VQ-VAE: VQ-VAE optimization combines reconstruction, embedding, and commitment losses, with smooth L1 reconstruction and velocity regularization used to improve motion reconstruction.The commitment coefficient β controls the commitment loss, while α balances reconstruction terms.
- 3.1. Motion VQ-VAE: EMA and Code Reset address codebook collapse by smoothly updating codebook entries and reassigning inactive codes from input data.The paper analyzes different quantization strategies.
- 3.1. Motion VQ-VAE: The motion VQ-VAE uses 1D convolutions, residual blocks, ReLU activations, and temporal downsampling and upsampling.Convolution with stride 2 performs downsampling, while nearest interpolation performs upsampling; the rate is l = 2^L.
- 3.2. T2M-GPT: T2M-GPT predicts motion-code indices autoregressively from CLIP text embeddings using causal self-attention and a sequence log-likelihood objective.The causal mask prevents future tokens from attending to the current-token calculation, and generation stops at the End token.
- 3.2. T2M-GPT: During training, randomly replacing τ×100% of ground-truth code indices reduces the discrepancy between teacher-forced training and inference conditions.τ may be fixed or sampled uniformly from [0, 1].
4. Experiment
Experiments on HumanML3D and KIT-ML evaluate reconstruction, text-motion consistency, distribution quality, diversity, and multimodality against established approaches. T2M-GPT performs competitively while benefiting from quantization recipes and additional training data.
- Datasets and evaluation: Experiments use HumanML3D and KIT-ML, with models selected by validation FID and evaluated on held-out test sets.HumanML3D contains 14,616 motions and 44,970 descriptions; KIT-ML contains 3,911 motions and 6,278 annotations.
- Datasets and evaluation: The evaluation measures text-motion retrieval, feature-distribution distance, multimodal distance, diversity, and multimodality.R-Precision uses motion-to-text retrieval, while FID compares generated and real motion-feature distributions.
- Comparison to state-of-the-art approaches: T2M-GPT matches MotionDiffuse on text-motion consistency across datasets while substantially outperforming it on FID.The comparison includes HumanML3D and KIT-ML and reports robustness across their different scales.
- Comparison to state-of-the-art approaches: Visual comparisons report higher-quality generations for T2M-GPT than Guo et al., MotionDiffuse, and MDM on HumanML3D.The comparison highlights distorted motions and sliding, while MDM generations are described as unrelated to the text.
- Analysis: EMA or Code Reset substantially improves VQ-VAE reconstruction and generation over naive quantizer training.The quantizer comparison uses the same VQ-VAE and GPT architectures and reports FID and Top-1 for reconstruction and generation.
- Discussion: Performance improves as the HumanML3D training subset grows, indicating that additional data could improve both reconstruction and generation.With only 10% of the data, motion quality may remain good while text correspondence is inadequate.
5. Conclusion
The paper concludes that a VQ-VAE-and-GPT framework remains competitive with diffusion-based motion-generation approaches. It also identifies larger datasets as a likely source of further improvement and documents practical limitations.
- Conclusion: VQ-VAE and GPT achieve comparable or better performance than concurrent diffusion-based approaches, supporting the competitiveness of this classic framework.The conclusion covers text-conditioned human-motion synthesis and includes analysis of quantization strategies.
- Conclusion: Larger datasets could bring additional improvement to the proposed approach.The paper reports dataset-size analysis as part of its broader investigation.
A. Ablation study of T2M-GPT architecture
The ablation study examines transformer scale and VQ-VAE reconstruction-loss choices for T2M-GPT. Larger transformers perform better, while L1 Smooth and α = 0.5 provide the reported preferred settings.
- T2M-GPT architecture: Larger transformer architectures obtain better performance, leading the authors to use an 18-layer, 16-head transformer with 1,024 dimensions.The selected architecture is the final configuration reported in the ablation.
- Reconstruction loss: L1 Smooth achieves the best VQ-VAE reconstruction performance, with L1 loss performing closely.The loss comparison is reported for models trained for 300K iterations.
- Reconstruction loss: α = 0.5 produces the best reported performance among the tested reconstruction-loss settings.The hyper-parameter α is part of the reconstruction-loss formulation.
C. Impact of τ for the corruption strategy in T2M-GPT training
The study evaluates corruption rates τ during T2M-GPT training. Using τ = 0.5 improves Top-1 accuracy and FID over uncorrupted training and offers better FID than uniformly sampled corruption.
- Corruption strategy: τ = 0.5 significantly improves Top-1 accuracy and FID compared with τ = 0.The ablation studies corruption of sequences during T2M-GPT training on HumanML3D.
- Corruption strategy: τ = 0.5 achieves comparable Top-1 accuracy to τ ∈U[0, 1] while obtaining much better FID on HumanML3D.The authors identify τ = 0.5 as probably preferable for this dataset.
D. Ablation study of the number of codes in VQ-VAE
The ablation examines how codebook size affects reconstruction and overall performance. A 512-code configuration performs slightly better than 1,024 codes, while 256 codes are insufficient for reconstruction.
- 512 codes perform slightly better than 1,024 codes.
- 256 codes are insufficient for reconstruction.
E.1. Evaluation metrics
The evaluation uses pretrained-network features for ground-truth motion, generated motion, and text, then measures generation quality, text-motion alignment, diversity, and multimodality.
- Evaluation features are extracted with pretrained networks from the HumanML3D evaluation setup.The features are denoted fgt for ground-truth motion, fpred for generated motion, and ftext for text.
- FID evaluates the overall quality of generated motion.It compares the means and covariance matrices of ground-truth and generated motion features.
- MM-Dist measures the average Euclidean distance between text features and generated-motion features.It is computed over N randomly generated samples and their corresponding text-motion pairs.
- Diversity measures variation among motion sequences across the dataset.The metric is computed from randomly sampled pairs of generated-motion features, using Sdis = 300 in the experiments.
- MModality measures the diversity of motions generated from the same text description.For each text, the method generates motion 30 times and compares two subsets containing 10 motions each.
F. VQ-VAE Architecture
The appendix specifies the Motion VQ-VAE architecture and evaluates convolutional dilation rates for reconstruction. A dilation-rate setting of (9, 3, 1) is reported as most effective and stable.
- VQ-VAE Architecture: The HumanML3D and KIT-ML motion features have dimensions 263 and 259, respectively.These dimensions are reported alongside the detailed Motion VQ-VAE architecture.
- Ablation study: The dilation-rate setting (9, 3, 1) gives the most effective and stable reconstruction performance.The comparison evaluates different dilation rates in the convolution layers of VQ-VAE on HumanML3D.
G. Limitations
The authors identify two limitations: very long texts may lose descriptive details, and some generated sequences may jitter in leg and hand movements.
- Limitations: Excessively long texts may cause generated motion to miss details of the textual description.The authors note that this failure case also exists for all competitive approaches.
- Limitations: Some generated motion sequences slightly jitter in leg and hand movements.The authors attribute this problem to the VQ-VAE architecture and suggest temporal smoothing as postprocessing for real applications.