Source-linked AI summary
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts
Chuan Guo, Xinxin Zuo, Sen Wang, Li Cheng
TL;DR
TM2T addresses limited bidirectional modeling of 3D human motions and language, including static or deterministic text2motion outputs. It introduces discrete motion tokens with autoregressive NMT mappings and inverse alignment, and reports superior performance on both tasks across two benchmark datasets.
Problem
Existing motion-language systems are often unidirectional, deterministic, and prone to static pose sequences, while motion2text results can be short or incomplete.
Method
TM2T quantizes motions into discrete tokens, maps motion and text tokens with autoregressive NMT models, and reuses motion2text for inverse alignment during text2motion training.
Results
TM2T achieves superior performance over varied state-of-the-art methods on both text2motion and motion2text across two benchmark datasets.
Takeaways & Limitations
The framework supports semantically faithful, sharp, diverse, and variable-length 3D motion generation while remaining usable for both mapping directions.
Takeaways & Limitations
Motion quantization is not lossless, long and complex text descriptions remain difficult, and the two mapping functions are trained independently.
Abstract
from arXiv · showhide
Inspired by the strong ties between vision and language, the two intimate human sensing and communication modalities, our paper aims to explore the generation of 3D human full-body motions from texts, as well as its reciprocal task, shorthanded for text2motion and motion2text, respectively. To tackle the existing challenges, especially to enable the generation of multiple distinct motions from the same text, and to avoid the undesirable production of trivial motionless pose sequences, we propose the use of motion token, a discrete and compact motion representation. This provides one level playing ground when considering both motions and text signals, as the motion and text tokens, respectively. Moreover, our motion2text module is integrated into the inverse alignment process of our text2motion training pipeline, where a significant deviation of synthesized text from the input text would be penalized by a large training loss; empirically this is shown to effectively improve performance. Finally, the mappings in-between the two modalities of motions and texts are facilitated by adapting the neural model for machine translation (NMT) to our context. This autoregressive modeling of the distribution over discrete motion tokens further enables non-deterministic production of pose sequences, of variable lengths, from an input text. Our approach is flexible, could be used for both text2motion and motion2text tasks. Empirical evaluations on two benchmark datasets demonstrate the superior performance of our approach on both tasks over a variety of state-of-the-art methods. Project page: https://ericguo5513.github.io/TM2T/
1 Introduction
TM2T studies bidirectional mappings between full-body 3D human motions and language, focusing primarily on text2motion. It introduces discrete motion tokens, inverse alignment, and autoregressive modeling to generate diverse, semantically faithful motions and support motion2text.
- Motivation: Existing bidirectional motion-language studies are limited and often produce static, deterministic sequences requiring an initial pose and target length.The reported static behavior occurs for motions longer than 3–4 seconds.
- Motion representation: Motion tokens compactly encode 3D motions into semantically rich codebook entries describing local spatial-temporal segments.The representation is learned through deep vector quantization on training pose sequences.
- Inverse alignment: The motion2text module is reused during text2motion training as inverse alignment, penalizing semantic deviation between generated and input text.The paper reports empirical benefit from this strategy for text2motion performance.
- Bidirectional modeling: Autoregressive NMT networks map between motion-token and text-token sequences, while sampled motion-token distributions enable non-deterministic text2motion generation.The approach can generate pose sequences with distinct lengths and styles from the same input script.
- Evaluation: Evaluations on two motion-language benchmark datasets report superior performance over varied state-of-the-art methods on both tasks.The framework is designed for both text2motion and motion2text.
2 Related Work
Prior work has separately studied motion captioning and text-conditioned motion generation, but integrated 3D motion-language modeling remains limited. Existing text2motion methods are predominantly deterministic, while motion2text systems often produce short, incomplete, or insufficiently detailed descriptions.
- Motion captioning: 3D human motion captioning has received considerably less attention than image and video captioning.Prior motion-captioning methods include statistical models, recurrent networks, autoencoders, and sequence-to-sequence RNNs.
- Bidirectional modeling: Existing efforts primarily address either motion2text or text2motion, with only two studies integrating both modalities.The integrated studies tend to produce static pose sequences for longer motions and require initial pose and target length.
- Motion captioning: Earlier motion2text results are typically short, often incomplete, and sometimes lack detail.The passage contrasts these limitations with the broader development of human motion modeling.
- Text2motion: Text2motion techniques include sequence-to-sequence RNNs, shared language-pose embeddings, and hierarchical skeleton-based generators.The passage notes that existing techniques are predominantly deterministic, unlike TM2T’s stochastic process.
3 Our Approach
TM2T first quantizes pose sequences into discrete motion tokens, then uses autoregressive NMT models for motion2text and text2motion. Inverse alignment reuses motion2text during text2motion training, and token sampling supports variable-length motion generation.
- Motion tokens: A 1D CNN latent quantization model reconstructs motions and converts them into discrete codebook-index sequences with [BOM] and [EOM] boundaries.The quantized sequence is decoded back into a pose sequence through a de-convolutional decoder.
- Inverse alignment: Inverse alignment feeds sampled motion tokens through the fixed motion2text model to align generated motions with the input language.Gumbel-Softmax approximates discrete sampling so alignment losses can be optimized.
- Motion tokens: The encoder maps pose sequences to latent vectors, which are replaced by nearest entries in a learnable codebook before decoding.Quantization operates row-wise by selecting the nearest codebook vector.
- Motion tokens: Reconstruction and embedding-commitment losses encourage latent alignment and stabilize quantization training.A straight-through estimator permits gradient backpropagation through the non-differentiable quantization operation.
- Motion token contexts: Each motion token captures local spatial-temporal characteristics, while global dependencies are maintained through token sequences.The codebook entries are visualized as 4-frame motion segments.
- Motion2text: Autoregressive NMT models translate motion tokens into text tokens by predicting target-token distributions conditioned on the source sequence.Motion2text training maximizes target log-likelihood through models such as Transformer.
- Text2motion: Text2motion predicts discrete motion tokens autoregressively from language using attentive GRU or Transformer backbones.During generation, tokens are sampled recursively until [EOM].
4 Experiments
Experiments evaluate TM2T on two motion-language datasets using objective metrics, user preferences, qualitative comparisons, and generation speed. The method outperforms baselines across both motion-to-text and text-to-motion tasks while supporting diverse motion outputs and identifying quantization and long-description limitations.
- Evaluation Setup: Experiments evaluate motion-to-text and text-to-motion mappings on HumanML3D and KIT-ML using extensive objective metrics.The evaluation includes language metrics, retrieval-based semantic alignment, multimodal distance, FID, diversity, and multimodality.
- Motion-to-Text Results: TM2T clearly outperforms all motion-to-text baselines across datasets and metrics, with motion-token modeling improving linguistic quality and retrieval precision toward real-description scores.The reported metrics include Bleu, BertScore, and R precision, while real descriptions provide a reference for semantic alignment.
- Motion-to-Text Results: Human users prefer TM2T descriptions most, while RAEs receives last-place rankings for 97% of descriptions.Users rank descriptions by accuracy and fluency, and TM2T receives the strongest overall appreciation among the evaluated methods.
- Qualitative Comparisons: Qualitatively, TM2T produces fluent, descriptive captions covering body parts, actions, and spatial relations more accurately than competing approaches.Other methods tend to produce simple, incomplete, or insufficiently detailed descriptions, while the no-motion-token variant can miss motion characteristics.
- Efficiency: TM2T generates 300 motions in 9 seconds on an Nvidia2080Ti, faster than most listed baselines except MoCoGAN and Dance2Music.The comparison reports 14 seconds for Seq2Seq, 10 for Language2Pose, 1 for MoCoGAN and Dance2Music, 250 for Text2Gesture, and 39 for Hier.
5 Conclusion
TM2T provides a bidirectional framework for mapping 3D human motions and text, using motion tokens and inverse alignment to support accurate descriptions and sharp, diverse motions.
- TM2T maps between 3D human motions and texts in both directions, integrating motion2text into text2motion learning through inverse alignment.
- Motion tokens compress 3D motions into short sequences of discrete variables for cross-modal mapping.
- The framework produces accurate descriptions and sharp, diverse 3D human motions.
- TM2T achieves state-of-the-art results on two motion-language datasets for both tasks.
A Data Preprocess
The preprocessing represents each motion with a normalized pose vector containing root, joint, rotation, velocity, and foot-contact information, while training adds perturbations to improve robustness.
- Pose vectors contain root angular velocity, root linear velocities, root height, local joint positions and velocities, 6D rotations, and foot contacts.The resulting vectors have 263 dimensions for HumanML3D and 251 for KIT-ML.
- Z-score normalization is applied to both HumanML3D and KIT-ML pose representations.
- The decoder additionally predicts foot-contact information to mitigate foot sliding during motion quantization.
- Root and foot-contact features are scaled by 5 to amplify their importance during quantization training.
- Training randomly cuts off 0 to 4 frames at the head or tail to improve robustness.
B Implementation Details
The implementation uses a 1024-entry motion codebook, convolutional motion quantization networks, and attention-based or recurrent sequence models for the two translation directions.
- The motion codebook contains 1024 embedding vectors, each with 1024 dimensions.
- Motion quantization uses 1D convolutional and upsampling encoder-decoder layers with residual blocks.
- Transformers for motion2text and text2motion use 4 and 3 attention layers, respectively, with 8 heads and hidden size 512.
- The GRU-based text2motion model uses hidden size 512 in its encoder and a 1-layer GRU decoder.
C Evaluation Metrics
Evaluation combines learned cross-modal feature extractors with semantic-alignment and motion-quality metrics, including R-Precision, multimodal distance, FID, diversity, and multimodality.
- Text and motion feature extractors bring matched pairs close and separate mismatched pairs by a margin m=10 using contrastive learning.
- FID compares generated and real motion feature distributions to evaluate overall generated-motion quality.
- Diversity measures variation across generated motions from different descriptions.
- Multimodality measures how much generated motions vary within each individual text description.
D Baseline Implementation
The experiments re-implement unavailable motion2text baselines and adapt several text2motion implementations for variable-length generation and the target scenario. Motion2text evaluation uses a six-description ranking survey on Amazon Mechanical Turk.
- The authors re-implement SeqGAN, RAEs, and Seq2Seq(Att) because motion2text baseline implementations were unavailable.
- Seq2Seq is re-implemented for text2motion, while Hier is extended with curriculum learning for variable-length motion generation.
- Text2Gesture and Language2Pose receive modifications to fit the scenario, including its kinematic structure.
- Six descriptions from different sources are randomly reordered for each animation, and master-recognized AMT users rank them by accuracy and detail.
F Motion Token Contexts
The quantization decoder maps individual motion tokens to short motion segments and combines token contexts through seamless transitions rather than simple concatenation.
- Each motion token decodes into a short 4-frame motion segment representing its local spatial-temporal context.
- For token tuples, the decoder naturally mingles local contexts with seamless transitions instead of simply concatenating decoded segments.
G Network Architecture
The HumanML3D networks combine convolutional motion encoding and decoding, a vector-quantized codebook, and recurrent attention-based translation components.
- Configuration: Input-vector dimensions may differ when the networks are applied to the KIT-ML dataset.
- Motion encoder and decoder: The motion encoder begins with Conv1d layers and LeakyReLU activations, using residual blocks to process motion features.
- Motion encoder and decoder: The decoder upsamples intermediate features and applies Conv1d layers to reconstruct motion representations.
- Motion tokenization: The architecture includes a 1024-dimensional codebook embedding for discrete motion representation.
- Sequence translation: Text or token inputs use embeddings, GRU cells, attention projections, softmax, and a linear attention layer for sequence modeling.
- Configuration: The HumanML3D configuration uses 2048-dimensional input features, 1024-dimensional hidden representations, a 128-dimensional mu output, and a 1027-dimensional target-word projection.