Source-linked AI summary
MotionGPT: Human Motion as a Foreign Language
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, Tao Chen
TL;DR
Existing methods lack a general pre-trained model that unifies human motion and language, despite motion’s semantic coupling with language. MotionGPT addresses this gap by tokenizing motion and jointly modeling motion and text with prompt-based instruction tuning, achieving competitive performance across diverse motion tasks using one model. Its scope remains limited to articulated human bodies and excludes human-object and human-environment interactions.
Problem
Existing approaches treat motion and language as separate modalities, while a general pre-trained model supporting numerous motion-relevant tasks through prompts has yet to emerge.
Method
MotionGPT converts raw human motion into discrete motion tokens, combines them with text tokens in a unified vocabulary, and trains the resulting model with motion-language instructions.
Results
MotionGPT achieves competitive performance across text-to-motion, motion-to-text, motion prediction, and motion in-between tasks using one pre-trained generative model.
Takeaways & Limitations
A single prompt-based motion-language model can generate plausible human motion, describe motion, predict motion, and perform motion in-between tasks.
Takeaways & Limitations
MotionGPT is limited to articulated human bodies and does not model human-object or human-environment interactions.
Abstract
from arXiv · showhide
Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.
1 Introduction
MotionGPT addresses the absence of a general pre-trained model that unifies human motion and language by treating motion as a foreign language. It uses a shared motion-language framework and prompts to support diverse motion tasks with one model.
- Existing motion-language approaches treat motion and language as separate modalities and often require strictly paired motion-text data.
- Human motion’s semantic coupling with language motivates encoding motion and text within a single vocabulary for unified pre-training.
- MotionGPT converts raw motion into discrete tokens using a motion-specific VQ-VAE, creating a motion vocabulary analogous to English vocabulary.
- MotionGPT treats human motion as a foreign language and performs diverse motion tasks with a single uniform generative model.
- The framework is evaluated across text-to-motion, motion captioning, motion prediction, and motion in-between tasks.
2 Related Work
Prior work addresses motion synthesis and captioning with task-specific or modality-separated systems, while MotionGPT combines motion and language tokens for unified modeling. This design targets instruction-based support for diverse motion tasks.
- Human Motion Synthesis: Prior research has explored motion generation, motion captioning, and motion prediction as distinct human-motion tasks.
- Human Motion Synthesis: Text-to-motion systems commonly use language representations or diffusion models to condition motion generation.
- Human Motion Captioning: Motion captioning methods map motion to language using statistical models, recurrent networks, or discrete motion representations with translation networks.
- Language Models and Multi-Modal: Large language models provide unified text-to-text formulations, but language models have not been widely applied to motion tasks.
- Motion Language Pre-training: Existing text-to-motion methods generally support caption-to-motion generation but are limited in accepting context-specific user instructions.
- Motion Language Pre-training: MotionGPT combines motion tokens and text tokens, then jointly learns motion and language using a language model backbone.
3 Method
MotionGPT unifies human motion and language by converting motion into discrete tokens and processing both modalities through a shared vocabulary and language-modeling framework. Its three-stage pipeline trains a motion tokenizer, performs motion-language pre-training, and applies instruction tuning for diverse motion tasks.
- Motion Tokenizer: MotionGPT converts raw human motion into discrete motion tokens using a VQ-VAE tokenizer with an encoder, codebook, and decoder.The encoder produces latent vectors, quantization maps them to nearest codebook entries, and the decoder reconstructs motion sequences.
- Motion-aware Language Model: A unified vocabulary combines text tokens and motion-token indices, allowing inputs and outputs to represent language, motion, or mixtures of both.Motion boundary indicators are included alongside the original text vocabulary.
- Motion-aware Language Model: MotionGPT uses a transformer encoder-decoder to autoregressively predict target tokens from source sequences drawn from the unified vocabulary.The model predicts each next token conditionally and samples target sequences recursively during inference.
- Training Strategy: The training strategy has three stages: motion-tokenizer training, motion-language pre-training on mixed data, and instruction tuning.The tokenizer is kept fixed after its initial optimization, while later stages learn motion-language relationships and task-oriented behavior.
- Training Strategy: Instruction tuning formats 15 core motion tasks with varied prompts, producing more than one thousand instruction-defined tasks.Examples include text-based motion generation, motion captioning, and motion prediction; the paper reports gains on diverse and unseen tasks or prompts.
4 Experiments
Experiments evaluate MotionGPT as a uniform framework across datasets, motion tasks, and ablations. Results show competitive multi-task performance, with instruction tuning improving versatility but larger models offering limited gains.
- Datasets: HumanML3D provides 14,616 motion sequences and 44,970 sequence-level textual descriptions, while KIT provides 3,911 motion sequences and 6,353 textual descriptions.Both datasets use a consistent representation combining joint velocities, positions, and rotations for fair comparisons.
- Comparisons on Multiple Tasks: MotionGPT is evaluated across text-to-motion, motion-to-text, motion prediction, and motion in-between using a uniform benchmark.The benchmark compares MotionGPT with state-of-the-art methods across diverse tasks and datasets.
- Comparisons on Motion-to-Text: MotionGPT outperforms recent work on motion descriptions evaluated against directly used ground-truth texts on HumanML3D.The comparison uses unprocessed ground-truth descriptions, unlike prior reported results that ignored grammatical tense and plural forms.
- Comparisons on Motion Prediction and In-between: MotionGPT achieves the best motion completion quality and diversity on prediction and in-between tasks using motion-only AMASS data.Prediction uses approximately the first 20% of a sequence, while in-between completion masks approximately 50% randomly.
- Ablation Studies: The 220M model performs remarkably better than the 60M model, whereas larger models provide insignificant or occasionally worse results.Worse results are observed in some cases, including the motion in-between task.
- Ablation Studies: Instruction tuning enhances MotionGPT versatility and text-to-motion performance, but downgrades pure text-generation performance.The reported downgrade is attributed to the limited amount of paired textual descriptions and motions.
5 Disscusion
The discussion presents MotionGPT as a single prompt-based model for motion and language tasks, while identifying boundaries in the modeled entities and interactions. The supplementary materials provide qualitative results, experiments, implementation resources, and release plans.
- Limitations: MotionGPT’s scope is limited to articulated human bodies and excludes human-object and human-environment interactions.The discussion identifies faces, hands, animals, and interaction scenarios as directions for future modeling.
- Discussion: MotionGPT produces plausible human motion and natural-language descriptions through prompt-based instructions using one pre-trained generative model.The framework reports competitive results on motion generation, captioning, prediction, and in-between tasks.
- Supplementary Materials: Supplementary materials include qualitative comparisons, component experiments, inference-time analysis, vocabulary statistics, hyperparameter evaluations, user studies, and implementation details.A supplemental video presents text-to-motion, motion captioning, motion prediction, and other task results.
- Reproducibility: The authors state that code, data, and pre-trained models will be fully released, although large model files are difficult to upload.Example code covers training and evaluation processes and includes example results.
A Qualitative Results
Qualitative results compare MotionGPT with other methods for text-driven generation and motion captioning, and showcase unified outputs across several tasks. The figures emphasize motion-text alignment and grammatical quality.
- Text-driven Motion Generation: Figure 4 compares text-driven motion generation under the same HumanML3D training and inference setting.Red words and boxes identify misaligned motions, while the reported results indicate promising text understanding for motion generation.
- Unified Results: Figure 5 presents samples from MotionGPT for text-to-motion synthesis, motion captioning, and textual question-to-answer tasks.The samples come from the best model and use instruction-based motion-language supervision based on previous motion datasets.
- Motion Captioning: Figure 6 compares motion captioning under the same HumanML3D training and inference setting.Orange words mark matching results and red marks indicate incorrect grammar; the captions are described as better aligned and grammatically correct.
- Additional Experiments: Additional experiments include KIT text-to-motion evaluation, motion-tokenizer hyperparameters, training-scheme analysis, T2M-GPT comparisons, and motion-completion implementation details.These experiments are described as continuing the evaluation of MotionGPT models.
B.1 Text-to-Motion on KIT dataset.
On KIT, a 220M MotionGPT base model trained without pre-training achieves performance comparable to previous state-of-the-art methods for text-driven motion generation.
- Text-to-Motion on KIT: A 220M MotionGPT base model trained on KIT without pre-training achieves comparable performance to previous state-of-the-art methods.The evaluation follows the HumanML3D procedure and uses the benchmark’s settings.
B.2 Ablation on Motion Tokenizer.
The motion tokenizer is evaluated through VQ-VAE reconstruction and codebook-size ablations, with K = 512 selected for most experiments because it provides the best motion reconstruction.
- Tokenizer comparison: VQ-VAE improves motion reconstruction compared with the VAE models used in prior work.The comparison covers VPoser-t and ACTOR baselines on HumanML3D motion data.
- Codebook size: K = 512 shows the best performance of motion reconstruction and is therefore selected for most experiments.K denotes the motion codebook size.
B.3 Effectiveness of Training Scheme
The experiments examine training-stage combinations, task-specific tuning, inference efficiency, and motion-vocabulary usage. Full-stage training performs best on most metrics, task-specific fine-tuning can further improve individual tasks, and all model sizes support real-time animation.
- Training scheme: Full-stage MotionGPT achieves higher performance on most metrics than other training combinations.The ablation compares motion-language pre-training and instruction tuning using the same 600K iterations.
- Task-specific tuning: Further fine-tuning can enhance performance on specific tasks beyond the unified instruction-tuned model.Text-to-motion and motion in-between are used as illustrative examples.
- Unified framework: MotionGPT is contrasted with task-specific motion-generation systems because it uses a pre-trained language model within a unified framework.T2M-GPT is described as a VQ-VAE-and-Transformer framework for motion generation only.
- Inference efficiency: All MotionGPT model sizes can support real-time human animations and reach hundreds of FPS during inference.Inference time is evaluated with batch size one on Tesla V100 hardware.
- Motion vocabulary: The motion codebook is used as a concise but informative vocabulary across motions sampled from the HumanML3D test set.The analysis counts usage of each motion-vocabulary item generated by the tokenizer.
E Evaluation of Hyperparameters
Generation hyperparameters create a quality–diversity trade-off, while a user study evaluates realism and text alignment for text-to-motion and motion-to-text outputs.
- Sampling strategies: Greedy or beam search can slightly improve generation quality but significantly reduce motion diversity compared with sampling.Beam search expands the search space for improved sequence-probability matching.
- User study: MotionGPT is preferred over other state-of-the-art methods in both text-to-motion realism/alignment and motion-to-text description comparisons.The study uses 30 HumanML3D test descriptions for text-to-motion and 15 users for motion-to-text.
G Evaluation Protocols on the Uniform Motion-Language Generation.
The evaluation protocol converts prior motion datasets into instruction-based data spanning 14 core motion-language tasks and organizes metrics around matching, diversity, language, motion quality, and time cost.
- Uniform task protocol: The unified evaluation dataset contains 14 core motion-language tasks derived from previous datasets.Each task has dozens of instruction prompts, and the pre-processed dataset is planned for release.
- Metric classes: Evaluation metrics cover text-motion matching, generation diversity, linguistic quality, motion quality, and time cost.The linguistic metrics include BLEU, Rouge, CIDEr, and BERTScore.
- Motion quality: ADE is the average L2 distance over the whole predicted sequence, whereas FDE is the L2 distance at the final frame.Both metrics evaluate predicted motion against ground-truth motion.
- Time cost: Inference efficiency is measured as average Frames Per Second on HumanML3D with batch size one, excluding model and dataset loading.FPS evaluates computing efficiency during motion generation.
H.1 Implementation Details
The implementation evaluates smaller, base, and larger MotionGPT configurations with different parameter scales and transformer dimensions. The base model performs best overall under the current data regime, while larger models may benefit from substantially more data.
- Model configurations: MotionGPT is evaluated in smaller and larger configurations alongside a 220M-parameter base model.The smaller model uses 6 encoder and decoder layers; the larger model has 770 million parameters and 24 layers per transformer.
- Model configurations: The smaller configuration reduces computational requirements and provides faster inference.
- Performance and scaling: The base MotionGPT model achieves the best overall performance across tasks in the reported comparison.
- Performance and scaling: The authors suggest the base model’s advantage may reflect the small size of current motion datasets.
- Performance and scaling: The larger model could perform best when training data reaches millions or billions of examples.