Source-linked AI summary
MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, Wanli Ouyang
TL;DR
Existing human-motion generators often accept only one control modality, motivating a unified model that can use text and pose inputs. MotionGPT quantizes these signals into discrete codes, embeds them in instructions for an LLM, and generates motion with efficient tuning. It demonstrates multimodal motion generation on established datasets while leaving broader modality validation for future work.
Problem
Existing motion-generation methods often support only a single control signal, limiting applications that require multiple modalities such as text and human poses.
Method
MotionGPT quantizes textual and pose conditions into discrete codes, combines them in unified instructions, and fine-tunes an LLM to generate human-motion sequences.
Results
MotionGPT demonstrates multimodal motion generation on HumanML3D and KIT-ML using 33 M training parameters and about 4 hours of training.
Takeaways & Limitations
MotionGPT provides a unified model for generating subsequent, preceding, or in-betweening motions from text and pose conditions.
Takeaways & Limitations
The paper validates MotionGPT only on text and human poses, leaving broader modalities such as music for future validation.
Abstract
from arXiv · showhide
Generating realistic human motion from given action descriptions has experienced significant advancements because of the emerging requirement of digital humans. While recent works have achieved impressive results in generating motion directly from textual action descriptions, they often support only a single modality of the control signal, which limits their application in the real digital human industry. This paper presents a Motion General-Purpose generaTor (MotionGPT) that can use multimodal control signals, e.g., text and single-frame poses, for generating consecutive human motions by treating multimodal signals as special input tokens in large language models (LLMs). Specifically, we first quantize multimodal control signals into discrete codes and then formulate them in a unified prompt instruction to ask the LLMs to generate the motion answer. Our MotionGPT demonstrates a unified human motion generation model with multimodal control signals by tuning a mere 0.4% of LLM parameters. To the best of our knowledge, MotionGPT is the first method to generate human motion by multimodal control signals, which we hope can shed light on this new direction. Visit our webpage at https://qiqiapink.github.io/MotionGPT/.
Introduction
MotionGPT addresses the limitation of single-condition motion generators by unifying multimodal control, including text and pose inputs, within an LLM-based framework. It discretizes motion information, combines language and motion prompts, and achieves multimodal generation with efficient tuning and training.
- Motivation: Existing motion-generation methods commonly support only one control condition, limiting their applicability to real-world digital-human scenarios.The motivating conditions include textual descriptions, music, and human poses.
- Framework: MotionGPT unifies text and pose control by formulating task instructions and input motions as prompts for human-motion generation.The framework can generate subsequent, preceding, or in-betweening motions from multimodal conditions.
- Method: MotionGPT adapts an LLM to motion generation by quantizing human poses into discrete codes and combining language and motion codes in instructions.The model is fine-tuned with LoRA, incorporating pose-sequence information while retaining motion priors from the original LLM.
- Efficiency: 33 M training parameters and about 4 hours of training demonstrate MotionGPT’s efficient multimodal motion-generation training.The reported training time is about 10% of that taken by other methods.
- Results: Experiments on HumanML3D and KIT-ML show strong motion generation with multiple control conditions, while joint multi-condition training outperforms single-signal training.The authors also report that more powerful LLMs produce superior motion-generation quality.
Related Work
Prior work spans general LLMs and diverse human-motion generation methods. MotionGPT draws on efficient LLM fine-tuning while introducing motion as a new modality for unified generation.
- Large language models: Large language models evolved from task-specific systems toward broad downstream-task capabilities through extensive data, parameters, and fine-tuning.Efficient strategies such as prompt tuning and LoRA address the difficulty of tuning models with extensive parameters.
- Research gap: MotionGPT differs from prior LLM work by introducing human motion as a new modality for language-model-based generation.This connects LLM progress with the multimodal control requirements of motion generation.
- Human motion generation: Human-motion generation has been studied under conditions including motion descriptions, actions, music, prior poses, and trajectories.Representative approaches include sequence-to-sequence models, transformer VAEs, and trajectory-conditioned generation.
Generator
MotionGPT formulates multimodal human-motion generation as instruction following for a fine-tuned LLM. It quantizes motion controls into discrete codes, combines task and control tokens in unified prompts, and decodes generated motion codes into human motions.
- Instruction Generation: MotionGPT represents text controls, pose controls, and task descriptions as discrete codes organized within a unified instruction format.The framework supports text and arbitrary numbers of human poses as control conditions.
- Motion Quantization: A motion VQ-VAE quantizes human poses into codebook indices and reconstructs human poses from the resulting latent representations.The nearest embedding in a learnable codebook supplies the discrete motion code, while the decoder produces reconstructed poses.
- LLM Fine-tuning: LoRA fine-tunes an LLM to generate motion-code sequences that complete task instructions conditioned on text and pose inputs.The predicted motion codes are trained against ground-truth motion codes with cross-entropy loss.
- Motion Decoding: The generated motion-code sequence is decoded into human motion using the VQ-VAE decoder.MotionGPT’s pipeline maps the LLM’s motion answer back to human motions after generation.
- Task Generalization: By changing instructions, MotionGPT addresses existing and new motion-generation tasks with a single general-purpose model.The formulation includes text-to-motion generation and motion generation conditioned on initial, last, or key poses.
Experiment
Experiments evaluate MotionGPT across datasets, control conditions, pretrained LLaMA variants, and training strategies. Results show competitive text-to-motion quality, robust multimodal control, improved pose consistency, and benefits from pretrained and joint training.
- Multiple control conditions: MotionGPT supports text, initial-pose, last-pose, and random-keyframe-pose conditions in its multiple-control experiments.Initial and last conditions use four input poses, while random keyframe conditions use 12 to 20 poses.
- Text-to-motion generation: MotionGPT achieves comparable text-to-motion performance across all metrics on HumanML3D and KIT-ML.MotionGPT-13B is compared with state-of-the-art methods on both test sets.
- Multiple control conditions: 0.520 and 0.367 FID values for initial or key pose controls outperform the text-only counterpart’s 0.567 on HumanML3D.Final-pose conditioning slightly decreases performance because the predetermined end pose makes generation more challenging.
- Pretrained language models: Pretrained LLaMA models provide motion priors that influence text-to-motion performance, compared with an unpretrained model.The study evaluates LLaMA-7B, LLaMA-13B, and an identically structured randomly initialized LLaMA with LoRA training parameters held constant.
- Training strategy: Joint training achieves better performance for all tasks than separate training for each task.The comparison uses MotionGPT-7B on the HumanML3D validation set.
- Pose consistency: Pose-conditioned generation produces better keyframe pose consistency than text-only generation.Consistency is evaluated for initial, last, and key pose conditions on the HumanML3D test set.
Conclusion and Limitations
MotionGPT generates human motion from multimodal control signals by discretizing pose conditions and combining textual and pose prompts into unified instructions. The paper only validates text and human-pose controls, leaving broader modalities untested.
- Conclusion: MotionGPT generates human motion using multimodal control signals, including text and single-frame poses.It discretizes pose conditions and combines textual and pose codes into unified instructions.
- Conclusion: The method aims to support more practical and versatile motion-generation systems.
- Limitations: The paper validates MotionGPT only with text and human-pose controls, although the method may support additional modalities.Music is identified as one broader modality requiring future validation.
Implementation Details
The implementation preprocesses motion into structured features and fine-tunes a frozen 13B LLaMA model with LoRA. Training uses specified optimization settings and takes approximately four hours on HumanML3D and three hours on KIT-ML.
- Motion data pre-processing: Motion data is transformed to face the Z+ direction and converted into motion features before training.Features include foot contact, global rotations and translations, local joint positions, velocities, and 6D rotations.
- Training details: A frozen 13B LLaMA model is fine-tuned using LoRA as the foundational language model.
- Training details: Training uses 37,500 epochs, a 3e-3 initial learning rate, batch size 256, AdamW, and 0.01 weight decay.Micro-batches of 4 are used to accommodate memory constraints.
- Training details: Training takes approximately 4 hours on HumanML3D and 3 hours on KIT-ML.
Additional Experiments
Additional experiments examine LoRA settings, batch size, and prompt design for MotionGPT-7B on HumanML3D. Performance improves with larger LoRA rank, peaks at α=16, favors batch size 256, and benefits from prompt design.
- LoRA hyper-parameters: Increasing LoRA rank improves performance across almost all metrics when α remains constant.The scale factor α/r is compared with the learning rate in this analysis.
- LoRA hyper-parameters: The optimal performance is achieved when α is set to 16 with LoRA rank held constant.
- Batch size: Batch size 256 significantly outperforms batch sizes 128 and 512 across nearly all metrics.Batch sizes 128 and 512 produce comparable performance.
- Prompt design: The experiments compare the baseline prompt V0 with alternative prompts V1 and V2 for text-to-motion generation.The alternatives modify how task and motion-generation conditions are expressed.
- Prompt design: Prompt design comparisons indicate that carefully designed prompts improve model performance.
Qualitative Results
Qualitative examples show MotionGPT-13B generating motions from text alone and from text combined with initial, last, or key pose tokens. The examples use HumanML3D test-set descriptions, with pose conditions highlighted in blue.
- Text-to-motion: MotionGPT-13B generates text-to-motion samples from HumanML3D test-set descriptions.
- Initial-pose control: Text combined with initial pose tokens provides conditions for generating subsequent motions.The initial pose condition is highlighted in blue.
- Last-pose control: Text combined with last pose tokens provides conditions for generating preceding motions.The last pose condition is highlighted in blue.
- Key-pose control: Text combined with key pose tokens provides conditions for generating in-betweening motions.Key pose conditions are highlighted in blue.