Source-linked AI summary
MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, Ziwei Liu
TL;DR
Human motion generation seeks to produce diverse, fine-grained motions from natural-language inputs without requiring professional skills. MotionDiffuse introduces a diffusion-based framework with probabilistic mapping, realistic synthesis, and multi-level manipulation, and evaluations report improvements over existing methods across text-driven and action-conditioned tasks.
Problem
Existing text-conditioned motion methods struggle to generate diverse, fine-grained motions for varied language inputs, limiting accessible motion synthesis.
Method
MotionDiffuse uses a DDPM-based text-conditioned pipeline with soft text guidance, a Cross-Modality Linear Transformer, and controls for body parts and time-varied prompts.
Results
MotionDiffuse outperforms existing methods on text-driven and action-conditioned motion-generation tasks, while qualitative evaluations demonstrate controllable comprehensive generation.
Takeaways & Limitations
The framework provides probabilistic, high-fidelity motion synthesis with per-part manipulation and long-sequence generation from comprehensive text conditions.
Takeaways & Limitations
Inference requires many diffusion steps, making real-time motion generation challenging, and the current pipeline accepts only one motion representation.
Abstract
from arXiv · showhide
Human motion modeling is important for many modern graphics applications, which typically require professional skills. In order to remove the skill barriers for laymen, recent motion generation methods can directly generate human motions conditioned on natural languages. However, it remains challenging to achieve diverse and fine-grained motion generation with various text inputs. To address this problem, we propose MotionDiffuse, the first diffusion model-based text-driven motion generation framework, which demonstrates several desired properties over existing methods. 1) Probabilistic Mapping. Instead of a deterministic language-motion mapping, MotionDiffuse generates motions through a series of denoising steps in which variations are injected. 2) Realistic Synthesis. MotionDiffuse excels at modeling complicated data distribution and generating vivid motion sequences. 3) Multi-Level Manipulation. MotionDiffuse responds to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts. Our experiments show MotionDiffuse outperforms existing SoTA methods by convincing margins on text-driven motion generation and action-conditioned motion generation. A qualitative analysis further demonstrates MotionDiffuse's controllability for comprehensive motion generation. Homepage: https://mingyuan-zhang.github.io/projects/MotionDiffuse.html
1 Introduction
MotionDiffuse addresses the difficulty of generating diverse, fine-grained human motion from text by combining diffusion-based probabilistic generation with realistic synthesis and multi-level control. It supports body-part-specific instructions and time-varied prompts while outperforming existing methods across text-driven and action-conditioned tasks.
- Core framework: MotionDiffuse uses DDPM-based probabilistic generation rather than a deterministic language-motion mapping, increasing motion diversity through denoising-time variation.The framework softly guides generation with input text and processes text-conditioned noise during denoising.
- Core framework: A Cross-Modality Linear Transformer enables arbitrary-length motion synthesis instead of the fixed-size generation of classical DDPMs.The method generates motion according to the requested duration.
- Evaluation: The framework achieves state-of-the-art performance on two conditional motion generation tasks and improves text-driven generation on HumanML3D and KIT-ML.It also outperforms existing works on action-conditioned generation using HumanAct12 and UESTC.
- Multi-Level Manipulation: MotionDiffuse supports fine-grained text descriptions that coordinate multiple body parts, such as drinking while walking.Its design separately controls body parts while considering their correlations.
- Multi-Level Manipulation: Time-varied prompts specify ordered actions, such as walking and then running, enabling long motions with changing textual conditions.The paper evaluates temporal diversity through sequences containing multiple actions across different time intervals.
2 Related Work
Prior motion-generation research spans unconditional prediction, VAEs, GANs, normalizing flows, implicit representations, and conditional generation from actions or music. Text-driven generation introduces especially demanding requirements for body-part control and broad motion coverage, motivating MotionDiffuse’s diffusion-based pipeline.
- Earlier Motion Generation: Earlier motion-generation methods included unconditional synthesis, future-motion prediction, and statistical models such as PCA and Motion Graphs.These approaches addressed motion generation before the expansion of deep generative architectures.
- Deep Generative Models: Deep-learning approaches to motion generation include VAEs, GANs, Normalization Flow Networks, and Implicit Neural Representations.The methods differ in how they model, sample, or fit motion distributions.
- Deep Generative Models: VAEs reconstruct future motion from current sequences, while ACTOR uses a transformer-based encoder-decoder architecture.These methods represent major VAE-based directions in motion synthesis.
- Deep Generative Models: GAN-based methods use discriminator networks to assess generated motion quality and validity, including supervision without specific ground truth.HP-GAN is presented as an example of this strategy.
- Deep Generative Models: Normalization flows generate motion by mapping data to a multidimensional Gaussian distribution and reversing the transformation from random vectors.Implicit representations instead fit individual samples and can generalize across spatial or temporal dimensions.
- MotionDiffuse: MotionDiffuse introduces DDPMs for motion generation because the denoising process retains the original motion sequence and supports additional constraints and diverse samples.The paper evaluates the pipeline on text-driven and action-conditioned motion-generation datasets.
- Conditional Motion Generation: Conditional motion generation includes action-conditioned synthesis and music-to-dance generation, with methods using recurrent conditional VAEs, joint embeddings, or codebooks.Bailando generates complete dance sequences from quantized dance elements.
- Text-Driven Motion Generation: Text-driven motion generation requires accurate body-part control and covers a broad range of motions, from direct commands such as touching the head to concepts such as playing the violin.These requirements distinguish language conditioning from other conditional signals.
3 Methodology
MotionDiffuse is a diffusion model-based framework for text-driven motion generation that supports variable-length synthesis and fine-grained control. Its pipeline combines text-conditioned denoising with transformer components and additional controls for body parts and time-varying actions.
- Problem Definition: MotionDiffuse formulates text-driven motion generation as a text-to-motion translation task using paired motion sequences and language descriptions.During inference, descriptions condition the generation of motion sequences.
- Diffusion Model: The framework builds a text-conditioned DDPM pipeline that gradually denoises Gaussian noise into motion conditioned on text.The reverse process uses a learned network to predict noise during iterative denoising.
- Transformer Architecture: MotionDiffuse uses a Cross-Modality Linear Transformer with a text encoder and motion decoder because motion sequences are variable-length.The architecture customizes the motion decoder for diffusion-model denoising.
- Transformer Architecture: Efficient attention replaces quadratic pair-wise attention with a global feature map to reduce the cost of processing long motion sequences.The paper states that classical attention costs O(n^2d), while efficient attention improves speed for sequences with several hundred elements.
- Fine-grained Control: The explicit motion representation enables additional manipulation, including independent body-part control and time-varied conditioning for arbitrary-length sequences.Noise interpolation coordinates different body parts, while overlapped-sequence sampling supports continuous actions across time.
4 Experiments
MotionDiffuse is evaluated across text-driven generation, action-conditioned generation, and motion manipulation, using benchmark comparisons, ablations, and qualitative analyses. It outperforms prior methods across reported tasks while supporting diverse, accurate, fine-grained, and temporally extended motion generation.
- Experiment Overview: MotionDiffuse significantly outperforms previous state-of-the-art methods across text-driven, action-conditioned, and motion-manipulation evaluations.The experiments cover HumanML3D, KIT-ML, HumanAct12, and UESTC datasets.
- Text-driven Motion Generation: MotionDiffuse achieves strong precision, FID, multimodality distance, and diversity results on HumanML3D and KIT-ML, with precision close to real motions.The quantitative comparison includes Language2Pose, Text2Gesture, MoCoGAN, Dance2Music, and Guo et al. (2022).
- Ablation Studies: Removing pretrained CLIP causes severe performance drops, while efficient attention is beneficial with CLIP but can limit performance without it.The authors suggest that global relations may be misleading when text provides insufficient semantic information.
- Ablation Studies: A 512 latent dimension significantly outperforms 256, whereas adding layers helps at latent dimensions 128 or 256 but has little effect at 512.These architecture-size comparisons are reported on the KIT-ML test set.
- Text-driven Motion Generation: Qualitative comparisons show a balance between diversity and realness, including more faithful responses to golf-swing prompts and complex push-and-recovery instructions.Figure 4 compares MotionDiffuse with Guo et al. (2022) using two examples for each prompt.
- Action-conditioned Motion Generation: MotionDiffuse achieves the best FID and Accuracy on HumanAct12 and UESTC, with HumanAct12 results notably close to real motion.The action-conditioned comparison uses Action2Motion, ACTOR, and INR as baselines.
- Motion Manipulation: The T2M-S and T2M-T variants evaluate simultaneous body-part actions and ordered actions across different time intervals, respectively.Qualitative results show coordination across upper and lower body parts and generation of long sequences containing multiple actions.
5 Conclusion, Limitations and Future Work
MotionDiffuse combines probabilistic mapping, realistic synthesis, and multi-level manipulation, with evaluations showing advantages across motion-generation tasks. Its remaining limitations are slow diffusion inference and support for only one motion representation.
- MotionDiffuse provides probabilistic mapping, realistic synthesis, and multi-level manipulation for diverse motion generation.
- Quantitative and qualitative evaluations show MotionDiffuse outperforms existing methods across text-driven and action-conditioned motion-generation tasks.
- Diffusion inference requires many steps, making real-time motion-sequence generation challenging.
- The current pipeline accepts only a single motion representation, limiting versatility across datasets and scenarios.