PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Motion Diffusion Model

Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, Amit H. Bermano

arXiv:2209.14916v2cs.CVcs.GR

TL;DR

Human motion generation remains difficult because of diverse motions, costly data, and ambiguous many-to-many text descriptions, while diffusion models are expressive but resource hungry and hard to control. MDM adapts classifier-free diffusion with a transformer architecture, sample prediction, and geometric losses for multiple motion-generation tasks. It achieves state-of-the-art conditioned-generation results with lightweight training resources, although inference requires about 1000 forward passes and roughly one minute per result.

  • Problem

    Human motion generation is challenging because motion is diverse, data are costly, and text-to-motion descriptions form a many-to-many mapping; diffusion models are promising but resource hungry and hard to control.

  • Method

    MDM is a transformer-based classifier-free diffusion model that predicts motion samples and supports geometric losses, varied conditioning, and multiple generation tasks.

  • Results

    MDM achieves state-of-the-art results on text-to-motion and action-to-motion benchmarks, and evaluators preferred its motions over real motions in 42% of cases.

  • Takeaways & Limitations

    MDM combines generative quality, diversity, lightweight training, and broad conditioning across several human motion generation tasks.

  • Takeaways & Limitations

    Inference is slow, requiring about 1000 forward passes and approximately one minute for a single result.

Abstract

from arXiv · show

Natural and expressive human motion generation is the holy grail of computer animation. It is a challenging task, due to the diversity of possible motion, human perceptual sensitivity to it, and the difficulty of accurately describing it. Therefore, current generative solutions are either low-quality or limited in expressiveness. Diffusion models, which have already shown remarkable generative capabilities in other domains, are promising candidates for human motion due to their many-to-many nature, but they tend to be resource hungry and hard to control. In this paper, we introduce Motion Diffusion Model (MDM), a carefully adapted classifier-free diffusion-based generative model for the human motion domain. MDM is transformer-based, combining insights from motion generation literature. A notable design-choice is the prediction of the sample, rather than the noise, in each diffusion step. This facilitates the use of established geometric losses on the locations and velocities of the motion, such as the foot contact loss. As we demonstrate, MDM is a generic approach, enabling different modes of conditioning, and different generation tasks. We show that our model is trained with lightweight resources and yet achieves state-of-the-art results on leading benchmarks for text-to-motion and action-to-motion. https://guytevet.github.io/mdm-page/ .

1 INTRODUCTION

MDM adapts classifier-free diffusion to human motion, targeting the domain’s many-to-many expressiveness while remaining lightweight and supporting geometric motion losses. Its generic framework handles multiple conditioning and generation tasks, achieving state-of-the-art results on text-to-motion and action-to-motion benchmarks.

  • Motivation: Human motion generation is difficult because motions are diverse, high-quality data are costly, and text-to-motion labels create a many-to-many mapping.A label such as “kick” can describe different actions, while one motion can have many descriptions.
  • Motivation: Diffusion models are promising for human motion because they avoid restrictive target-distribution assumptions, but they are resource demanding.Earlier auto-encoder and VAE approaches limit the learned distribution through one-to-one or normal-latent assumptions.
  • Method: MDM is a transformer-based, diffusion generative model adapted to motion, combining domain knowledge with geometric losses and lightweight training.The model is designed for temporal, non-spatial joint data rather than using the ubiquitous U-net backbone.
  • Capabilities: MDM supports text-to-motion, action-to-motion, unconditioned generation, completion, and semantic editing through generic conditioning and diffusion-based inpainting.Joint-space inpainting edits specific body parts while preserving others; textual conditioning guides gap filling while maintaining input semantics.
  • Results: MDM achieves state-of-the-art results on HumanML3D, KIT, HumanAct12, and UESTC, while a user study preferred its motions over real motions 42% of the time.The framework is reported to require about three days of training on a single mid-range GPU.

2 RELATED WORK

Prior work uses motion guidance, shared language-motion latent spaces, and unsupervised manifold learning, while diffusion models provide an alternative generative framework. Concurrent motion-diffusion methods exist, but MDM emphasizes lower resource requirements and geometric-loss-enabled design choices.

  • Motion generation: Motion-generation research conditions synthesis on motion prefixes, intermediate poses, or other signals using recurrent and transformer architectures.These approaches include prediction, in-betweening, and super-resolution tasks.
  • Text-to-motion: Text-to-motion methods commonly learn shared language-motion latent spaces with auto-encoders, VAEs, or CLIP-based representations.JL2P limits mapping to one-to-one text-motion relationships, while TEMOS and T2M use normal latent distributions.
  • Unsupervised generation: Unsupervised methods learn the human motion manifold without labels, including dedicated approaches such as MoDi.MDM is also evaluated in an unsupervised setting.
  • Diffusion models: Diffusion models gradually noise data and learn a reverse denoising process for sampling from the learned distribution.This generative framework has been applied across vision, graphics, and related domains.
  • Motion diffusion: Concurrent motion-diffusion methods motivate MDM’s emphasis on significantly fewer GPU resources and design choices that enable geometric losses.The comparison is framed around computational requirements and motion-specific loss integration.

3 MOTION DIFFUSION MODEL

MDM synthesizes human motion under arbitrary conditions using a transformer-based diffusion process that predicts clean motion samples while supporting geometric regularization and editing. Its generic framework accepts multiple motion representations and conditioning modes, including conditional, unconditional, and inpainting-based generation.

  • Framework: MDM synthesizes a length-N human motion sequence x1:N from an arbitrary condition c, including audio, natural language, discrete classes, or the null condition.Motion can be represented using joint locations, rotations, or both.
  • Diffusion process: Instead of predicting diffusion noise, MDM predicts the clean sample x̂0 at each reverse-diffusion step and noises it back to obtain xt−1.Sampling starts from random noise xT and iterates from T to 1.
  • Geometric losses: Geometric losses regulate positions, velocities, and foot contact to encourage natural motion and mitigate artifacts such as foot sliding.The foot-contact formulation nullifies foot velocities when the feet touch the ground.
  • Model: The model uses an encoder-only transformer whose temporal awareness supports arbitrary-length motions and whose inputs combine noised motion, diffusion timestep, and condition embeddings.The timestep and condition are projected into the transformer dimension and combined with frame-wise motion projections.
  • Conditioning: Classifier-free training learns conditioned and unconditioned distributions in one model, enabling a diversity–fidelity trade-off during conditional sampling.The condition is randomly removed for 10% of training samples, and guidance combines conditional and unconditional predictions.
  • Editing: Diffusion inpainting enables temporal completion and spatial body-part editing during sampling without additional training, while preserving specified input portions.Textual conditions can guide the motion filled into a temporal gap.

4 EXPERIMENTS

MDM is evaluated across text-to-motion, action-to-motion, unconditioned generation, and editing-related settings using established datasets, metrics, user studies, and qualitative comparisons. Across these experiments, it achieves strong benchmark performance while supporting varied conditioning and lightweight training.

  • Text-to-motion: MDM is evaluated on text-to-motion benchmarks KIT and HumanML3D using relevance, distributional similarity, diversity, and multimodality metrics.HumanML3D contains 14,616 motions with 44,970 descriptions, while KIT contains 3,911 samples and is more limited in diversity.
  • Editing: The experiments also cover motion in-betweening and body-part editing, including text-conditioned gap filling and upper-body changes while preserving lower-body input joints.Figure 3 uses light blue frames for input motion and bronze frames for generated motion; in-betweening can be conditioned or unconditioned.
  • Text-to-motion: MDM achieves state-of-the-art FID, Diversity, and MultiModality on HumanML3D, indicating high-quality and diverse samples for individual text prompts.The comparison includes JL2P, Text2Gesture, and T2M using the metrics proposed for the benchmark.
  • Text-to-motion: In a KIT user study, MDM was preferred over compared models most of the time and over ground-truth samples in 42.3% of cases.The study involved 31 users comparing samples generated from the same prompts, with repeated evaluations.
  • Action-to-motion: For action-to-motion, MDM is evaluated on HumanAct12 and UESTC with FID, recognition accuracy, diversity, and multimodality metrics.The experiments use an action embedding in place of the text embedding and evaluate checkpoints selected by minimum FID.
  • Action-to-motion: MDM leads the reported action-to-motion results on both datasets, while HumanAct12 performance leads on three of four metrics and UESTC shows a clear gap from state-of-the-art.HumanAct12 results are averaged over 20 evaluations with 95% confidence intervals; the model is tested with and without foot-contact loss.

5 ADDITIONAL APPLICATIONS

MDM supports unconstrained synthesis and motion editing alongside conditioned generation. Its editing approach completes fixed temporal or spatial motion portions while preserving coherence and available conditions.

  • 5.1 MOTION EDITING: For in-betweening, MDM fixes the first and last 25% of a motion and generates the remaining 50% between them.This applies the same editing approach in the temporal domain.
  • 5.1 MOTION EDITING: For body-part editing, MDM fixes unchanged joints and generates the rest, producing smooth motion that follows the fixed input and any condition.The experiments edit upper-body joints only.
  • 5.2 UNCONSTRAINED SYNTHESIS: MDM evaluates unconstrained synthesis on HumanAct12 while ignoring its action labels.The dataset choice preserves compatibility with previous publications.
  • 5.2 UNCONSTRAINED SYNTHESIS: MDM uses the same architecture for conditioned and unconstrained settings, removing only the conditional input.In the unconstrained version, the temporal embedding is projected without a conditional signal.
  • 5.2 UNCONSTRAINED SYNTHESIS: MDM supports any condition while achieving plausible results in unconstrained synthesis and outperforming methods not designed for that setting.The evaluation compares MDM with MoDi, which was specifically designed for unconstrained generation.

6 DISCUSSION

MDM is a lightweight, versatile diffusion framework for human motion generation, using signal prediction and a transformer encoder to support geometric losses. Its main practical limitation is slow diffusion inference, despite the model’s small size.

  • 6 DISCUSSION: MDM uses classifier-free diffusion with a transformer encoder and predicts the signal rather than the noise.The design supports a lightweight model and the use of geometric losses.
  • 6 DISCUSSION: MDM lends itself to various human motion generation tasks and shows superiority in conditioned generation.The experiments also indicate limited sensitivity to the architecture choice.
  • 6 DISCUSSION: About 1000 forward passes are required for a single diffusion result, making inference substantially slower than conventional generation.For the small motion model, inference takes about a minute rather than less than a second.
Loading 2209.14916v2…