PaperlayerView exact revision on arXiv

Source-linked AI summary

Executing your Commands via Motion Diffusion in Latent Space

Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, Jingyi Yu, Gang Yu

arXiv:2212.04048v3cs.CVcs.GR

TL;DR

Conditional human motion generation must connect diverse motion sequences with disparate conditional modalities while handling redundancy and noise in raw motion data. MLD learns a low-dimensional motion representation with a VAE, then performs conditional diffusion in that latent space. It produces plausible, diverse, condition-conforming motions with substantially lower computational overhead and two orders of magnitude faster inference than prior raw-motion diffusion models.

  • Problem

    The problem is learning conditional mappings from modalities such as text or action classes to diverse human motions despite distribution differences and noisy, redundant raw sequences.

  • Method

    MLD first trains a transformer-based motion VAE to obtain representative, low-dimensional latent codes, then applies conditional diffusion to those codes.

  • Results

    MLD achieves competitive performance across unconditional, action-to-motion, and text-to-motion tasks, with two orders of magnitude faster inference than previous raw-motion diffusion models.

  • Takeaways & Limitations

    Latent-space diffusion provides an efficient framework for generating plausible and diverse human motions under action or language conditions.

  • Takeaways & Limitations

    MLD can generate arbitrary-length results only within the dataset maximum length and focuses on articulated human bodies.

Abstract

from arXiv · show

We study a challenging task, conditional human motion generation, which produces plausible human motion sequences according to various conditional inputs, such as action classes or textual descriptors. Since human motions are highly diverse and have a property of quite different distribution from conditional modalities, such as textual descriptors in natural languages, it is hard to learn a probabilistic mapping from the desired conditional modality to the human motion sequences. Besides, the raw motion data from the motion capture system might be redundant in sequences and contain noises; directly modeling the joint distribution over the raw motion sequences and conditional modalities would need a heavy computational overhead and might result in artifacts introduced by the captured noises. To learn a better representation of the various human motion sequences, we first design a powerful Variational AutoEncoder (VAE) and arrive at a representative and low-dimensional latent code for a human motion sequence. Then, instead of using a diffusion model to establish the connections between the raw motion sequences and the conditional inputs, we perform a diffusion process on the motion latent space. Our proposed Motion Latent-based Diffusion model (MLD) could produce vivid motion sequences conforming to the given conditional inputs and substantially reduce the computational overhead in both the training and inference stages. Extensive experiments on various human motion generation tasks demonstrate that our MLD achieves significant improvements over the state-of-the-art methods among extensive human motion generation tasks, with two orders of magnitude faster than previous diffusion models on raw motion sequences.

1. Introduction

Conditional human motion generation is difficult because language and motion have different distributions, while raw motion data can be redundant and noisy. MLD addresses these challenges by diffusing in a learned low-dimensional motion latent space rather than directly over raw motion sequences.

  • Motivation: Text-based motion synthesis offers a convenient interface for controlling virtual characters and humanoid robots.Applications include games, film, VR/AR, and robotic assistance.
  • Motivation: Different distributions between natural-language descriptors and motion sequences make the conditional probabilistic mapping difficult to learn.Prior approaches use cross-modal compatible latent spaces or conditional diffusion models.
  • Proposed Method: MLD learns a representative, low-dimensional motion latent distribution with a transformer-based autoencoder and performs conditional diffusion over latent codes.UNet-like long skip connections are used in the autoencoder, replacing diffusion over raw motion sequences.
  • Proposed Method: Latent-space diffusion produces vivid motions conforming to conditions while substantially reducing computational overhead in training and inference.The latent space can be pretrained on large-scale non-annotated or weakly annotated motion data such as AMASS.
  • Results: MLD achieves competitive performance across unconditional motion generation, action-to-motion, and text-to-motion tasks.The contributions also report state-of-the-art motion reconstruction and diverse generation from the proposed motion VAE.

2. Related Work

Related work spans multimodal motion synthesis, motion representations, VAEs, and raw-motion diffusion models. MLD separates a universal motion generator from task-specific latent diffusion models to support varied conditions while reducing the drawbacks of raw-motion diffusion.

  • Conditional Motion Synthesis: Human motion synthesis uses text, action categories, incomplete poses, control signals, music, and images as conditional inputs.The paper focuses on unconditional motion generation, action-to-motion, and text-to-motion.
  • Conditional Motion Synthesis: Most existing methods handle one task and provide limited flexibility in changing condition inputs.MLD separates a universal motion generative model from latent diffusion models for different motion generation tasks.
  • Motion Data: Motion datasets combine raw captured sequences with textual, action, or behavioral annotations, supporting text-to-motion and action-to-motion evaluation.AMASS-derived datasets include HumanML3D for text and Babel for action and behavior annotations.
  • Motion Representation: The framework uses SMPL-based motion parameters and redundant hand-crafted motion features, favoring the latter to avoid foot-sliding issues.The hand-crafted representation combines joint features and is mainly used in character animation.
  • Generative Models: VAEs are used to compress and reconstruct human motion for diffusion-model learning because GANs are more difficult to train.The paper follows prior recommendations favoring VAEs for motion generation.
  • Diffusion Models: Raw-motion diffusion models can be misdirected by noise and temporal redundancy, while also incurring high computational overhead and low inference speed.MLD applies diffusion in motion latent space to reduce computational resources and improve generative quality.

3. Method

MLD combines a transformer-based motion VAE with conditioned diffusion in a representative, low-dimensional motion latent space. It supports unconditional, text-to-motion, and action-to-motion generation while reducing the computational burden of diffusion on raw motion sequences.

  • Motion latent diffusion: MLD first learns a motion VAE and then trains a conditioned denoiser through diffusion in the learned latent space.During inference, the denoiser predicts a latent motion representation that the decoder reconstructs into a motion sequence.
  • Motion representation: The transformer-based VAE encodes variable-length motion features into a latent z and decodes z back into motion sequences.It uses reconstruction and KL losses, with long skip connections added to the encoder and decoder.
  • Motion latent diffusion: Diffusing in latent space addresses the inefficiency and noise sensitivity of diffusion models operating directly on raw motion sequences.The encoder is frozen during denoiser training, and the decoder reconstructs motions after iterative latent denoising.
  • Conditional generation: Conditional generation uses a shared motion VAE and condition-specific denoisers for text prompts, action labels, or empty conditions.Text and action embeddings are injected into the transformer denoiser, with concatenation found more effective than cross-attention in the described setting.
  • Qualitative comparison: On HumanML3D, MLD generations better match three text prompts than comparison methods, whose results include downgraded motions or improper semantics.The comparison uses the same training and inference setting for the evaluated methods.
  • Conditional generation: Classifier-free guidance trains conditioned and unconditioned distributions with 10% sample dropout, while guidance scale s > 1 strengthens conditioning.The guided denoising process predicts the latent representation that the decoder converts into motion.

4. Experiments

MLD is evaluated across text-to-motion, action-to-motion, and unconditional generation using multiple datasets and motion-quality, diversity, condition-matching, and efficiency metrics. The experiments report strong performance across tasks, with qualitative and quantitative comparisons against prior methods.

  • Experimental Settings: Experiments cover text-to-motion on HumanML3D and KIT, action-to-motion on UESTC and HumanAct12, and unconditional generation on HumanML3D motion data.The study uses multiple datasets and compares MLD with state-of-the-art methods across distinct motion-generation settings.
  • Evaluation Metrics: FID evaluates motion quality, Diversity and MultiModality evaluate generated-motion diversity, and R Precision and MM Dist evaluate condition matching.VAE reconstruction is additionally evaluated with MPJPE, PAMPJPE, and acceleration error.
  • Text-to-Motion: MLD achieves the best FID, R Precision, and MM Dist on HumanML3D and KIT while maintaining rich motion diversity and text-prompt matching.The reported results compare MLD with cross-modal and motion-diffusion models under repeated evaluations and confidence intervals.
  • Action-to-Motion: MLD achieves state-of-the-art accuracy and diversity on UESTC and competitive results on HumanAct12 for action-conditioned generation.The comparison includes ACTOR, INR, and MDM and reports FID on training and test splits.
  • Unconditional Generation: Unconditional generation is evaluated on part of AMASS using FID and Diversity, with comparisons against real motion, ACTOR, VPoser-t, and MDM.MLD supports both latent sampling and diffusion sampling, while this evaluation focuses on diffusion sampling.

5. Ablation Studies

The ablations examine how motion-VAE and latent-diffusion design choices affect representation quality, generation, and inference efficiency. Smaller latent representations and skip connections are particularly effective, while latent diffusion substantially reduces inference cost.

  • Ablation Scope: The ablations separately evaluate the motion VAE and latent diffusion models for reconstruction, generation, text-to-motion, unconditional synthesis, and inference time.The study treats the VAE and denoiser as jointly influencing MLD performance.
  • Motion Representation: The proposed VAEs outperform comparison models in motion-sequence representation according to the controlled evaluations.The study varies latent shape, skip connections, and transformer-layer count against ACTOR and VPoser-t baselines.
  • Latent Diffusion Components: MLD-1, using the smallest latent z ∈R1×256, achieves the best performance in most text-to-motion metrics.The ablation varies latent shape, condition injection, skip connections, and denoiser depth.
  • Latent Diffusion Components: Skip connections significantly improve motion latent diffusion, while different denoiser depths produce similar effects on the evaluated text-to-motion dataset.The smallest latent with skip connection also outperforms other configurations in unconditional diffusion sampling.
  • Inference Time: Two orders of magnitude faster speed is achieved by MLD than prior motion diffusion models, while also using less computational overhead and achieving higher performance.MDM requires 24.74 seconds for average inference and up to one minute for maximum inference on a Tesla V100.

6. Disscusion

MLD generates conditional human motions and is reported to be more efficient than raw-motion diffusion, while remaining bounded by dataset maximum sequence length and focused on articulated bodies.

  • The method can generate arbitrary-length results only within the maximum sequence length represented in the dataset.
  • MLD focuses on articulated human bodies rather than faces, hands, or animal motion.
  • MLD generates plausible human motion sequences conforming to action classes or natural language descriptions.
  • MLD produces more diverse and plausible motions than compatible cross-modal latent-space methods.
  • MLD uses less computational overhead and is two orders of magnitude faster than diffusion methods operating on raw motion sequences.

A. Qualitative Results

The qualitative evaluations compare MLD with state-of-the-art action-conditional methods and show text-conditioned and action-conditioned motion samples, alongside component studies.

  • Qualitative Results: MLD generations correspond better to action labels and exhibit richer diversity than the compared methods on HumanAct12.The comparison uses the same training and inference setting and generates three motions per action label.
  • Qualitative Results: MLD-1 provides additional text-to-motion samples generated from test-set prompts on HumanML3D.
  • Qualitative Results: The experiments study how language models and text-embedding shape affect motion generation.
  • Qualitative Results: The experiments also evaluate long skip connections and regularization in motion latent space.

B.1. Evaluation of Language Models τ w

The appendix evaluates language encoders, skip connections, latent-space regularization, denoising targets, inference cost, and latent diffusion trajectories through quantitative and visual studies.

  • B.1. Evaluation of Language Models τ_w: CLIP is better suited than BERT for the task, while word-wise tokens are competitive with a single token but reduce diffusion efficiency.The authors therefore choose CLIP with a single text token.
  • B.2. Effectiveness of Long Skip Connection: Long skip connections improve performance and accelerate convergence during diffusion training.The comparison is evaluated on HumanML3D under the same training process.
  • Latent-Space Regularization: Regularized motion latent spaces provide stronger generation ability and support latent diffusion models.
  • Denoising Target: Predicting the noise ϵ rather than directly predicting the denoised latent vector z_0 performs better in text-to-motion experiments.This result follows the denoising target used in DDPM.
  • Inference Cost: MLD reduces diffusion computational cost, and increasing diffusion iterations further widens the inference-cost gap.The evaluation measures total inference time, FLOPs, diffusion-input size, and FID.
  • Latent-Space Visualization: The t-SNE visualization tracks how latent codes evolve across reverse diffusion steps for different action labels.The visualization samples 30 motions for each action label, from initial random noise to the prediction.

E. Evaluation of Hyperparameters

Hyperparameter studies reveal a trade-off between text correspondence and motion quality under classifier-free guidance, while batch-size results remain close across settings.

  • Classifier-Free Diffusion Guidance: Increasing dropout p from 0.1 to 0.25 worsens text correspondence measured by R Precision but improves motion quality measured by FID.
  • Classifier-Free Diffusion Guidance: The same R Precision–FID trade-off occurs when changing guidance scale s from 7.5 to 2.5.The setting (0.25, 7.5) achieves the best FID of 0.229, but the authors recommend (0.1, 7.5) overall.
  • Batch Size: Batch-size results are close across 32, 64, 128, 256, and 512, with 64 used in the other experiments and 64 or 128 suggested for this task.

F. User Study

The user study compares MLD with prior methods and real motions, while the surrounding material describes the motion representations, VAE losses, and MLD architecture used in the system.

  • User Study: MLD was preferred over MDM and T2M, and was competitive with ground-truth motions in pairwise user comparisons.Twenty users evaluated motions from 30 randomly selected HumanML3D text descriptions using realism and text-matching questions.
  • User Study: The study used forced-choice comparisons asking which motion was more realistic and which better matched the text prompt.Comparisons included MLD versus MDM, MLD versus T2M, and MLD versus real dataset motions.
  • Motion Representations: The paper considers HumanML3D, SMPL-based, MMM, and latent motion representations for modeling human motion.The listed representations encode joint features, body parameters, skeleton coordinates, trajectories, or latent vectors.
  • VAE Training: The VAE loss combines a reconstruction data term with a low-weight regularization term, including KL regularization toward a standard Gaussian latent distribution.The motion reconstruction term uses mean squared error, while q(z|x1:L) is regularized against N(z; 0, 1).
  • Architecture: The MLD architecture comprises a motion encoder, motion decoder, and latent denoiser.These components form the conditional MLD network architecture.

H.3. Implementation Details

Implementation experiments compare conditional MLDs with different latent shapes and report that smaller latent shapes reduce computational requirements and speed inference.

  • Latent Shapes: MLD-7 performs best for VAE evaluation, whereas MLD-1 performs best on text-to-motion, action-to-motion, and unconditional generation tasks.The paper attributes the larger latent shape’s downgrade to the limited size of HumanML3D and suggests MLD-7 may benefit from million-scale motion data.
  • Computational Cost: Smaller latent shapes lower computational requirements and provide faster inference.The conditional models were trained on 8 Tesla V100 GPUs.

I. Metric Definitions

The evaluation defines metrics for motion quality, diversity, condition matching, and computational efficiency across human motion generation tasks.

  • Motion Quality: FID measures distribution similarity between generated and real motions, while MPJPE and PAMPJPE measure VAE reconstruction errors in millimeters.Acceleration Error is also listed among motion-capture reconstruction metrics.
  • Generation Diversity: Diversity measures variance across the generated motion set, while MultiModality measures diversity among motions generated for each text input.Diversity compares two randomly sampled subsets of generated motions; MultiModality samples two subsets for each selected text description.
  • Condition Matching: R Precision evaluates text-motion retrieval accuracy, MM Dist measures motion-text feature distance, and ACC evaluates action recognition accuracy.For R Precision, generated motions are mixed with 31 mismatched motions and assessed at top-1, top-2, and top-3 accuracy.
  • Time Costs: AITS measures average inference time per sentence in seconds with batch size one, excluding model and dataset loading.The measure is computed on the HumanML3D test set to evaluate diffusion inference efficiency.
Loading 2212.04048v3…