PaperlayerView exact revision on arXiv

Source-linked AI summary

MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis

Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, Christian Theobalt

arXiv:2212.04495v3cs.CV

TL;DR

Human motion synthesis still faces challenges in variability, realism, and faithful conditioning. MoFusion applies denoising diffusion with scheduled kinematic losses, improving music- and text-conditioned synthesis while supporting interactive motion editing.

  • Problem

    Human motion synthesis remains challenged by limited motion variability, realism, and fidelity under user-specified conditioning.

  • Method

    MoFusion uses denoising diffusion with a time-varying loss-weight schedule to incorporate kinematic constraints for conditional motion synthesis.

  • Results

    MoFusion improves music-conditioned choreography and text-conditioned motion synthesis, producing diverse, temporally and kinematically plausible, semantically accurate motions.

  • Takeaways & Limitations

    MoFusion’s diffusion formulation supports conditional synthesis across music and text and enables interactive applications such as forecasting and inbetweening.

  • Takeaways & Limitations

    The kinematic losses are not exhaustive, and MoFusion has comparatively restricted textual vocabulary and inference time that could be improved.

Abstract

from arXiv · show

Conventional methods for human motion synthesis are either deterministic or struggle with the trade-off between motion diversity and motion quality. In response to these limitations, we introduce MoFusion, i.e., a new denoising-diffusion-based framework for high-quality conditional human motion synthesis that can generate long, temporally plausible, and semantically accurate motions based on a range of conditioning contexts (such as music and text). We also present ways to introduce well-known kinematic losses for motion plausibility within the motion diffusion framework through our scheduled weighting strategy. The learned latent space can be used for several interactive motion editing applications -- like inbetweening, seed conditioning, and text-based editing -- thus, providing crucial abilities for virtual character animation and robotics. Through comprehensive quantitative evaluations and a perceptual user study, we demonstrate the effectiveness of MoFusion compared to the state of the art on established benchmarks in the literature. We urge the reader to watch our supplementary video and visit https://vcai.mpi-inf.mpg.de/projects/MoFusion.

1. Introduction

MoFusion applies denoising diffusion models to conditional 3D human motion synthesis, targeting motions that are semantically aligned with conditioning while preserving diversity. It introduces time-varying kinematic-loss weighting and supports music- and text-conditioned generation with interactive motion applications.

  • Motivation: Conditional motion synthesis must balance semantic agreement with conditioning and diversity across motions generated from the same signal.Existing approaches include CVAEs, normalizing flows, and GANs, each with limitations such as mode collapse or a quality–diversity trade-off.
  • Method: MoFusion is presented as the first method for conditional 3D human motion synthesis using denoising diffusion models.The framework is motivated by diffusion models’ ability to learn data distributions while allowing diverse sampling.
  • Method: A time-varying weight schedule incorporates kinematic losses to make synthesized motions temporally plausible and semantically accurate.The method introduces domain-inspired kinematic losses during diffusion-model training.
  • Applications: MoFusion conditions motion synthesis on music and text through a framework architecture designed for both signals.The paper analyzes music-conditioned choreography generation and text-conditioned motion synthesis.
  • Applications: For music-to-choreography generation, MoFusion generalizes to new music without degenerate repetitiveness.This addresses the repetitive, loopy motions produced by many existing choreography-generation methods.

2. Related Works

Prior human-motion synthesis methods commonly rely on seed motion or action conditioning, while music-conditioned approaches can produce repetitive motions. MoFusion instead targets non-repetitive choreography without seed sequences and uses a distinct 1D U-Net design with time-varying kinematic-loss weighting.

  • Human Motion Synthesis: Traditional human-motion synthesis uses statistical or sequence modeling, typically guided by an initial seed sequence; synthesis from scratch requires conditioning.
  • Human Motion Synthesis: Action-conditioned methods commonly use class descriptions with conditional VAEs to learn latent motion representations.
  • Dance-Conditioned Motion Synthesis: Music-conditioned methods often predict motion from audio without seed motion but can converge to mean poses because dance contains repetitive poses.
  • Dance-Conditioned Motion Synthesis: MoFusion produces non-repetitive choreographies without seed motion, unlike existing music-conditioned methods that depend on seed input and often generate repetitive choreography.
  • Diffusion Models: Against concurrent diffusion-based methods, MoFusion uses a 1D U-Net with cross-modal transformers and applies a time-varying weighting schedule to kinematic losses.

3. Method

MoFusion formulates conditional human-motion synthesis as reverse diffusion from Gaussian noise, using a noise-prediction network conditioned on audio or text. It augments the diffusion objective with scheduled kinematic regularization and implements conditioning through a 1D U-Net with cross-modal transformers.

  • 3.1 Denoising diffusion: MoFusion generates motion by iteratively reverse-diffusing Gaussian noise while a neural network predicts the original noise under conditioning signal c.The forward process progressively adds Gaussian noise, and the learned reverse process reconstructs meaningful motion sequences.
  • 3.2 Kinematic losses: The training objective combines the L2 noise-estimation data loss with kinematic losses that regularize synthesized motion.The kinematic terms address artifacts such as jitter, illegal skeletons, and foot-sliding that the data loss alone does not prevent.
  • 3.2 Kinematic losses: A time-varying weight schedule assigns exponentially lower kinematic-loss weight near the noisiest diffusion steps, stabilizing training.The schedule varies with λ(t), ensuring motions at t≈T receive lower weight than those at t≈0.
  • 3.2 Kinematic losses: The regularizer includes skeleton consistency for temporally stable bone lengths and anatomical symmetry constraints on corresponding left/right bones.For joint-angle representations, joint-angle-limit regularization can replace the bone-length constraint.
  • 3.3 Architecture and conditioning: The denoising network is a fully convolutional 1D U-Net whose residual blocks are followed by cross-modal transformers that incorporate conditioning context.The architecture supports motions of varying lengths and can switch to self-attention for unconditional generation.
  • 3.3 Architecture and conditioning: Music conditioning uses log-Mel spectrograms, whereas text conditioning uses pre-trained CLIP token embeddings projected into the U-Net feature space.Audio is resampled to 16kHz and represented with 80 Mel bands; the context-embedding layer learns the projection for music features.

4. Experiments

MoFusion is evaluated for audio- and text-conditioned motion synthesis, with additional studies of interactive motion applications. Experiments report strong diversity and perceptual results while noting limitations of standard evaluation metrics.

  • Experimental scope: Experiments evaluate MoFusion on music-conditioned dance, text-conditioned motion, and applications including seed-motion forecasting, editing, and inbetweening.The evaluation is organized around music-to-choreography generation, text-conditioned motion generation, and interactive applications.
  • Music-to-choreography: AIST++ provides 1,408 unique dance sequences spanning 10 genres and durations from 7.4 to 48.0 seconds for music-conditioned training.The motions are captured using multiview recording and supplied as 3D motion targets.
  • Music-to-choreography: MoFusion achieves state-of-the-art beat alignment and Diversity performance, while measuring multi-modality across K=50 generated sequences for each music input.The comparison notes that MoFusion does not explicitly train with beat alignment score as a reward or loss, unlike Bailando.
  • Text-to-motion: On HumanML3D, MoFusion achieves state-of-the-art synthesis variety, with multi-modality of 2.52 versus 2.09.HumanML3D contains approximately 28k text-annotated AMASS motion sequences, averaging 7.1 seconds and receiving 3–4 annotations each.
  • Perceptual evaluation: In the user study, MoFusion was judged more realistic than ground-truth choreography on 51.4% of occasions and achieved 52.3% on semantics.The study addresses weaknesses of Diversity, Multi-modality, and hand-crafted-feature FID as imperfect indicators of meaningful motion quality.
  • Interactive applications: Motion inbetweening fixes selected keyframes and reverse-diffuses the remaining frames, supporting keyframe filling for virtual character animation.Seed-conditioned synthesis forecasts N=160 future frames from a user-provided S=40-frame, 2-second seed sequence.

5. Discussion and Conclusion

MoFusion demonstrates the potential of denoising-diffusion probabilistic models for conditional motion synthesis. Its large latent space helps avoid mean-pose convergence and motion-flicker artifacts associated with quantized codebook-based methods.

  • Discussion: MoFusion highlights the ability of denoising-diffusion probabilistic models to perform conditional motion synthesis.
  • Discussion: MoFusion avoids convergence to a mean pose despite synthesizing motion non-autoregressively.
  • Discussion: Its large latent space avoids motion-flicker artifacts associated with quantized codebook-based methods.

6. Additional Results

MoFusion’s kinematic training losses improve Beat Alignment Score for music-to-dance synthesis, including performance above the 0.237 ground-truth reference. At inference, a pretrained model can incorporate a two-second seed sequence without retraining and produce smooth transitions.

  • Performance with different training objectives: Adding kinematic losses improves Beat Alignment Score, indicating better alignment between generated dance sequences and music beats.The experiments evaluate how different training objectives affect music-to-dance synthesis quality.
  • Performance with different training objectives: 0.237 is the ground-truth Beat Alignment Score that MoFusion’s best configuration exceeds.The reported improvement is attributed to generation quality that matches music beats across the motion sequence.
  • Music-to-Dance Synthesis with Seed Motion Input: MoFusion synthesizes dance motion from scratch using only the music melspectrogram, rather than training with a seed pose.Previous methods use seed poses to guide training and generation, whereas MoFusion is not trained with seed motion input.
  • Music-to-Dance Synthesis with Seed Motion Input: At test time, reverse diffusion uses the first two seconds of ground-truth motion as a seed to predict corresponding dance motion.The seed length follows prior work [29].
  • Music-to-Dance Synthesis with Seed Motion Input: Inference with seed motion increases Beat Alignment Score, with higher performance as additional losses are included.The pretrained dance synthesis model performs this inference without retraining, and the resulting transition from seed to forecast motion is smooth.

7. Implementation Details

MoFusion uses a 1000-step diffusion process with linearly scheduled variances and task-specific training configurations. Motion is represented with SMPL-derived 3D joint positions, while music and text conditions use dedicated latent encoders and transformer-based components.

  • Diffusion Model: The diffusion process uses 1000 steps, with β_t increasing linearly from 0.0001 to 0.02, and trains on a single NVIDIA RTX A40.Task-specific batch sizes are used during training.
  • Music-to-Dance Synthesis: Music-to-dance synthesis uses a 1024-dimensional audio encoder for melspectrograms and 16-head cross-modal transformer blocks with cross-attention dimension 1024.The framework uses AdamW optimization for this configuration.
  • Motion Representation: Motion is represented by 24 SMPL joint positions from AIST++, with 3D joint positions selected because joint-angle representations performed worse.COCO Keypoints-format representations can also be used for training.
  • Text-to-Motion Synthesis: Text-to-motion synthesis uses CLIP ViT-B/32 with a 512-dimensional text latent, batch size 128, AdamW, learning rate 0.0002, and a 500-iteration warm-up.Motion is represented by 22 SMPL-X joint positions extracted from HumanML3D’s SMPL data.

8. Details of User Study

The user study involved 40 participants who compared MoFusion’s synthesis results with other state-of-the-art methods across 18 questions, completing the survey in 8–10 minutes.

  • Study protocol: 40 participants each answered 18 comparison questions about MoFusion and other state-of-the-art methods, taking 8–10 minutes to respond.The study interface is shown in Fig. 7.
Loading 2212.04495v3…