PaperlayerView exact revision on arXiv

Source-linked AI summary

Guided Motion Diffusion for Controllable Human Motion Synthesis

Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, Siyu Tang

arXiv:2305.12577v3cs.CV

TL;DR

Diffusion models generate high-quality text-conditioned human motion, but incorporating trajectories, obstacles, and other spatial constraints remains difficult. GMD addresses this gap with emphasis projection, imputation, and dense guidance, improving text-to-motion results while enabling spatial control. The paper also uses classifier-based guidance for flexible conditioning without retraining, while noting challenges in obtaining data and designing classifier-free learning for non-explicit goals.

  • Problem

    Spatial constraints such as trajectories and obstacles remain difficult to integrate into diffusion-based human motion generation.

  • Method

    GMD combines emphasis projection, a new imputation formulation, and dense signal propagation within a U-Net-based architecture for spatially controllable motion generation.

  • Results

    GMD significantly outperforms state-of-the-art methods on traditional text-to-motion tasks while supporting trajectory conditioning, keyframe conditioning, and obstacle avoidance.

  • Takeaways & Limitations

    GMD enables human motion synthesis conditioned jointly on free text and spatial constraints.

  • Takeaways & Limitations

    Non-explicit goals such as obstacle avoidance can require difficult data collection and classifier-free learning design, so GMD uses classifier-based guidance without retraining.

Abstract

from arXiv · show

Denoising diffusion models have shown great promise in human motion synthesis conditioned on natural language descriptions. However, integrating spatial constraints, such as pre-defined motion trajectories and obstacles, remains a challenge despite being essential for bridging the gap between isolated human motion and its surrounding environment. To address this issue, we propose Guided Motion Diffusion (GMD), a method that incorporates spatial constraints into the motion generation process. Specifically, we propose an effective feature projection scheme that manipulates motion representation to enhance the coherency between spatial information and local poses. Together with a new imputation formulation, the generated motion can reliably conform to spatial constraints such as global motion trajectories. Furthermore, given sparse spatial constraints (e.g. sparse keyframes), we introduce a new dense guidance approach to turn a sparse signal, which is susceptible to being ignored during the reverse steps, into denser signals to guide the generated motion to the given constraints. Our extensive experiments justify the development of GMD, which achieves a significant improvement over state-of-the-art methods in text-based motion generation while allowing control of the synthesized motions with spatial constraints.

1. Introduction

GMD addresses spatially conditioned human motion generation, where diffusion models struggle to integrate spatial constraints with local poses and sparse guidance. It combines emphasis projection, imputation, and dense signal propagation to improve controllability.

  • Spatial constraints remain underexplored in diffusion-based human motion generation despite governing physical interaction with surroundings.
  • GMD uses emphasis projection to improve trajectory-motion coherence and dense signal propagation to make sparse guidance more controllable.
  • Motion representations can overemphasize local poses relative to global orientations, causing guided spatial information to appear as noise and produce foot skating.A cited representation uses 4 values for global orientation and 259 for local pose per frame.
  • Sparse keyframe guidance may be ignored during denoising or cause warping artifacts when the motion satisfies constraints only at provided keyframes.
  • The method integrates these contributions into a U-Net-based architecture for spatially conditioned motion generation and also improves traditional text-to-motion performance.The paper evaluates trajectory conditioning, keyframe conditioning, and obstacle avoidance.

2. Related Work

Related work covers diffusion models, their conditioning mechanisms, and human motion generation across text, trajectories, music, objects, and other signals.

  • Diffusion probabilistic models have been applied to image, speech, video, 3D shape, and reinforcement-learning generation.
  • Diffusion models support controllable generation through text conditioning, image editing, inpainting, classifier guidance, and classifier-free guidance.
  • Human motion generation has explored partial poses, trajectories, images, music, text, objects, action labels, and unconditioned synthesis.
  • Recent diffusion-based motion models demonstrate better quality than GAN- and VAE-based alternatives and use CLIP to improve motion generation.

3. Background

The background introduces diffusion probabilistic models as sequential denoising processes and reviews classifier guidance, classifier-free guidance, and imputation or inpainting for conditional generation.

  • Diffusion-based generative models: Diffusion probabilistic models learn sequential denoising across noise levels, mapping a Gaussian prior to a data distribution through successive reverse steps.The noising process is defined with α_t and a scheduler β_t.
  • Diffusion-based generative models: Denoising models may predict the clean input, noise, or one-step denoised target, each trained with a corresponding squared loss.
  • Controllable generation with diffusion models: Classifier-free guidance combines conditional and unconditional denoiser predictions, with w controlling conditional strength.Its conditioning signals must be known beforehand, and adapting the denoiser for each case restricts flexibility.
  • Controllable generation with diffusion models: Classifier guidance shifts the sampling mean using a scaled score function and can flexibly add conditioning without changing the diffusion model.The guidance signal diminishes as t approaches 0 because the variance schedule decreases.
  • Imputation and inpainting: Imputation and inpainting adjust diffusion samples toward partial observations such as global trajectories or keyframe locations.Imputation can be applied after every denoising step using a resized observation mask and a noised target.

4. Guided Motion Diffusion

GMD incorporates spatial constraints into motion diffusion through emphasis projection, imputation, dense signal propagation, and guided sampling. The framework targets incoherence from sparse trajectory features and the tendency of sparse temporal guidance to be ignored.

  • Guided Motion Diffusion: GMD models spatially guided motion with trajectory and motion diffusion stages, supporting text and spatial conditioning.The pipeline first generates a trajectory and then generates motion conditioned on that trajectory.
  • Emphasis projection: The projected representation addresses motion incoherence caused when the trajectory occupies only a small part of the full motion representation.Without this adjustment, the diffusion model may ignore imputed trajectory changes and produce local motion misaligned with the trajectory.
  • Emphasis projection: Emphasis projection increases the relative importance of trajectory-related motion features to improve coherence between global orientation and local pose.The method emphasizes pelvis rotation and ground location while preserving unit variance in the projected representation.
  • Guided sampling: Classifier guidance adjusts diffusion samples toward lower goal-function values, while epsilon modeling is used to manage model bias during guided sampling.The paper states that epsilon modeling is effective for managing this bias and that emphasis projection outperforms retraining with increased trajectory loss weight.
  • Dense guidance: Dense signal propagation expands sparse temporal guidance through a learned denoising function that relates individual frames to nearby motion frames.The existing diffusion model denoiser can perform this expansion through autodiff without an additional model.

5. Applications

GMD applies spatially guided motion generation to trajectory matching, sparse keyframe locations, and obstacle avoidance. Its two-stage pipeline generates or refines a trajectory before producing trajectory-conditioned motion, while obstacle avoidance relies on classifier guidance.

  • Trajectory-conditioned generation: Trajectory-conditioned generation aims to produce realistic motion that matches a given trajectory.The objective is defined by minimizing the distance between generated motion and the target trajectory.
  • Location-conditioned trajectory planning: Sparse keyframe conditioning generalizes trajectory-conditioned generation by specifying locations at only selected motion steps.A trajectory and mask describe the keyframe locations and the constrained portions of the motion.
  • Location-conditioned trajectory planning: GMD uses two stages for keyframe-conditioned generation: first generate a trajectory satisfying keyframe locations, then generate motion given that trajectory.The trajectory can be constructed point-to-point, generated by a trajectory diffusion model, or combined using a timing parameter τ.
  • Obstacle avoidance: Obstacle avoidance combines a goal for reaching the destination with a boundary-repulsion goal based on a specified safe distance.Because obstacle constraints cannot be imputed, minimizing the obstacle-avoidance goal relies solely on classifier guidance.

6. Experiments

Experiments evaluate GMD on text-to-motion, trajectory-conditioned, keyframe-conditioned, and obstacle-avoidance tasks using standard quality, coherence, and spatial-adherence measures. GMD improves text-motion quality, emphasis projection improves trajectory-motion coherence, and dense propagation supports sparse keyframe conditioning.

  • 6.1. Settings: Experiments use HumanML3D and evaluate text-to-motion with FID, R-Precision, Diversity, and Foot skating ratio.Keyframe-conditioned generation additionally uses Trajectory diversity, Trajectory error, Location error, and Average error.
  • 6.2. Text-to-motion generation: GMD significantly outperforms MotionDiffuse, MDM, MLD, and PhysDiff in FID while maintaining similar R-Precision and Diversity.
  • 6.3. Trajectory-conditioned generation: Emphasis projection reduces Foot skating ratio and produces more coherent trajectory-conditioned motion than MDM.The comparison uses ground-truth trajectories for imputation after each denoising step; Figure 5 shows the conditioning comparison for “walking forward”.
  • 6.3. Trajectory-conditioned generation: Increasing trajectory loss strength marginally improves FID and Foot skating ratio, but excessive strength degrades both metrics; emphasis projection improves both consistently.
  • 6.4. Keyframe-conditioned generation: Keyframe-conditioned generation requires both FID and Foot skating ratio because FID alone does not adequately measure trajectory-motion coherence.Without dense signal propagation, the model ignores keyframe conditioning.
  • 6.4. Keyframe-conditioned generation: The trajectory model reduces Location error by more than half versus the single-stage model at τ = 100, while increasing τ raises Trajectory diversity and Location error.Increasing τ also lowers FID; at τ = 0, forced trajectories produce relatively high Foot skating.
  • 6.5. Obstacle avoidance motion generation: GMD is also evaluated for obstacle avoidance by conditioning on a target point and obstacles represented as a 2D SDF.

7. Discussion and Limitations

GMD supports diverse goal functions and produces high-quality, diverse motions, while obstacle avoidance is demonstrated qualitatively and classifier guidance offers flexible conditioning without retraining. The method’s reliance on classifier-based guidance reflects challenges in obtaining data and designing classifier-free learning for non-explicit goals.

  • GMD produces high-quality, diverse motions and supports diverse goal functions.
  • Qualitative obstacle-avoidance results use keyframe locations and obstacles, with crossed regions indicating areas to avoid.
  • Classifier-based guidance enables more flexible conditioning without retraining the model.
  • Obtaining necessary data and designing classifier-free learning for non-explicit goals such as obstacle avoidance can be challenging.
  • The authors suggest effective classifier guidance can support the inclusion of more guiding signals.

A. Analysis on x0,θ vs. ϵθ DPMs

The analysis contrasts x0- and ϵ-predicting diffusion models, showing that their influence over denoising steps differs and motivating the use of ϵ prediction for classifier-guided trajectory modeling.

  • x0 prediction has greater influence near the end of denoising, whereas ϵ prediction has greater influence when t is large.
  • The coefficients a and b determine how much x0 and the previous output xt contribute to the denoising mean.
  • An ϵ model makes smaller changes over time, while an x0 model can make large changes even at the end of diffusion.
  • Because classifier guidance weakens as t approaches zero, an x0 model can overpower late guidance and undo prior corrections.
  • The trajectory model therefore uses an ϵ model, with Figure A.1 visualizing the models’ relative contributions over time.

A.1. Challenges of modeling ϵ in practice

Modeling ϵ imposes practical architectural constraints for motion diffusion, while absolute root representations simplify the imputation and optimization required for spatial guidance.

  • A.1. Challenges of modeling ϵ in practice: ϵ prediction is practically restricted when its shape is not relatively small compared with the denoising model’s latent dimension.
  • A.1. Challenges of modeling ϵ in practice: The original MDM architecture produced low-quality jagged motions for ϵ prediction, whereas x0 prediction with the same architecture produced high-quality motions.
  • A.1. Challenges of modeling ϵ in practice: Increasing MDM’s latent dimension from 512 to 1,536 did not fully solve the ϵ-prediction problem, indicating that different architectural designs may be required.
  • A.1. Challenges of modeling ϵ in practice: Absolute root coordinates make trajectory imputation and optimization straightforward because each location can be updated without depending on other motion steps.
  • A.1. Challenges of modeling ϵ in practice: Absolute root representation is necessary for the final model because spatial guidance combines imputation with optimization.

C. Analysis on Emphasis projection

Emphasis projection increases the relative importance of trajectory features while preserving a usable motion representation, and it outperforms trajectory loss scaling in motion coherence.

  • C. Analysis on Emphasis projection: Emphasis projection increases the relative importance of trajectory features within the motion representation.
  • C. Analysis on Emphasis projection: A random matrix projection redistributes scaled trajectory values to maintain uniform variance while preserving distances with high probability and enabling exact inversion.
  • C.1. Trajectory loss scaling: Loss scaling multiplies trajectory terms inside the squared loss, so k = 10 gives the trajectory 100-fold greater importance.
  • C.1. Trajectory loss scaling: Across k ∈ 1, 2, 5, 10, Emphasis projection consistently outperformed loss scaling in motion coherence.

D. GMD’s Model Architecture

GMD uses 1D UNET architectures with Adaptive Group Normalization, adapted for sequential motion and trajectory prediction. This architecture substantially improves motion-generation FID over the original MDM.

  • Architecture: GMD’s trajectory and motion architectures use 1D UNETs with Adaptive Group Normalization for sequential prediction.The architecture is adapted from AdaGN-based image-generation models by replacing image-oriented processing with 1D convolutions.
  • Architecture: GMD’s convolution-based architecture differs from the transformer-based architecture used in the original MDM.The paper notes that transformer-based architectures are less studied for diffusion probabilistic models than convolution-based architectures.
  • Results: 0.212 FID versus 0.556 for the original MDM, reducing FID by more than half through the proposed architecture alone.This comparison is reported in Table 1 of the main paper.
  • Architecture: Table D.1 summarizes the network architecture of GMD’s models based on the proposed 1D UNET with AdaGN.The table provides the architecture specification for the GMD models.

E. Training details

GMD training uses separate batch sizes for motion and trajectory models, mixed precision, moving-average weights, and AdamW optimization. Additional settings include gradient clipping, trajectory-model regularization choices, and guidance termination at t = 20.

  • Optimization: GMD trains motion models with batch size 64 and trajectory models with batch size 512.These settings are specified separately for the two model types.
  • Optimization: Training uses AdamW with learning rate 0.0001, weight decay 0.01, and gradient-norm clipping at 1.The clipping value was selected because it increased training stability.
  • Conditioning: The model receives sinusoidally encoded time steps and CLIP text embeddings as conditioning signals.These signals are processed by the 1D UNET with AdaGN architecture.
  • Conditioning: Each ResBlock projects the shared MLP conditioning signal through Mish activation followed by a ResBlock-specific resizing linear projection.All convolutional kernel sizes in the described ResBlock are 5.
  • Training schedule: Motion models are trained for 32,000,000 samples, with mixed precision and an exponential moving average using β = 0.9999.The averaged model is used for generation quality, although the architecture also improves over MDM without the moving average.
  • Trajectory model: The trajectory model trains on rotation and x, y, z components, although ground x-z locations are the crucial trajectory elements.The additional rotation and y components are reported to help learning and reduce overfitting.
  • Guidance: Classifier guidance uses strength s = 100 and p = 1 goal-function norms, while guidance signals stop at t = 20.The stopping point slightly improved motion coherence, and imputation and classifier guidance were used concurrently when feasible.
  • Guidance: Obstacle avoidance omits point-to-point trajectory imputation because a straight trajectory could collide with an obstacle.The task therefore does not use point-to-point trajectory imputation.
Loading 2305.12577v3…