Source-linked AI summary
PhysDiff: Physics-Guided Human Motion Diffusion Model
Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, Jan Kautz
TL;DR
Existing motion diffusion models lack explicit physical constraints and can produce artifacts that limit real-world applications. PhysDiff inserts physics-based motion projection into the denoising process using motion imitation in a physics simulator. The paper reports state-of-the-art motion quality and substantially improved physical plausibility on large-scale motion data.
Problem
Existing motion diffusion models overlook physical laws, producing artifacts such as floating, foot sliding, and ground penetration that hinder real-world applications.
Method
PhysDiff uses a physics-based motion projection module to simulate denoised motions and feed physically plausible projections into subsequent diffusion steps.
Results
PhysDiff achieves state-of-the-art motion quality and substantially improves physical plausibility on large-scale motion data.
Takeaways & Limitations
Embedding physics during diffusion, rather than applying it only as post-processing, keeps motions close to the data distribution while moving them toward physical plausibility.
Takeaways & Limitations
Physics simulation makes PhysDiff two-to-three times slower than state-of-the-art models during inference.
Abstract
from arXiv · showhide
Denoising diffusion models hold great promise for generating diverse and realistic human motions. However, existing motion diffusion models largely disregard the laws of physics in the diffusion process and often generate physically-implausible motions with pronounced artifacts such as floating, foot sliding, and ground penetration. This seriously impacts the quality of generated motions and limits their real-world application. To address this issue, we present a novel physics-guided motion diffusion model (PhysDiff), which incorporates physical constraints into the diffusion process. Specifically, we propose a physics-based motion projection module that uses motion imitation in a physics simulator to project the denoised motion of a diffusion step to a physically-plausible motion. The projected motion is further used in the next diffusion step to guide the denoising diffusion process. Intuitively, the use of physics in our model iteratively pulls the motion toward a physically-plausible space, which cannot be achieved by simple post-processing. Experiments on large-scale human motion datasets show that our approach achieves state-of-the-art motion quality and improves physical plausibility drastically (>78% for all datasets).
1. Introduction
Existing motion diffusion models generate high-quality motions but do not explicitly enforce physics, leading to artifacts that limit real-world use. PhysDiff integrates physics-based projection iteratively into diffusion and reports improved motion quality and physical plausibility.
- Existing motion diffusion models overlook physical laws and lack explicit mechanisms for enforcing physical constraints or modeling force- and contact-induced dynamics.
- Floating, foot sliding, and ground penetration are common artifacts that hinder applications such as animation and virtual reality.
- PhysDiff projects each denoised diffusion step into a physically plausible space, then feeds the projected motion into the next diffusion step.
- Iterative physics and diffusion are preferred over final-step post-processing because severely implausible motions may be unnatural after correction or pushed away from the data distribution.
- Motion imitation in a physics simulator enforces constraints and removes artifacts including floating, foot sliding, and ground penetration.
- On HumanML3D text-to-motion generation, PhysDiff reduces physical errors by more than 86% and improves motion quality by more than 20%.
2. Related Work
Diffusion models have become prominent for human motion generation because they combine strong distribution modeling with high-quality, multimodal synthesis. Related physics-based approaches use simulation, reinforcement learning, trajectory optimization, and motion imitation for physically grounded motion modeling.
- Denoising diffusion models are trained with denoising autoencoder objectives interpretable as score matching and generate samples through iterative denoising.
- Human motion generation has progressed from deterministic models to generative models that address the mode averaging problem and support varied conditioning signals.
- Recent motion diffusion models demonstrate state-of-the-art motion generation performance but often produce physically implausible motions.
- Physics-based human motion modeling has used deep reinforcement learning for locomotion and character animation, plus trajectory optimization and motion imitation for 3D pose estimation.
3. Method
PhysDiff integrates physics into iterative motion diffusion by projecting intermediate denoised motions into a physically plausible space before subsequent denoising steps. The method uses a simulator-based motion imitation policy and schedules projections to balance physical plausibility, sample quality, and efficiency.
- Physics-Guided Motion Diffusion: PhysDiff generates conditional human motions by iteratively denoising a noisy motion from timestep t toward the final output.The model can generate motions conditioned on information such as text or action labels.
- Physics-Based Motion Projection: The physics-based projection Pπ uses a motion imitation policy in a physics simulator to map denoised motions to physically plausible motions.The simulated motion is used to guide the diffusion process rather than only correcting the final output.
- Physics-Guided Motion Diffusion: Each projected motion is incorporated into the next diffusion update, while non-projection steps retain the denoised motion before the DDIM-style sampling update.The projection is applied conditionally at selected diffusion timesteps and the remaining update samples from a Gaussian distribution.
- Physics-Based Motion Projection: PhysDiff is agnostic to the denoiser architecture and applies physics-based projection only during inference, enabling compatibility with pretrained motion diffusion models without retraining.This design separates the physics projection from denoiser training.
- Scheduling Physics-based Projection: Physics projection is scheduled away from high-noise timesteps because correcting mode-averaged motions can push samples away from the data distribution and hinder diffusion.The authors report that their scheduling study identifies a strategy balancing sample quality and efficiency.
4. Experiments
PhysDiff is evaluated on text-to-motion and action-to-motion generation using standard motion-quality metrics alongside physical-plausibility metrics. Across datasets, it improves physical plausibility substantially, with projection scheduling and diffusion–physics integration affecting the quality–plausibility trade-off and inference cost.
- Experimental Setup: Experiments cover text-to-motion and action-to-motion generation, including HumanML3D, HumanAct12, and UESTC.The evaluation compares PhysDiff with state-of-the-art motion diffusion and action-to-motion methods.
- Experimental Setup: Evaluation combines FID and text or action alignment with Penetrate, Float, Skate, and aggregate Phys-Err metrics.R-Precision is used for text-to-motion, while Accuracy replaces it for action-to-motion; the physical metrics are measured in millimeters.
- Text-to-Motion Generation: More than 86% lower Phys-Err and more than 20% higher motion quality are reported for HumanML3D text-to-motion generation.PhysDiff with MDM achieves state-of-the-art FID, while the MotionDiffuse variant maintains high R-Precision and improves FID significantly.
- Action-to-Motion Generation: 78% lower Phys-Err on HumanAct12 and 94% lower Phys-Err on UESTC are reported while maintaining competitive FID.Qualitative comparisons also report substantially fewer artifacts such as penetration and floating.
- Schedule of Physics-Based Projection: Increasing projection steps consistently lowers Phys-Err, but FID and R-Precision eventually deteriorate, revealing a physical-plausibility–motion-quality trade-off.The authors hypothesize that projections at early diffusion steps can push motion away from the data distribution.
- Schedule and Post-Processing: Four consecutive projections toward the end of diffusion are selected as a good schedule, while physics simulation makes single-motion inference 2.5x slower than MDM.The reported single-motion times are 51.6s for PhysDiff versus 19.6s for MDM; larger batches reduce the slowdown to 1.7x.
5. Conclusion and Future Work
PhysDiff integrates physics into diffusion to generate physically plausible human motions, achieving state-of-the-art motion quality and substantially improving physical plausibility. Its main limitation is slower inference caused by physics simulation.
- PhysDiff instills the laws of physics into diffusion and uses motion imitation in physics simulation to enforce physical constraints.
- Experiments on large-scale motion data show state-of-the-art motion quality and substantially improved physical plausibility.
- Inference speed can be two-to-three times slower than state-of-the-art models because of physics simulation.
A. Details of Evaluation Metrics
The evaluation uses motion-based and physics-based metrics to assess generated-motion quality and physical plausibility.
- Motion-based metrics include FID, R-Precision, and Accuracy, computed using the open-source MDM code.
- Ground penetration and floating measure distances between the ground and the lowest body mesh vertex, using a 5 mm tolerance for geometry approximation.
B. Details of Motion Diffusion
PhysDiff is evaluated with pretrained state-of-the-art denoisers and uses specified diffusion, guidance, policy-training, and physics-imitation settings.
- PhysDiff is tested with the pretrained MDM and MotionDiffuse denoiser networks, improving both models.
- Diffusion sampling uses 50 timesteps, η = 0, and classifier-free guidance with coefficient 2.5.
- The motion imitation policy is a three-layer MLP trained on AMASS using the HumanML3D training split.
- Physics-based motion imitation uses PPO with clipping coefficient ϵ = 0.2, discount factor γ = 0.99, and GAE coefficient λ = 0.95.
C. Details of Physics-Based Motion Imitation
The physics simulation uses IsaacGym for parallel GPU simulation, with a 60 Hz simulator and a character controlled at 30 Hz.
- IsaacGym provides massively parallel GPU physics simulation for PhysDiff.
- The simulation runs at 60 Hz while the policy controls the SMPL-based character at 30 Hz.