Source-linked AI summary
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics
Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, Chenfanfu Jiang
TL;DR
PhysGaussian addresses novel-dynamics generation without separating simulation geometry from rendering geometry. It integrates continuum mechanics and a custom MPM directly into 3D Gaussian kernels, enabling shared simulation and rendering across diverse materials. The paper reports versatile generative dynamics and real-time performance for scenes with simple dynamics.
Problem
Existing novel-dynamics methods often focus on quasi-static editing and require meshing or embedded proxy geometry, while traditional pipelines separate simulation from rendering.
Method
PhysGaussian evolves 3D Gaussian kernels with physical kinematic and mechanical attributes using continuum mechanics and a custom Material Point Method, then directly renders the deformed Gaussians.
Results
PhysGaussian synthesizes generative dynamics across elastic objects, metals, non-Newtonian viscoplastic substances, and granular materials, achieving real-time performance for scenes with simple dynamics.
Takeaways & Limitations
Using the same 3D Gaussians for simulation and rendering removes geometry-embedding mechanisms and supports direct simulation of scenes reconstructed from real data.
Takeaways & Limitations
Shadow evolution is not considered, and material parameters are manually set; future work proposes automatic parameter assignment and broader material support.
Abstract
from arXiv · showhide
We introduce PhysGaussian, a new method that seamlessly integrates physically grounded Newtonian dynamics within 3D Gaussians to achieve high-quality novel motion synthesis. Employing a custom Material Point Method (MPM), our approach enriches 3D Gaussian kernels with physically meaningful kinematic deformation and mechanical stress attributes, all evolved in line with continuum mechanics principles. A defining characteristic of our method is the seamless integration between physical simulation and visual rendering: both components utilize the same 3D Gaussian kernels as their discrete representations. This negates the necessity for triangle/tetrahedron meshing, marching cubes, "cage meshes," or any other geometry embedding, highlighting the principle of "what you see is what you simulate (WS$^2$)." Our method demonstrates exceptional versatility across a wide variety of materials--including elastic entities, metals, non-Newtonian fluids, and granular materials--showcasing its strong capabilities in creating diverse visual content with novel viewpoints and movements. Our project page is at: https://xpandora.github.io/PhysGaussian/
1 UCLA, 2 Zhejiang University, 3 University of Utah
PhysGaussian is presented as a unified simulation-rendering pipeline based on 3D Gaussians and continuum mechanics. The passage also marks equal contributions.
- PhysGaussian uses 3D Gaussians and continuum mechanics in a unified simulation-rendering pipeline.
- An asterisk indicates equal contributions.
1. Introduction
PhysGaussian addresses the gap between novel-dynamics generation and existing pipelines that rely on quasi-static editing, meshing, or embedded proxy geometry. It unifies physics and rendering through physically endowed 3D Gaussians and demonstrates material versatility and real-time performance for simple dynamics.
- Existing novel-dynamics methods often target quasi-static shape editing and require meshing or coarse proxy geometry.
- Traditional physics-based visual generation uses separate geometry construction, simulation preparation, simulation, and rendering stages.
- PhysGaussian applies the WS2 principle by integrating simulation, capturing, and rendering around a shared representation.
- PhysGaussian equips 3D Gaussian kernels with kinematic and mechanical attributes evolved through continuum mechanics and a custom MPM.
- The method synthesizes dynamics across elastic objects, metals, non-Newtonian viscoplastic substances, and granular materials, with real-time performance for simple dynamics.
2. Related Work
Related work spans radiance-field rendering, dynamic neural fields, and Material Point Method simulation. PhysGaussian combines 3D Gaussian splatting with continuum mechanics and MPM to support physics-based dynamics and photo-realistic rendering.
- NeRF models scenes with neural radiance fields, while 3D Gaussian Splatting provides an explicit point-based representation for real-time rendering.
- Dynamic NeRF methods introduce temporal dimensions using inverse displacement fields and canonical time-invariant fields.
- MPM supports topology changes and frictional interactions across elastic objects, fluids, sand, and snow.
- PhysGaussian uses a unified 3D Gaussian splatting and continuum-mechanics pipeline to generate physics-based dynamics and photo-realistic renderings simultaneously.
- MPM imports physical dynamics into scenarios through a shared particle representation alongside Gaussian Splatting.
3. Method Overview
PhysGaussian uses 3D Gaussians as both simulated continuum particles and rendering primitives, evolving their physical state with continuum mechanics and MPM. Deformation, internal filling, and spherical-harmonic rotation support physically grounded dynamic rendering without intermediate geometry.
- Unified simulation and rendering: PhysGaussian reconstructs a static 3D Gaussian scene, treats its kernels as simulation discretization, and directly splats their deformed states for rendering.The pipeline can optionally fill internal object regions for improved physics compliance.
- Continuum mechanics and MPM: MPM combines Lagrangian particles and Eulerian grids to evolve positions, velocities, deformation, stresses, and plasticity without mesh construction.Particle quantities are transferred through grid nodes, while elastic deformation is regularized using return mappings for plasticity.
- Physics-integrated Gaussian kinematics: Gaussian centers and covariance matrices follow the deformation map, with local affine transformations preserving Gaussian form in world space.The covariance transforms as a_p(t) = F_p(t) A_p F_p(t)^T under the local deformation approximation.
- Dynamic Gaussian state: The method assumes Gaussian opacity and spherical-harmonic coefficients remain invariant over time while harmonics rotate, and infers particle mass from volume and user-specified density.Particle volume is initialized from background cell volume divided by contained particles, followed by m_p = ρ_p V_p^0.
- Evolving orientations of spherical harmonics: Spherical-harmonic orientations rotate with the local rotational component R from the polar decomposition F_p = R_p S_p.This prevents appearance changes when an object rotates while the relative view direction remains fixed.
- Physics-integrated Gaussian kinematics: A rate-form kinematics update offers an alternative Gaussian covariance evolution that avoids dependence on the total deformation gradient F.The update is derived by discretizing the covariance rate using the velocity gradient.
4. Experiments
Experiments evaluate PhysGaussian on diverse materials, benchmark deformations, and additional studies of rendering, physical control, volume conservation, and anisotropy. The method supports multiple dynamics, achieves real-time rates in some cases, outperforms comparison methods across benchmark cases, and benefits from its design enhancements.
- Evaluation of Generative Dynamics: The experiments cover elastic entities, plastic metals, fracture, granular materials, viscoplastic materials, and collisions.Examples include fox, plane, toast, ruins, jam, and sofa suite.
- Evaluation of Generative Dynamics: 30 FPS for plane, 25 FPS for toast, and 36 FPS for jam demonstrate real-time performance under the 1/24-s frame duration.The reported rates are achieved with explicit MPM optimized for GPUs.
- Lattice Deformation Benchmarks: The lattice deformation benchmark compares PhysGaussian with NeRF-Editing, Deforming-NeRF, and PAC-NeRF using synthetic multi-view ground truth.The benchmark applies identical lattice deformations as inputs to all methods.
- Lattice Deformation Benchmarks: PhysGaussian outperforms the comparison methods across all Table 1 cases on PSNR, where higher scores are better.The table also evaluates ablations of the proposed design choices.
- Lattice Deformation Benchmarks: Non-extensible Gaussians cause severe deformation artifacts, while rotating spherical harmonics improves consistency and the full enhancements achieve the best performance.The ablations test covariance behavior and spherical-harmonic rotation.
- Additional Qualitative Studies: Internal filling enables more realistic simulation and material-parameter control, with larger Young’s modulus indicating greater stiffness and larger Poisson ratio indicating better volume preservation.The filling method uses a density field reconstructed from Gaussian opacity.
- Additional Qualitative Studies: Compared with geometry-based NeRF-Editing, the physics-based method captures and maintains volumetric behavior during deformation.The comparison uses NeRF-Editing with surface ARAP deformation.
- Additional Qualitative Studies: An anisotropy regularizer mitigates burr-like artifacts caused by excessively slender Gaussian kernels during large deformations.The regularizer constrains Gaussian anisotropy through an additional loss.
5. Discussion
PhysGaussian unifies physics simulation and photo-realistic rendering in one pipeline, while leaving shadow evolution and material-parameter assignment as current limitations.
- PhysGaussian simultaneously generates physics-based dynamics and photo-realistic renderings through a unified simulation-rendering pipeline.
- The framework does not model the evolution of shadows and requires manual material-parameter settings.
- Future work includes geometry-aware 3DGS reconstruction, more versatile materials such as liquids, and more intuitive user controls.
A. MPM Algorithm
The MPM algorithm alternates particle-to-grid transfer, grid updates, and grid-to-particle transfer while evolving particle and grid states over fixed time steps.
- MPM discretizes a continuum into Lagrangian particles and advances time using fixed steps t_n = n∆t.
- Each time step transfers particle masses and momenta to grid nodes, updates grid velocities, then transfers velocities back for particle advection.
- Particles track mass, position, velocity, deformation gradient, Kirchhoff stress, and affine momentum, while grid nodes track mass, position, and velocity.
- The method uses APIC for momentum transfer and forward Euler for grid-velocity updates based on stress forces and gravity.
- The return map and Kirchhoff stress computation complete the constitutive update, with model settings and material parameters summarized in Tables 2 and 3.
B. Elasticity and Plasticity Models
PhysGaussian adopts constitutive models for elastic and plastic materials, representing plastic deformation multiplicatively and computing stress from the elastic component.
- The framework adopts constitutive models from prior work and selects models for each scene using specified material parameters.
- Plasticity models multiplicatively decompose the deformation gradient as F = F^E F^P.
- A hyperelastic constitutive model computes Kirchhoff stress from F^E after yield-stress processing.
- For a purely elastic continuum, the elastic deformation gradient equals the total deformation gradient, F^E = F.
B.1. Fixed Corotated Elasticity
The constitutive models compute Kirchhoff stress from elastic deformation, while return mappings constrain plastic evolution for materials such as sand and other yield-stress media.
- Fixed Corotated Elasticity: Fixed corotated elasticity computes Kirchhoff stress from the elastic deformation gradient, its rotational factor, and volumetric change.
- Elasticity Models: Log-strain elasticity uses the singular value decomposition F^E = UΣV^T and logarithmic strain ϵ = log(Σ).
- Neo-Hookean Elasticity: Neo-Hookean elasticity computes stress from F^E F^{E T}, the identity matrix, and log(J), where J is the determinant of F^E.
- Drucker-Prager Plasticity: Drucker-Prager plasticity applies a return map to the singular values of F, using friction-angle-dependent conditions on volumetric and deviatoric strain.
- Plasticity Models: The plastic update preserves the trial state when the yield condition is not exceeded and otherwise projects the strain according to the plastic increment.
C. Additional Evaluations
The method receives additional evaluation on elastic, fracture, viscoplastic, metallic, and granular examples, combining real and synthetic data sources.
- The vasedeck example comes from the NeRF dataset, while the other examples are synthetic data generated using BlenderNeRF.
- Figure 9 evaluates vasedeck, bread, cake, can, and wolf examples spanning elastic, fracture, viscoplastic, metal, and granular materials.The examples are listed from top to bottom in the figure.
- The evaluations illustrate the method across diverse material behaviors, including elastic entities, fracture, viscoplastic materials, metals, and granular materials.