PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation

Yuchen Lin, Chenguo Lin, Jianjin Xu, Yadong Mu

arXiv:2501.18982v1cs.CV

TL;DR

Existing physics-based 3D synthesis methods often restrict materials to predefined categories, limiting heterogeneous-scene modeling. OmniPhysGS uses learnable Constitutive Gaussians, expert material models, and video-diffusion supervision to synthesize broader physics-based dynamics. It reports more general and realistic dynamics across materials and interactions, surpassing existing methods by about 3%–16% on visual-quality and text-alignment metrics.

  • Problem

    Existing methods assume predefined material categories, limiting physics-based synthesis for heterogeneous objects and broader material behaviors.

  • Method

    OmniPhysGS models each Gaussian particle with a learnable constitutive model combining 12 expert-designed material-model combinations, optimized through MPM simulation and text-to-video diffusion supervision.

  • Results

    OmniPhysGS generates physically plausible motions and interactions across varied materials, surpassing current methods by about 3%–16% in text-alignment metrics for different material cases.

  • Takeaways & Limitations

    The framework automatically models different materials per Gaussian and supports dynamics involving heterogeneous materials without manual tuning of physical properties.

  • Takeaways & Limitations

    The method uses expert-designed constitutive models limited to several representative materials, and per-scene SDS optimization is time-intensive.

Abstract

from arXiv · show

Recently, significant advancements have been made in the reconstruction and generation of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically assume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heterogeneous objects in real scenarios and tend to render less physically plausible animation given a wider range of objects. We propose OmniPhysGS for synthesizing a physics-based 3D dynamic scene composed of more general objects. A key design of OmniPhysGS is treating each 3D asset as a collection of constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensemble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementation, we define a scene by user-specified prompts and supervise the estimation of material weighting factors via a pretrained video diffusion model. Comprehensive experiments demonstrate that OmniPhysGS achieves more general and realistic physical dynamics across a broader spectrum of materials, including elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.

1 INTRODUCTION

The paper targets physics-based 3D dynamics across heterogeneous materials, addressing limitations of narrowly constrained physical models. OmniPhysGS introduces learnable Constitutive Gaussians and combines physical simulation with video-diffusion supervision for automatic, physically plausible synthesis.

  • Existing data-driven dynamic-scene methods can produce artifacts and inconsistencies because they are not strictly constrained by physical laws.
  • General physics-based dynamics covers elastic, viscoelastic, plastic, and fluid materials, including interactions between heterogeneous materials in one scene.
  • Constitutive Gaussians assign learnable material models to Gaussian particles, using physical-aware predictions and expert constitutive models to represent varied materials.
  • OmniPhysGS combines MPM physical constraints, Constitutive Gaussians, and material knowledge from a video diffusion model for automatic heterogeneous-material dynamics synthesis.
  • Extensive experiments report physically plausible motions and interactions across varied materials without manual tuning of physical properties.

2 RELATED WORK

Prior work on 4D generation distills dynamic objects using text-to-image and video diffusion models, while interactive dynamics generation explores user preferences and constraints across 2D and 3D content.

  • 4D Generation: 4D generation commonly uses score distillation from text-to-image and video diffusion models.These methods employ dynamic 3D representations, including grids, NeRFs, 3D Gaussians with deformation fields, and warped Gaussian surfels.
  • 4D Generation: Dynamic 4D objects have been represented with HexPlane, multi-scale 4D grids, Kplane, hash encoding, canonical NeRFs, and deformable 3D Gaussians.The surveyed representations span grid-based, neural-field, and Gaussian-based designs.
  • Interactive Dynamics Generation: Interactive dynamics generation addresses both 2D and 3D content under user preferences or constraints.For image animation, prior methods use driving videos, keypoint trajectories, or text prompts as initial conditions.

3 METHOD

OMNIPHYSGS synthesizes prompt-aligned, physically plausible 3D dynamics by coupling Gaussian scene representations with differentiable MPM and learnable constitutive material models. Its physics-guided network combines expert constitutive models with diffusion-based video guidance to represent diverse material behaviors.

  • Problem Statement: 3D Gaussian kernels represent the scene, allowing physical properties to be assigned to individual Gaussian particles.Each kernel stores position, covariance, spherical harmonic coefficients, and opacity, and can be rendered from arbitrary viewpoints.
  • OMNIPHYSGS: The method simulates Gaussian particles with MPM and optimizes rendered video using a pretrained text-to-video diffusion model with Score Distillation Sampling.The simulation produces a video clip that is used for diffusion-based optimization.
  • Constitutive Gaussian: Constitutive Gaussians assign each Gaussian learnable constitutive models whose parameters can vary across objects with different materials.The learnable parameters represent hyperelastic energy, plasticity return functions, and physical material parameters.
  • Physics-Guided Constitutive Network: The physical-aware decoder selects among expert-designed constitutive models, using 12 combinations to cover elastic, viscoelastic, plastic, and fluid material behaviors.A 3D feature encoder processes local Gaussian neighborhoods, while hardmax assignment assumes homogeneous properties within each neighborhood.
  • Constitutive Gaussian: Unlike methods with fixed constitutive models, Constitutive Gaussians capture material behaviors spanning elastic and plastic deformation with a more flexible representation.The framework combines physical simulation with learned material properties and prompt-driven synthesis.
  • Physics-Guided Constitutive Network: Expert constitutive priors guide the network because neural networks trained without physical priors can struggle to fit material behavior.The decoder integrates these models to avoid convergence issues faced by vanilla neural networks.

4 EXPERIMENTS

Experiments evaluate OmniPhysGS on material variation, multi-object scenes, motion generalization, and ablations of its architecture and training strategy. The method models diverse material behaviors and heterogeneous interactions, while experiments identify the importance of physical priors and staged training.

  • Single Object in Different Materials: OmniPhysGS surpasses current methods by about 3%–16% in text-alignment metrics across single-object material cases.Baselines perform similarly on pure elasticity but struggle with plasticity, viscoelasticity, and fluid behaviors.
  • Multiple Objects in Different Materials: OmniPhysGS surpasses baselines by about 4% in text-alignment metrics for multi-object scenes with different materials.Unlike baselines that tend to assign homogeneous properties, it predicts material properties for each object from the prompt.
  • Generalization of Motion: MPM enables zero-shot motion generalization by producing different deformations under changed impulses, colliders, or gravity directions.These variations can be synthesized after training on a single scene.
  • Network Architecture: A simple MLP struggles to fit the highly nonlinear constitutive models, supporting the use of expert-designed physical priors.The ablation replaces the physical-aware decoder with an MLP that directly predicts stress and deformation tensors.
  • Training Strategy: Training without frame grouping causes out-of-memory failure, while removing multi-batch training significantly reduces performance.The reported failures reflect long gradient propagation and optimization difficulty, respectively.

5 CONCLUSION

OmniPhysGS introduces Constitutive Gaussians for general physics-based 3D dynamic scene synthesis, combining domain-expert constitutive models with text-to-video supervision. The framework supports flexible material modeling and text-aligned dynamic scenes, but remains limited by its representative material set and time-intensive per-scene optimization.

  • 5 CONCLUSION: OmniPhysGS uses domain-expert constitutive models to automatically and flexibly model different materials for each Gaussian particle.Pretrained text-to-video supervision provides a user-facing interface for generating physically plausible, visually realistic scenes aligned with prompts.
  • 5 CONCLUSION: The framework is intended for applications including immersive video games, virtual reality, robotic simulation, and computer-aided design.
  • 5 CONCLUSION: The method is limited to several representative expert-designed materials, and per-scene SDS optimization is time-intensive.Future work proposes expanding the material set and developing amortized optimization for instant inference.

A MATERIAL POINT METHOD

The Material Point Method models material deformation through continuum mechanics and simulates Gaussian particles with particle–grid transfers. Constitutive models determine stress, plasticity correction, and material-specific behavior during the simulation.

  • Continuum mechanics: Continuum mechanics represents deformation with a map from undeformed configuration X to deformed position x, using deformation gradient F.The governing equations enforce conservation of mass and momentum.
  • MPM algorithm: MPM discretizes the continuum into particles and uses a background grid to solve conservation equations through P2G and G2P transfers.Boundary conditions can be applied to particles before P2G and grid nodes before G2P.
  • MPM algorithm: Each MPM time step initializes particles, updates stress, transfers information particle-to-grid and grid-to-particle, updates the grid, and applies plasticity correction.The algorithm is summarized as an MLS-MPM step with boundary-condition handling.
  • MPM algorithm: MPM particle states include position, velocity, deformation gradient, and affine momentum, while constitutive models calculate stress and correct deformation.Physical parameters such as Young’s modulus and Poisson’s ratio are included in the constitutive models.
  • Gaussian simulation: Gaussian kernels are updated from particle positions and deformation gradients so their distributions follow deformation during simulation and rendering.Spherical harmonics and opacity remain unchanged, while the view direction is modified by deformation.
  • Constitutive models: The framework combines expert constitutive models for elasticity, plasticity, viscoelasticity, and fluids, including rubber, snow, metal, honey, water, oil, and lava.The models map deformation information to stress and corrected deformation, with physical parameters listed in Table 5.

B.1 HYPERPARAMETERS AND TRAINING SETTINGS

Experiments use fixed simulation, rendering, and optimization settings across scenes, while baselines are reimplemented and trained with adjusted iteration counts for fair comparison.

  • Simulation settings: The simulation uses timestep ∆t = 3×10^-4, 150 frames at 30 fps, a 1 × 1 × 1 domain, 25 × 25 × 25 grids, and gravity of 9.8 m/s2.Training samples simulated states every 10 steps and renders them with a fixed camera.
  • Training settings: Training uses one NVIDIA A6000 GPU, Adam with learning rate 5 × 10^-5, five scene iterations, and ten sequential stages optimized by a pretrained text-to-video diffusion model.Each stage renders 15 frames and optimizes its video clips for 30 iterations.
  • Baseline comparison: Baselines are reimplemented from their repositories, and their internal training iterations are multiplied to match OmniPhysGS’s multi-batch training effort.The same experimental settings are used across methods for comparison.

B.3 EVALUATION DETAILS

Evaluation measures text–video alignment and training-related expressiveness and robustness without requiring ground-truth dynamic 3D scenes.

  • Metrics: CLIPSIM measures alignment between generated videos and text prompts using average cosine similarity between text and frame embeddings.The score is computed with the ViT-L/14 version of CLIP, and higher CLIPSIM indicates better alignment.
  • Metrics: DiffSSIM and DiffCLIP compare trained-model videos with randomly initialized-model videos using SSIM and CLIPSIM differences.These metrics reduce the influence of initialization when evaluating model improvement.
  • Metrics: Higher DiffSSIM and DiffCLIP indicate better learning ability and robustness.The metrics measure changes between trained and randomly initialized models.

B.4 TRAINING STRATEGY

The training strategy divides simulation into stages, groups rendered frames into mini-batches, and repeatedly optimizes each clip before proceeding.

  • Training strategy: Training partitions the simulation into multiple stages, groups each stage’s frames into mini-batches, and optimizes every video clip multiple times.Frames are sampled at a fixed interval of simulation steps.

C ADDITIONAL RESULTS

The trained model generalizes to diverse dynamic behaviors under different boundary conditions, with additional visualizations for single and multiple objects. The training strategy uses staged MPM simulation, rendering, score distillation, backpropagation, and optimization.

  • C.1 VISUALIZATIONS OF MOTION GENERALIZATION: Different boundary conditions produce diverse dynamic behaviors, demonstrated with a rubber wolf example.Additional motion-generalization visualizations are presented in Figure 6.
  • C.2 ADDITIONAL VISUALIZATIONS: The method visualizes 3D dynamic synthesis for single objects with different materials and for multiple objects.These results are presented in Figures 7 and 8.
  • Training Strategy: Training alternates staged simulation updates, rendering, score distillation sampling, backpropagation, and optimizer steps.The procedure preserves the end state of each stage for subsequent multi-batch processing and advances dynamics through MPM steps.

C.3 VISUAL COMPARISONS WITH BASELINES

Visual comparisons cover single and multiple objects across different materials. Baselines tend to produce similar material behavior across prompts, whereas the proposed method synthesizes a wider range of dynamics by learning material properties from pretrained video diffusion.

  • Single Objects: The comparisons evaluate single-object synthesis across different materials in Figures 9, 10, and 11.The figures include the proposed method and baseline results.
  • Multiple Objects: The comparisons evaluate multiple-object synthesis across different materials in Figures 12 and 13.The figures include the proposed method and baseline results.
  • Baseline Comparison: Baselines tend to predict the same material across different prompts, producing visually similar dynamics.The passage attributes this limitation to the fixed physics model’s limited expressiveness.
  • Method Comparison: The proposed method synthesizes a wider range of dynamic behaviors by learning material properties from pretrained video diffusion.This is reported as improving generalizability and expressiveness relative to the baselines.
Loading 2501.18982v1…