PaperlayerView exact revision on arXiv

Source-linked AI summary

Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion

Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, Yueqi Duan

arXiv:2406.04338v3cs.CVcs.AIcs.GR

TL;DR

3D generation often focuses on color and shape, while estimating diverse physical properties remains challenging. Physics3D combines viscoelastic simulation with video-diffusion priors to learn material properties, and reports high-fidelity, realistic dynamics across a range of materials. Its use in complex scenes with entangled objects still requires manual intervention to set movable-object scope and filling ranges.

  • Problem

    3D generative methods emphasize surface features such as color and shape, while estimating the diverse physical properties that govern object behavior remains challenging.

    Source: §1 · pp. 1–2

  • Method

    Physics3D expands material parameters to capture elasticity and viscosity, simulates dynamics with a viscoelastic MPM, and optimizes physical properties using video-diffusion priors via SDS.

    Source: §1 · p. 2

  • Results

    Physics3D is reported to create high-fidelity, realistic 3D dynamics across a wide range of materials.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The method addresses material behaviors beyond pure elasticity by modeling both elastoplastic and viscoelastic components and distilling physical priors from video.

    Source: §6 · p. 9

    In complex scenes with many entangled objects, the method requires manual assignment of movable-object scope and object filling ranges.

    Source: §6 · p. 9

Abstract

from arXiv · show

In recent years, there has been rapid development in 3D generation models, opening up new possibilities for applications such as simulating the dynamic movements of 3D objects and customizing their behaviors. However, current 3D generative models tend to focus only on surface features such as color and shape, neglecting the inherent physical properties that govern the behavior of objects in the real world. To accurately simulate physics-aligned dynamics, it is essential to predict the physical properties of materials and incorporate them into the behavior prediction process. Nonetheless, predicting the diverse materials of real-world objects is still challenging due to the complex nature of their physical attributes. In this paper, we propose \textbf{Physics3D}, a novel method for learning various physical properties of 3D objects through a video diffusion model. Our approach involves designing a highly generalizable physical simulation system based on a viscoelastic material model, which enables us to simulate a wide range of materials with high-fidelity capabilities. Moreover, we distill the physical priors from a video diffusion model that contains more understanding of realistic object materials. Extensive experiments demonstrate the effectiveness of our method with both elastic and plastic materials. Physics3D shows great potential for bridging the gap between the physical world and virtual neural space, providing a better integration and application of realistic physical principles in virtual environments. Project page: https://liuff19.github.io/Physics3D.

1 Introduction

Existing 3D generation emphasizes appearance, while conventional physics simulation can struggle with complex objects and large deformations. Physics3D estimates broader material properties by combining viscoelastic simulation with physical priors distilled from a video diffusion model.

1 Introduction

  • Existing 3D reconstruction and generation emphasize color, while mesh-based physical simulation can be time-consuming and struggle with large deformations or fracture-prone phenomena.These limitations matter for realistic interactive dynamics in applications such as virtual and augmented reality and animation.Source: §1 · p. 1
  • PhysDreamer estimates Young’s modulus from generated-video dynamics, but its single-parameter approach struggles with plastics, metals, and non-Newtonian fluids.Real-world objects can combine materials, motivating a broader range of physical properties.Source: §1 · p. 2
  • Physics3D learns diverse material dynamics by expanding physical parameters to capture elasticity and viscosity, then optimizing them through differentiable MPM and video-diffusion SDS.The method decomposes deformation into separate components whose stresses contribute to the overall force.Source: §1 · p. 2
  • A learnable internal-filling strategy optimizes selected 3D Gaussians alongside SDS-based physical-parameter optimization.This combines internal filling with video-diffusion priors in the optimization process.Source: §1 · p. 2
  • Experiments report effective creation of high-fidelity, realistic 3D dynamics for interactions across users and objects.The paper presents this as a future-facing application scope.Source: §1 · p. 2

2 Related Work

Related work covers dynamic 3D representations, models of nonrigid materials, and video-generation priors. Physics3D draws on video models’ capacity for realistic motion while addressing material behaviors beyond simple elasticity.

Dynamic 3D representations

  • Dynamic 3D Gaussian methods model temporal changes through per-frame regularization, deformation networks, or motion learned from generated videos.The cited approaches include Dynamic 3D Gaussians, 4D Gaussian Splatting, and DreamGaussian4D.Source: §2 · pp. 2–3

Viscoelastic materials

  • Viscoelastic models combine viscous-fluid and elastic-solid behavior, including creep, slow recovery, and permanent deformation under prolonged stress.These behaviors extend beyond conventional elastic models that return to their original shape after force removal.Source: §2 · p. 3

Video generation models

  • Video-generation models trained on extensive video datasets can produce realistic motion, and Physics3D uses Stable Video Diffusion to optimize physical properties.The paper aims to distill physical principles observed in videos for static 3D objects.Source: §2 · p. 3

3 Problem Formulation

The problem is to estimate physical attributes for 3D Gaussian particles and generate physically plausible motion. Physics3D expands beyond elastic properties by incorporating viscosity and simulating separate elastoplastic and viscoelastic stresses.

3 Problem Formulation

  • The formulation estimates particle mass, Young’s modulus, and Poisson’s ratio to organize force and velocity into physics-plausible motion.Young’s modulus and Poisson’s ratio control elastic dynamics; higher Young’s modulus yields smaller deformation under the same external force.Source: §3 · p. 3
  • Viscosity is added because elastic-only models can miss mixed-material behavior, especially rapid deformation where viscosity matters.Viscosity coefficients νv and νd resist rapid deformation as viscous dampers.Source: §3 · p. 3
  • The framework estimates viscosity coefficients and Lamé parameters, while particle mass is precomputed from density and volume and Poisson’s ratio is fixed across the object.The simulation combines independently calculated elastoplastic and viscoelastic stresses into overall stress.Source: §3 · p. 3, Figure 2 · p. 4

4 Method

Physics3D estimates physical properties for 3D objects using a viscoelastic MPM simulation. It combines elastoplastic and viscoelastic behavior and uses video-diffusion priors to guide optimization.

4.1 Preliminary

  • MPM represents materials as deformable particles and transfers information between particles and a background grid; integrating viscoelastic properties supports simulation of inelastic materials.The method uses a particle-to-grid and grid-to-particle loop.Source: §4.1 · p. 5

4.2 Physical Properties with Viscoelasticity Deformation

  • The elastic branch uses a fixed-corotated model, while the viscoelastic branch computes stress from FN and updates FN through trial-and-correction using FV.For simplicity, the viscosity parameters νd and νe are both set to νN.Source: §4.2 · p. 6

4.3 Optimization with Video Diffusion Model

  • Video-diffusion SDS guides optimization of simulated Gaussians, with image or text conditions and optional filling of internal volumes in selected solid objects.Filled Gaussians are optimized with the SDS process, and the two losses are optimized separately.Source: §4.3 · p. 6

5 Experiments

Experiments compare Physics3D with prior methods, evaluate rendered-video quality, and test the model’s components. Results report higher video-quality scores, more realistic oscillation and damping, and a contribution from both physics modules.

5.1 Experiment Setup

  • The evaluation uses real-world scenes, NeRF datasets, and synthetic BlenderNeRF scenes, comparing three baselines with video-quality metrics and a user study.The four real-world scenes are drawn from PhysDreamer.Source: §5.1 · pp. 7–8, §5 · p. 7

5.2 Qualitative Results

  • Physics3D balances high- and low-frequency oscillations with more realistic damping than the compared methods.PhysDreamer has subtle damping, PhysGaussian produces low-frequency motion, and DreamGaussian4D produces constant, low-magnitude periodic motion.Source: §5.2 · pp. 8–9, Figure 5 · p. 8

5.3 Quantitative Results

  • Physics3D receives higher average video-quality scores than the compared methods, indicating better video quality and motion consistency.The evaluation uses rendered videos from a fixed perspective.Source: §5.3 · p. 9, Table 1 · p. 9

5.4 Ablation Study

  • Removing either physics module degrades simulation realism: elastoplastic removal weakens elasticity, while viscoelastic removal reduces sustained damping and rebound.The damping and rebound deficiency is especially evident with minimal external disturbances.Source: §5.4 · p. 9, Figure 6 · p. 9

6 Conclusion

Physics3D learns diverse 3D-object physical properties from a video diffusion model using elastoplastic and viscoelastic modules. The conclusion reports realistic, high-fidelity dynamics, while noting manual intervention remains necessary in complex scenes with entangled objects.

6 Conclusion

  • Physics3D combines elastoplastic and viscoelastic modules and distills physical priors from a video generation model to learn diverse material properties.The elastoplastic component models pure elasticity; the viscoelastic module adds damping for materials exhibiting elasticity and viscosity.Source: §6 · p. 9
  • In complex environments with many entangled objects, users must manually assign movable-object scope and object filling ranges.The authors identify this as inefficient for more real applications.Source: §6 · p. 9

A More Results

The appendix reports additional experiments, including visual results and a user study.

A More Results

A.1 Additional Visual Results

Additional visual results show interactive motion sequences for varied objects and materials.

A.1 Additional Visual Results

  • The additional sequences apply external forces of different directions and magnitudes to static objects to show their motion states.The examples include non-elastic and composite materials such as rubber, fluffy bread, and fabric.Source: §A.1 · p. 14, Figure 7 · p. 14

A.2 Additional Ablation Study

Ablations show that both elastoplastic and viscoelastic components contribute to realistic simulation, while internal filling improves the deformation of hollow objects. In a separate user study, participants significantly preferred Physics3D across the assessed video qualities.

A.2 Additional Ablation Study

  • Removing either elastoplastic or viscoelastic components reduces simulation realism, with the former weakening elasticity and the latter reducing sustained damping and rebound.Without internal filling, the hollow cake tends to collapse; with it, deformation under external force is more realistic and smooth.Source: §A.2 · p. 15
  • Figure 8 presents an ablation study of internal filling, which supports assessing how filling affects simulated deformation.The accompanying text reports more realistic, smoother deformation with filling and collapse of the hollow cake without it.Source: §A.2 · p. 15, Figure 8 · p. 15

A.3 User Study

  • In a study of 30 volunteers across 20 rounds, users significantly preferred Physics3D over the compared methods and real-captured videos on quality, realisticity, object alignment, and fluency.Each round asked participants to choose their preferred video from five samples drawn from PhysDreamer, PhysGaussian, DreamGaussian4D, Physics3D, and real-captured videos.Source: §A.3 · p. 15
  • Table 2 describes user-study ratings of action coherence, motion realism, and overall quality on a 1–10 scale, with higher scores indicating better performance.The supplied caption does not report the table’s individual scores.Source: Table 2 · p. 15

B.1 Material Point Method (MPM) Algorithm

The appendix outlines MPM’s particle-grid simulation cycle and reviews dynamic 3D generation and score distillation concepts. It describes diffusion-model training as denoising and score-based guidance as a route for steering samples toward high-probability regions.

B.1 Material Point Method (MPM) Algorithm

  • MPM transfers particle mass and momentum to grid nodes, updates grid velocities using forces, then transfers velocities back to update particle states.The appendix identifies b as the B-spline degree and ∆x as Eulerian grid spacing.Source: §B.1 · p. 16

B.2 Dynamic 3D Generation.

  • Dynamic 3D generation adds temporal behavior to 3D representations, unlike static generation focused on spatial morphology.Gaussian Splatting represents 3D information with Gaussian kernels described by centers, scales, rotations, opacity, and color.Source: §B.2 · p. 16
  • Diffusion models learn a data distribution through progressive noising and backward denoising, with the final training loss described as simplified.The backward process estimates the origin distribution using an autoencoder.Source: §B.2 · pp. 16–17
  • Score Distillation Sampling aims to distill a complex generative model’s score function into a simpler form to guide sampling.The appendix describes an objective that learns an estimated score to approximate the model-distribution score.Source: §B.2 · p. 17
  • Score matching guides sampling toward high-probability regions by matching model and data-distribution score functions.The score function is described as the gradient of log density.Source: §B.2 · p. 17
  • The appendix notes that the U-Net Jacobian term is expensive to compute and poorly conditioned at small noise levels, and describes omitting it for an effective gradient.It attributes the omission to DreamFusion [43].Source: §B.2 · p. 17

B.3 Additional Analysis in Continuum Mechanics

The appendix explains continuum-mechanics quantities and the paper’s elastic and viscoelastic branches, including a trial-and-correction update that models dissipation. It also states that dissipation parameters are learned from videos and that equal component parameters simplify the simulation.

B.3 Additional Analysis in Continuum Mechanics

  • The paper uses Cauchy stress for most of its treatment; it relates force on a unit surface to stress and distinguishes Kirchhoff stress by its reference volume.Cauchy stress is measured with deformed volume, while Kirchhoff stress is measured with undeformed volume.Source: §B.3 · p. 18
  • The elastoplastic branch uses a fixed-corotated constitutive model for its elastic part and computes internal stress from the elastic deformation gradient.The appendix says the current paper considers only the elastic part of this branch.Source: §B.3 · p. 19
  • The viscoelastic branch factors total strain into elastic and viscous components; only the elastic component contributes to internal stress, while the viscous component affects its update.The update uses a trial-and-correction procedure that first evolves elastic strain, then introduces dissipation.Source: §B.3 · p. 19
  • The dissipation model takes its deviatoric and dilational parameters equal to represent homogeneous, isotropic material behavior, simplifying the simulation.The appendix says this equality assumption can be released, and that the update parameters A and B are learned from videos rather than using the dissipation parameters directly.Source: §B.3 · p. 20

C Ethical Statement

  • The ethical statement reports that research data were used with consent or under public-dataset terms and permissions, and that the work poses no societal or ethical harm.The authors also report precautions to respect privacy and research ethics.Source: C Ethical Statement · p. 20

Related papers

Loading 2406.04338v3…