Source-linked AI summary
PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation
Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y. Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, William T. Freeman
TL;DR
Action-conditioned 3D dynamics require material-property estimates that are difficult to obtain because real-object measurements and ground-truth data are limited. PhysDreamer distills dynamics priors from video generation models to estimate material properties for static 3D objects and synthesize interactive motion. It significantly outperforms existing techniques on motion realism and visual quality, while remaining computationally demanding and limited to elastic objects without collisions.
Problem
Action-conditioned 3D dynamics require physical material properties, but estimating them is difficult because real-object measurements and ground-truth material data are lacking.
Method
PhysDreamer distills dynamics priors learned by video generation models to estimate material properties for static 3D objects and uses differentiable MPM simulation for interactive dynamics.
Results
PhysDreamer significantly outperforms existing techniques on motion realism and visual quality.
Takeaways & Limitations
PhysDreamer enables static 3D objects to respond to novel interactive stimuli in a physically plausible manner.
Takeaways & Limitations
The method requires manual object-background separation and boundary-condition setup, takes approximately one minute per second of video on an NVIDIA V100, and is restricted to elastic objects without collisions.
Abstract
from arXiv · showhide
Realistic object interactions are crucial for creating immersive virtual experiences, yet synthesizing realistic 3D object dynamics in response to novel interactions remains a significant challenge. Unlike unconditional or text-conditioned dynamics generation, action-conditioned dynamics requires perceiving the physical material properties of objects and grounding the 3D motion prediction on these properties, such as object stiffness. However, estimating physical material properties is an open problem due to the lack of material ground-truth data, as measuring these properties for real objects is highly difficult. We present PhysDreamer, a physics-based approach that endows static 3D objects with interactive dynamics by leveraging the object dynamics priors learned by video generation models. By distilling these priors, PhysDreamer enables the synthesis of realistic object responses to novel interactions, such as external forces or agent manipulations. We demonstrate our approach on diverse examples of elastic objects and evaluate the realism of the synthesized interactions through a user study. PhysDreamer takes a step towards more engaging and realistic virtual experiences by enabling static 3D objects to dynamically respond to interactive stimuli in a physically plausible manner. See our project page at https://physdreamer.github.io/.
1 Introduction
PhysDreamer addresses action-conditioned 3D dynamics by distilling dynamics priors from video generation models to estimate material properties for static objects. It synthesizes physically plausible responses to novel interactions and outperforms existing techniques in motion realism and visual quality.
- Motivation: Existing 3D methods can create static or unconditioned dynamic assets but do not handle responses to new physical interactions.Examples include a rose reacting to a breeze or touch.
- Motivation: Estimating material properties is difficult because real-object measurements and ground-truth material data are scarce, while properties may vary spatially.This motivates using priors learned from diverse videos of the physical world.
- Approach: PhysDreamer distills video-generation dynamics priors to infer the physical material properties that drive static 3D objects’ dynamic behavior without ground-truth material data.The approach hypothesizes that video models implicitly capture relationships between object appearance and dynamics.
- Approach: The method represents objects with 3D Gaussians, models material fields with a neural field, and uses differentiable MPM simulation and rendering to optimize material and initial-velocity fields.Optimization matches pixel-space observations and the demonstrations focus on elastic objects including flowers, plants, a beanie hat, and a telephone cord.
- Results: PhysDreamer significantly outperforms existing techniques on motion realism and visual quality.The introduction reports this comparison against state-of-the-art methods without specifying aggregate percentages here.
- Conclusion: The approach enables static 3D objects to respond dynamically to interactive stimuli in a physically plausible manner.The paper presents this as a step toward more engaging and realistic virtual experiences.
2 Related work
Related work spans dynamic 3D reconstruction, video-model-based dynamic scene synthesis, interactive motion generation, and physics-based simulation. These approaches motivate PhysDreamer’s focus on interactive motions under new physical interactions rather than fixed-length or nonphysical motion generation.
- Dynamic 3D reconstruction: Dynamic 3D reconstruction recovers dynamic-scene representations from depth scans, RGBD videos, or monocular and multi-view videos.The monocular setting is especially challenging with slow-moving cameras and fast-moving scenes.
- Dynamic scene synthesis: Methods combining 3D generation with video generation models extend static representations temporally, but applying video diffusion SDS requires substantial computation and memory.Examples include Make-A-Video3D and later improvements using efficient representations, stronger priors, and stable training.
- Interactive motion generation: Existing dynamic-scene synthesis methods are limited to fixed-length 3D videos, whereas this work targets interactive 3D motions under new physical interactions.The distinction is between producing predetermined videos and responding to arbitrary physical interactions.
- Interactive motion generation: Interactive motion generation conditions animation on inputs such as text, motion fields, motion layers, or direct dragging and pulling.Earlier methods also animated images using image-space representations derived from subtle vibrational motions.
- Physics-based simulation: Physics-based 3D approaches synthesize motions under physical interactions by integrating simulations with NeRFs or 3D Gaussians.Related systems include Virtual Elastic Objects, PAC-NeRF, DANO, and PhysGaussian.
- Video generation: Current video generation models produce realistic videos but cannot support physics-based object interactions through external forces.This limitation separates video realism from force-driven physical interaction.
3 Problem formulation
PhysDreamer formulates interactive motion synthesis as estimating spatially varying material properties for a static 3D Gaussian object. Young’s modulus is central because it controls stiffness-related deformation and motion, while Poisson’s ratio is treated as homogeneous.
- Object and material representation: The formulation starts from a static object represented as 3D Gaussians and seeks physical material property fields enabling realistic interactive motion synthesis.The properties include mass m, Young’s modulus E, and Poisson’s ratio ν.
- Young’s modulus: Young’s modulus E is emphasized because it measures stiffness and strongly determines motion under applied forces.Higher values produce less deformation and more rigid, higher-frequency motion; lower values produce more flexible, elastic behavior.
- Young’s modulus: Under the same external force, a flower with Young’s modulus 100× has smaller oscillations and higher frequency than one with 1×.The 1× flower sways the most and oscillates at the lowest frequency.
- Material field: The target is a spatially varying Young’s modulus field E(x), queried at each particle position as E_p = E(x_p).This lets particle simulation use location-dependent stiffness values.
- Material field: Particle mass is computed from constant density and particle volume, while Poisson’s ratio is assumed homogeneous because preliminary experiments found negligible motion impact.Particle volume is estimated from background-cell volume divided by the number of particles in that cell.
4 PhysDreamer
PhysDreamer converts static 3D Gaussian objects into interactive elastic dynamics by matching differentiably simulated motion to video-generated reference motion. It estimates material and initial-velocity fields while accelerating simulation through particle downsampling.
- Method: PhysDreamer renders a static 3D Gaussian object, generates an image-to-video reference, and optimizes spatially varying material and initial-velocity fields to match rendered frames.The optimization minimizes pixel-space discrepancy between simulated and reference videos using differentiable MPM and rendering.
- Physics simulation: 3D Gaussian particles serve as the simulated object representation, while MPM provides a particle-based physics simulator for elastic dynamics.MPM supports GPU parallelization and topology changes, and directly operates on Gaussian particles.
- Physical model: The elastic model uses fixed-corotated hyperelastic energy, with parameters μ and λ determined by Young’s modulus E and Poisson’s ratio ν.The deformation gradient captures local rotation and strain, and the stress-strain formulation governs elastic dynamics.
- Simulation and rendering: The differentiable simulation advances particle positions, velocities, deformation gradients, and velocity gradients before rendering each frame through a differentiable Gaussian renderer.The simulation state includes particle physical properties such as mass, Young’s modulus, Poisson’s ratio, and volume.
- Acceleration: K-Means downsampling reduces computational cost by simulating only driving particles and reconstructing particle transforms from neighboring driving particles during rendering.This addresses the burden of simulating the millions of particles often required for high-fidelity Gaussian rendering.
5 Experiments
Experiments evaluate PhysDreamer on eight real-world static scenes using baselines, real captured videos, and a user study focused on motion realism and visual quality. Results favor PhysDreamer over both baselines, while multi-view supervision improves performance on self-occluding objects.
- Experimental setup: The evaluation uses eight scenes, including flowers, an alocasia plant, a telephone cord, and a beanie hat, with interaction videos for all but the red rose.Scenes are captured from multiple views, and most include natural-motion videos after interactions such as poking or dragging.
- Baselines: PhysDreamer is compared with PhysGaussian, which manually sets material parameters, and DreamGaussian4D, which generates non-interactive dynamics from a driving video.Both baselines use static 3D Gaussian representations, but only PhysGaussian incorporates MPM simulation.
- Evaluation protocol: The user study applies 2AFC comparisons of synchronized videos, asking participants to judge visual quality and motion realism.The study recruited 100 participants who judged all eight scenes, producing 800 samples per baseline comparison.
- Results: 80.8% of samples preferred PhysDreamer over PhysGaussian for motion realism, while 65.0% preferred it for visual quality.PhysGaussian produced large, unrealistic slow motion, which the authors attribute to lacking principled material-property estimation.
- Results: Compared with DreamGaussian4D, 70.0% and 63.5% of samples preferred PhysDreamer for visual quality and motion realism, respectively.Additionally, 73.6% preferred PhysDreamer for motion amount; the comparison highlights PhysDreamer’s simulated damping versus DreamGaussian4D’s small periodic motion.
- Multi-view supervision: Two-view supervision improved alocasia results, with 81.0% preferring it for visual quality and 86.0% for motion realism over single-view supervision.Multiple viewpoints provide more comprehensive coverage of self-occluding object parts and reduce artifacts in material estimation.
6 Conclusion
PhysDreamer endows static 3D objects with spatially varying physical material properties to synthesize interactive dynamics. Its demonstrated scope is diverse elastic objects, with important computational and interaction limitations.
- PhysDreamer distills video-generation dynamics priors to estimate spatially varying material properties for static 3D objects.
- The approach demonstrates interactive dynamics synthesis across a diverse set of elastic objects.
- The method requires manual object extraction and boundary-condition specification for stationary parts.
- Producing one second of video takes approximately one minute on an NVIDIA V100 GPU.
- The work is restricted to elastic objects without collisions.
A Metrics
The evaluation compares PhysDreamer with PhysGaussian and DreamGaussian4D using distributional video-quality metrics against real captured videos. FVD uses standardized temporal windows and preprocessing, while FID aggregates frames across objects.
- FVD compares PhysDreamer, PhysGaussian, and DreamGaussian4D against real captured videos.
- FVD uses a 16-frame window, 2-frame stride, I3D features, resizing to 144 pixels, and 128×128 center crops.
- FID provides an additional comparison among PhysDreamer, PhysGaussian, and DreamGaussian4D.
- FID incorporates all frames across all objects, totaling 4200 frames per method.
B User Study
The user-preference evaluation recruits participants through Prolific and presents an anonymized survey using Google Forms.
- Participants are recruited through Prolific for the human preference evaluation.
- The survey is presented through Google Forms and is fully anonymized for participants and the host.
C Algorithm details
The acceleration algorithm downsamples Gaussian particles into K-Means drive points, simulates those points, fits local rigid transforms, and renders transformed particles.
- Algorithm 1 accelerates the material point method using K-Means downsampling.
- Each Gaussian particle is represented by position, opacity, rotation, covariance, and color, with position x shaped [N, 3].
- K-Means clusters particle positions into drive points, producing drive_x with shape [M, 3].
- The algorithm precomputes nearest drive-point neighborhoods using pairwise distances and selects top_k indices.
- It queries initial velocity and material parameters at drive points, then simulates the drive points.
- Rigid transforms fitted from simulated drive points are applied to interpolate particle positions and orientations before rendering.