Source-linked AI summary
PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, Shenlong Wang
TL;DR
Single-image video generation must reconcile realistic appearance with physically plausible and controllable motion. PhysGen combines inferred physical understanding, rigid-body simulation, and diffusion-based rendering to produce realistic physics-grounded videos. The authors report strong visual and physical results, while noting limitations for non-rigid objects and out-of-plane motion.
Problem
Existing graphics methods rely on predefined physics and lighting, while data-driven image-to-video methods do not adequately incorporate real-world physics.
Method
PhysGen infers object properties from a single image, simulates rigid-body dynamics under user inputs, and refines rendered motion with generative video diffusion.
Results
PhysGen produces realistic, physically plausible, controllable videos and consistently surpasses existing state-of-the-art video generative models.
Takeaways & Limitations
The resulting videos support controllable and interactive applications such as animating images and creating dynamics from user inputs.
Takeaways & Limitations
The method focuses mainly on rigid objects and lacks comprehensive 3D understanding, limiting non-rigid and out-of-plane motion handling.
Abstract
from arXiv · showhide
We present PhysGen, a novel image-to-video generation method that converts a single image and an input condition (e.g., force and torque applied to an object in the image) to produce a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process, enabling plausible image-space dynamics. At the heart of our system are three core components: (i) an image understanding module that effectively captures the geometry, materials, and physical parameters of the image; (ii) an image-space dynamics simulation model that utilizes rigid-body physics and inferred parameters to simulate realistic behaviors; and (iii) an image-based rendering and refinement module that leverages generative video diffusion to produce realistic video footage featuring the simulated motion. The resulting videos are realistic in both physics and appearance and are even precisely controllable, showcasing superior results over existing data-driven image-to-video generation works through quantitative comparison and comprehensive user study. PhysGen's resulting videos can be used for various downstream applications, such as turning an image into a realistic animation or allowing users to interact with the image and create various dynamics. Project page: https://stevenlsw.github.io/physgen/
1 Introduction
PhysGen addresses the difficulty of combining realistic appearance with physically plausible, controllable dynamics in single-image video generation. It integrates inferred physical understanding, rigid-body simulation, and diffusion-based rendering to produce realistic videos.
- Conventional graphics provide controllable dynamics but assume predefined physics and lighting, limiting animation of real-world images.
- Data-driven image-to-video methods generate realistic videos from single images but do not adequately incorporate real-world physics.
- PhysGen infers object composition and physical parameters, simulates rigid-body dynamics under user inputs, and renders the motion with video diffusion.
- The framework combines data-driven appearance realism with model-based physical plausibility and supports user-specified physics parameters and initial conditions.
- PhysGen is evaluated on web and self-captured images against image-to-video and image-editing methods using quantitative, qualitative, and user-study comparisons.
- The authors report a training-free system whose videos are realistic, physically plausible, controllable, and consistently surpass existing state-of-the-art video generative models.
2 Related work
Prior work separately offers rule-based physical simulation or data-driven video generation, but PhysGen combines inferred physical parameters, rigid-body simulation, and generative rendering for image-based dynamics.
- Physics simulation: Rule-based and solver-based physics simulators support many physical processes but face limitations in expressiveness, efficiency, generalizability, and parameter tuning.
- Physics simulation: Most forward physics models still depend on physical parameters predefined by users.
- Physics-aware vision: Image-based physics pipelines infer scene representations before reasoning or simulation, but primarily rely on synthetic training data with potential generalization problems.
- Video generation: Video generative models can produce photorealistic and coherent videos, while remaining challenges include object permanence, lighting, unrealistic motion, and physics violations.
- Image animation: Image animation methods include class-specific physical heuristics and data-driven models that predict consecutive frames from an input image.
- PhysGen: PhysGen takes a single image and user-specific force or torque, explicitly models object dynamics with rigid physics, and automatically infers physical parameters using large foundation models.
3 Approach
PhysGen converts a single image and user-specified physical conditions into image-space motion, then renders and refines that motion into a coherent video. Its pipeline combines perception, rigid-body simulation, compositing, relighting, and diffusion refinement.
- 3.1 Perception: The perception stage uses pretrained models to segment objects, classify foreground and background entities, and infer physical properties from image masks.
- 3.1 Perception: For each object, PhysGen estimates mass, elasticity, and friction, models friction with Coulomb friction, and computes rotational inertia from mass and shape.
- 3.1 Perception: Objects are converted from raster masks into circle or polygon primitives, with circles supporting realistic rolling-motion simulation.
- 3.2 Image space dynamics simulation: Image-space rigid-body simulation is chosen because it couples to the output video and avoids requiring complete 3D reconstruction from one image.
- 3.2 Image space dynamics simulation: The simulator integrates rigid-body motion using forces, torques, mass, rotational inertia, numerical ODE integration, gravity, friction, elasticity, and collision responses.
- 3.4 Generative refinement with latent diffusion models: Rendering transforms and composites simulated objects, while diffusion-based video refinement preserves foreground consistency and synthesizes background content such as shadows.
4 Experiments
PhysGen is evaluated for realistic, physically plausible, controllable image-to-video generation across diverse data and comparisons. Experiments assess appearance, motion, perception, physical reasoning, human judgments, and limitations.
- Experimental setup: The evaluation uses internet and self-captured images, compares against image-to-video and image-manipulation baselines, and includes qualitative, quantitative, and human-study assessments.The human evaluation compares three image-to-video models, while quantitative evaluation separates appearance and motion using Image-FID and Motion-FID.
- Human evaluation: PhysGen ranks first for both physical-realism and photo-realism in human evaluation using a five-point rating scale.Fourteen participants rated 15 videos against three image-to-video baselines.
- Quantitative evaluation: PhysGen achieves both low Image-FID and Motion-FID, unlike methods that trade off image consistency against motion realism.The evaluation uses 800 images per method and computes FID separately on image appearance and colorized optical flow.
- Qualitative results: PhysGen generates reasonable complex procedures involving object-object and object-scene interactions under varied physical properties and initial conditions.Results cover 8 different physical procedures from multiple sources.
- Qualitative results: Compared with existing image-to-video models, PhysGen produces more physically plausible futures, while Motion Guidance introduces inconsistency and artifacts during interactions.The comparison reports failures in understanding direction, physical meanings, and interactions for the three image-to-video models, and artifacts in the domino case.
- Analysis: The perception module reaches 0.93 precision and 0.82 recall at 0.5 IoU on 118 movable instances, while removing GPT-4V physical reasoning worsens Image-FID from 105.70 to 111.01 and Motion-FID from 30.20 to 36.60.The perception evaluation uses 10 complex open-world images; the physical-reasoning comparison attributes the changes to replacing GPT-4V estimates with random human-range samples.
- Controllability: Varying initial force and speed direction and magnitude produces diverse physically plausible trajectories, demonstrating controllability.The results are presented by changing initial conditions across rows.
- Limitations: The method mainly targets rigid objects and cannot handle out-of-plane motions because it lacks comprehensive 3D understanding.The authors identify deformable physics and full 3D understanding as future work.
5 Conclusion
PhysGen turns static images into high-fidelity, physics-grounded, temporally coherent videos and supports controllable, interactive downstream applications.
- PhysGen combines a model-based physical simulator with data-driven video generation to animate static images.
- The system enables controllable and interactive downstream applications.
Supplementary Material – PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
The supplementary material documents PhysGen’s implementation, generative refinement procedure, runtime components, qualitative comparisons, and controllable generation results, while noting dedicated limitation analysis.
- The supplement includes implementation details, additional experiments, controllable generation results, and limitation analysis.
- Object segmentation uses GPT-4V for recognition and movability judgment, followed by Grounded-SAM instance segmentation.
- Background boundaries are extracted using estimated depth and normals, with candidate objects filtered by relative depth and planar structure.
- Geometry primitives are selected by fitting circles first and switching to generic polygons when mask IoU is below 0.85.
- Foreground masks are dilated before inpainting to recover a clean background, although heavy shadows may not be completely removed.
- Image-space simulation uses a 1-second timestep, gravity g = 980cm/s2, and a 1pixel-to-1cm mapping without metric-scale reasoning.
- Latent video diffusion progressively adds and removes noise, decoding the final latent representation into the output video.
- Rendered, relit, and diffusion-refined videos are compared qualitatively, with the diffusion-rendered output described as most photorealistic.
A.4 Runtime Analysis
A single PhysGen generation takes around 3 minutes, with perception and rendering dominating runtime while simulation and refinement are shorter.
- Around 3 minutes are required for one generation, compared with 70 minutes for Motion Guidance.
- Perception takes 1 minute, 120-step simulation takes 5 seconds, rendering takes 1 minute, and generative refinement takes 35 seconds.
- The runtime analysis summarizes each module’s duration for a single run.
B.1 Human evaluation
Human evaluation shows PhysGen receives substantially more agree and strongly-agree ratings for both physical realism and photorealism than compared I2V methods.
- PhysGen outperforms compared I2V methods by a large margin in physical-realism and photo-realism ratings.
- Its average rating falls within the Agree level for both evaluated claims.
B.2 Quantitative evaluation
PhysGen is quantitatively evaluated using real-captured videos with varied initial forces and open-world movable-object segmentation. Its perception pipeline achieves strong segmentation performance at 0.5 IoU.
- 50 real-captured videos per scene are recorded as ground truth by varying the initial forces applied to the object.
- 0.93 precision and 0.82 recall are achieved for open-world movable-object segmentation at 0.5 IoU.The evaluation uses 10 complex scenes containing 118 annotated movable instances.
C.2 GPT-4V physical property estimation evaluation
GPT-4V is evaluated as the physical-property estimator for mass, friction, and elasticity. The reported results indicate reasonable estimation across these properties.
- GPT-4V achieves 75% mass-estimation accuracy within 30% of ground truth, with a 0.39 kg average absolute error across 20 portable objects.
- GPT-4V matches reference rankings in 12 of 13 friction comparisons and 6 of 7 elasticity comparisons.Reference videos use toy cars sliding on surfaces and balls bouncing across materials because direct ground truth is unavailable.
D More qualitative results
Additional qualitative comparisons show that baseline image-to-video models struggle with physically realistic motion and temporal object consistency, while PhysGen provides more controllable generation results.
- Qualitative comparison: DynamiCrafter often produces static scenes or lighting and viewpoint changes instead of reasonable physical motion.
- Qualitative comparison: SEINE and I2VGen-XL generate apparent motion, but the motion is physically unrealistic and foreground objects are inconsistent across time.
- Controllable generation: PhysGen demonstrates additional controllable video-generation results.
E Limitation Analysis
The limitation analysis identifies artifacts arising from inpainting, primitive–segmentation mismatches, inaccurate segmentation, and diffusion refinement. Additional figures visualize sampled videos, property-estimation scenarios, segmentation results, and controllable outputs.
- Inpainting: Incorrect inpainting can leave shadows in generated frames because box shadows are not fully removed from the inpainted image.
- Segmentation and primitives: Differences between simulated primitives and real segmentation can create gaps between blocks in rendered videos.
- Segmentation: Inaccurate boundary segmentation can distort billiard-ball shapes and introduce artifacts near their synthesized boundaries.
- Diffusion refinement: Diffusion refinement can hallucinate or slightly alter foreground objects because generated latent information is injected into both foreground and background.
- Additional visualizations: Figures additionally show real-captured evaluation videos, physical-property testing scenes, segmentation outputs, qualitative comparisons, and controllable generation.