Source-linked AI summary
PhysGen3D: Crafting a Miniature Interactive World from a Single Image
Boyuan Chen, Hanxiao Jiang, Shaowei Liu, Saurabh Gupta, Yunzhu Li, Hao Zhao, Shenlong Wang
TL;DR
Generating physically plausible, controllable futures from a single image remains difficult because image-to-video models lack precise physical control, while many digital-twinning methods require multi-view or depth data. PhysGen3D reconstructs an interactive 3D scene, simulates its dynamics, and renders future motion; user studies report more flexible control, better intent alignment, and superior physical realism than Pika, Kling, and Gen-3 with comparable rendering quality. The method is scoped to object-centric scenes with simple spatial geometry and inter-object relationships.
Problem
Image-to-video models lack precise control and physical grounding, while many digital-twinning methods require multi-view images or depth sensors for interactive scenes.
Method
PhysGen3D combines single-image geometric and semantic reconstruction with physics simulation and physics-based rendering to create a camera-centric interactive 3D scene.
Results
PhysGen3D provides significantly more flexible motion control, better alignment with user intentions, and superior physical realism than Pika, Kling, and Gen-3 while maintaining comparable rendering quality.
Takeaways & Limitations
The system enables user-controlled simulation of diverse future motions, materials, object edits, and dynamic phenomena from a single image.
Takeaways & Limitations
The system targets object-centric scenes with simple spatial geometry and relationships; complete reconstruction of more complex scenarios remains an open challenge.
Abstract
from arXiv · showhide
Envisioning physically plausible outcomes from a single image requires a deep understanding of the world's dynamics. To address this, we introduce PhysGen3D, a novel framework that transforms a single image into an amodal, camera-centric, interactive 3D scene. By combining advanced image-based geometric and semantic understanding with physics-based simulation, PhysGen3D creates an interactive 3D world from a static image, enabling us to "imagine" and simulate future scenarios based on user input. At its core, PhysGen3D estimates 3D shapes, poses, physical and lighting properties of objects, thereby capturing essential physical attributes that drive realistic object interactions. This framework allows users to specify precise initial conditions, such as object speed or material properties, for enhanced control over generated video outcomes. We evaluate PhysGen3D's performance against closed-source state-of-the-art (SOTA) image-to-video models, including Pika, Kling, and Gen-3, showing PhysGen3D's capacity to generate videos with realistic physics while offering greater flexibility and fine-grained control. Our results show that PhysGen3D achieves a unique balance of photorealism, physical plausibility, and user-driven interactivity, opening new possibilities for generating dynamic, physics-grounded video from an image.
1. Introduction
PhysGen3D addresses the difficulty of generating physically grounded, user-controlled futures from a single image by reconstructing an interactive 3D scene and simulating its dynamics. Compared with closed-source image-to-video models, it provides more flexible control and physical realism while maintaining comparable rendering quality.
- Photographs omit alternative physical outcomes, motivating a model that generates video futures from a single static image.
- Image-to-video models produce photorealistic videos but lack precise control and physical grounding.Consequently, they cannot reliably achieve specified physical effects or guarantee physical realism.
- Existing digital-twinning methods support physical interactions but often require multi-view images or depth sensors, while single-image methods remain constrained by object type, physics, or dimensional scope.
- PhysGen3D transforms one image into an amodal, camera-centric, interactive 3D scene by estimating object shape, pose, physical and lighting properties, background geometry, and appearance.Pretrained vision models are integrated to construct an image-centric digital twin despite the task’s ill-posed nature.
- Material-point simulation and physics-based rendering produce realistic, stable, and controllable dynamics, including user-specified speed and material properties.The pipeline uses inferred physical properties and operates without task-specific training through large pretrained models.
- User studies found significantly more flexible motion control, better alignment with user intentions, and superior physical realism than Pika, Kling, and Gen-3, with comparable rendering quality.
2. Related work
Related work spans controllable 3D reconstruction, video generation, and image animation, while PhysGen3D combines these directions into a physics-based interactive pipeline. Its method overview links single-image world creation, particle-based dynamics, and physics-based rendering.
- Single-view 3D reconstruction methods often emphasize holistic geometry or foreground objects, leaving object understanding and environmental interactions incomplete.
- Controllable video generation conditions pretrained diffusion models with signals such as depth maps, translations, layouts, or combinations of these inputs.
- PhysGen3D’s framework comprises 3D world creation, Taichi-Elements particle-based dynamics simulation, and physics-based rendering with two-pass shadow mapping.
- Controllable image animation uses temporal prediction, physical heuristics, interaction controls, and motion-related priors to animate image content.
3. Approach
PhysGen3D reconstructs a camera-centric miniature world from one image, estimates geometry and physical properties, simulates material-dependent dynamics, and renders the resulting motion back into the scene. The pipeline is designed for controllable, photorealistic, physically plausible video generation.
- 3. Approach: The approach jointly targets control of object materials, dynamics, and motions despite partial observations and ill-posed physical reasoning from a single image.
- 3. Approach: The reconstructed scene passes through PBR material estimation, material-point simulation, and dynamic rendering with reintegration into the original scene.
- 3.1. Interactive 3D World from a Single Image: Pretrained vision foundation models provide priors for inferring objects’ relationships, geometry, appearance, materials, and physical properties in an amodal reconstruction.
- 3.1. Interactive 3D World from a Single Image: GPT-4o identifies foreground categories, Grounded-SAM segments instances, and InstantMesh reconstructs complete 3D meshes using synthesized multi-view images.
- 3.1. Interactive 3D World from a Single Image: Background depth is reconstructed to provide simulation support and collision geometry while serving as a static rendering backdrop for illumination and shadows.
- 3.1. Interactive 3D World from a Single Image: A coarse-to-fine registration strategy places generated meshes in camera coordinates by combining feature matching, PnP pose estimation, scale adjustment, and differentiable mask-depth alignment.The alignment losses jointly compare rendered and observed masks and rendered and predicted depths.
- 3.1. Interactive 3D World from a Single Image: Differentiable rendering minimizes Dice and depth losses to align estimated meshes with observed masks and predicted point-cloud depth.
- 3.1. Interactive 3D World from a Single Image: Inverse rendering estimates lighting and optimizes per-object albedo, roughness, and metallic PBR properties, using uniform roughness and metallic values to simplify unknown back views.
4. Experimental Results
PhysGen3D is evaluated against image-to-video baselines through qualitative comparisons, human and automated assessments, controllability tests, editing, tracking, and ablations. The results indicate stronger physical plausibility and control, while revealing rendering and simulation failure cases.
- Experimental Scope: The test set emphasizes object-centric scenes with one or a few objects and excludes excessive objects, heavy occlusion, and highly uneven surfaces.VEnhancer is optional and restores some details but introduces additional hallucinations.
- Evaluation Setup: The evaluation compares PhysGen3D with open-source motion controllers and commercial I2V models, using privileged motion guidance for several baselines.Pika 1.5 and Gen-3 receive text descriptions, while DragAnything, MOFA-Video, and Kling receive manually specified motion guidance.
- Qualitative Comparison: PhysGen3D produces more physically plausible and controllable videos than SOTA I2V models in motion-control and material comparisons.The toy-dog and book examples show more accurate collapse simulation and more physically realistic results than competing models.
- Dynamics: Different elasticities with fixed positions and velocities, or different velocity directions with fixed materials, generate varied dynamics from one input image.These comparisons demonstrate control over physical parameters and motion trajectories.
- Editing: PhysGen3D supports object removal, addition, and replacement, enabling video edits through manipulation of generated 3D assets.Examples include removing a chair so a toy falls and exchanging an object between scenes.
- Tracking: An explicit 3D representation combined with particle-based simulation supports dense 3D tracking for collapse and bounce scenarios.The framework generates tracking results across different physical events.
- Ablation: Ablations show that removing pose optimization harms scene replication, removing inverse texture harms color matching, and removing point sampling can crash the simulator.These components support reconstruction fidelity and simulation feasibility.
5. Limitations
PhysGen3D is designed for object-centric scenes with simple spatial geometry and inter-object relationships, while complete reconstruction of more complex scenes remains challenging.
- Scope Boundary: PhysGen3D is designed for object-centric scenes with simple spatial geometry and inter-object relationships.Complete-scene reconstruction for more complex scenarios remains an open challenge, alongside rendering, perception, and simulation failures.
6. Conclusions
PhysGen3D transforms a static image into an interactive 3D scene for simulating and rendering future motions based on user input. It integrates reconstruction, dynamic simulation, and physics-based rendering to produce realistic, controllable videos with diverse material behaviors.
- Conclusion: PhysGen3D transforms a static image into an interactive 3D scene for simulating and rendering future motions based on user input.The framework extends image-to-video generation with a 3D scene representation.
- Conclusion: The framework integrates 3D world reconstruction, model-based dynamic simulation, and physics-based rendering to generate realistic, controllable videos.These modules support more realistic motion and diverse material behaviors.
Supplementary Material
The supplementary material details PhysGen3D’s reconstruction, physics reasoning, scaling, and MPM simulation pipeline, including handling occlusions, shadows, and material properties.
- Scene Reconstruction: Occluded objects are segmented and inpainted sequentially from foreground to background, with each step building on prior object removal.The second inpainting result becomes the background.
- Scene Reconstruction: Object-related shadows are isolated using thresholded brightness regions, connected components, mask dilation, and inpainting.The method dilates each mask with a kernel of size 50; adaptable per-object shadow removal remains future work.
- Physical Parameter Reasoning: GPT-4o estimates object names, densities, stiffness categories, sizes, internal filling, and supporting-surface roughness for simulation.The prompt requests one-word or numeric answers in a fixed per-object format.
- Physical Parameter Reasoning: Because exact Young’s modulus estimates are unstable, soft, medium, and hard map to simulator elasticities E = 5 × 10^4, 5 × 10^5, and 5 × 10^6.Simulator elasticity does not directly equal real Young’s modulus.
- Simulation: The simulator rescales gravity and physical parameters to fit objects ranging from centimeters to tens of meters into a fixed simulation size and resolution.The scaling factor is k = s′/s0, and gravity is set to g′ = k × g0.
- Simulation: Modified Taichi Elements implements inhomogeneous-material MPM using particle-to-grid and grid-to-particle transfers for continuum-material simulation.The governing dynamics use density, velocity, stress, and external forces, while deformation and velocities are updated through the transfer steps.
B. Additional Details of Experiments
The experiments compare PhysGen3D with competitive baselines using human and GPT-based evaluations alongside multiple video-quality metrics.
- Experimental Design: Experiments compare competitive baselines using human evaluation, GPT-based evaluation, and multiple evaluation metrics.Detailed experimental settings, metrics, and additional results are provided in the supplementary material.
B.1. Experiments Settings
The comparative experiment matched baseline generation goals by translating manually assigned object velocities into shared textual prompts.
- Experimental Settings: Manually assigned initial 3D object velocities were converted into textual dynamics prompts shared with all three baseline models.Examples include an elephant hopping and falling or a book falling while an orange rolls forward.
B.2. Evaluation
Evaluation combines human ratings and GPT-4o judgments of generated videos, using physical realism, photorealism, and semantic consistency criteria.
- Human Evaluation: Thirty-one participants rated 27 videos on physical realism, photorealism, and semantic consistency using five-point questionnaire scales.Videos covered varied scenes, motion conditions, and effects, with outputs shown in random order.
- GPT-4o Evaluation: GPT-4o evaluated PhysGen3D and baselines on the same three criteria using 10 evenly sampled frames plus the input image and prompt.Physical realism includes adherence to physical rules and properties such as elasticity and friction.
- GPT-4o Evaluation: The evaluation discourages completely stable videos by penalizing outputs that lack dynamic behavior.This criterion is included within the physical-realism assessment.
- GPT-4o Evaluation: The GPT-4o evaluation scores each video from 0 to 1 for physical realism, photorealism, and semantic consistency.The evaluation prompt requests separate scores for each video and criterion.
B.3. Additional Results
Additional results show that PhysGen3D supports diverse material, collision, camera, and input-image effects while outperforming or matching alternatives on key qualities. They also expose a trade-off: post-processing can improve photorealism but may compromise physical accuracy.
- Human Evaluation Results: PhysGen3D significantly outperforms baseline generative models in physical realism and semantic consistency, while the four models perform comparably in photorealism.Human evaluations report participant agreement that most PhysGen3D results are physically realistic and semantically consistent.
- Additional Qualitative Results: Qualitative results demonstrate sandy-material transformation, bouncing and rolling effects, multi-object apple collisions, and video generation from a painting.The examples include a teddy bear transformed into sand and three apples colliding with one another.
- VEnhancer: VEnhancer recovers fine details and improves apparent photorealism, but can introduce hallucinations that illustrate a trade-off with physical accuracy.The post-processing uses diffusion-model priors to compensate for reconstruction and rendering errors but cannot guarantee real-world physics.
- Comparison with Open-Source Models: MOFA-Video and DragAnything introduce unrealistic deformations; DragAnything can lose background stability, while MOFA offers better motion control but lacks realism.The comparison reports these failures for two open-source diffusion models, with quantitative VBench results supporting the findings.
- Applications: PhysGen3D supports novel-view camera control with good cross-view consistency and environmental coherence.The extension combines background outpainting and meshing with rendering from novel views.
C. Limitations
The limitations concern reconstruction and rendering errors, ill-posed inference from partial observations, restricted collision and material modeling, and difficulty with complex scenes.
- Failure Modes: Failure cases span perception, simulation, and rendering, including floating mesh points, incorrect material estimates, and inaccurate inverse-rendering results.Depth and light estimation can also fail, although those errors are described as comparatively rare.
- Failure Modes: Reconstructing full geometry, physics, and textures from partial observations is inherently ill-posed and requires substantial prior knowledge.This ill-posedness is identified as a source of many observed failures.
- Simulation Scope: The current system supports only a single collider surface and models each object with homogeneous density and elasticity.Future extensions could use all stable components as colliders and assign different materials to different object parts.
- Scene Scope: PhysGen3D is designed for object-centric scenes, while reconstructing complete scenes with more complex spatial relationships remains an open challenge.The supported scope emphasizes rigid and deformable objects and simple scene structure.