Source-linked AI summary
Physical Simulator In-the-Loop Video Generation
Lin Geng Foo, Mark He Huang, Alexandros Lattas, Stylianos Moschoglou, Thabo Beeler, Christian Theobalt
TL;DR
PSIVG addresses the difficulty of enforcing physical constraints in diffusion-based video generation by integrating a 3D physical simulator into the generation process, with TTCO improving foreground texture consistency. It produces videos with superior physical realism and visual quality, and was preferred in 82.3% of user-study comparisons.
Problem
Diffusion-based video generation lacks an explicit mechanism for enforcing physical constraints, causing inconsistent motion and limiting realism and reliability.
Method
PSIVG is a training-free, inference-time framework that uses reconstructed 3D meshes, 4D scene motion, and simulator correspondences to guide video diffusion, while TTCO locally adapts foreground-related parameters.
Results
82.3% of user-study comparisons preferred PSIVG videos as the most physically plausible, while quantitative results showed best motion controllability among the evaluated baselines.
Takeaways & Limitations
PSIVG produces videos with superior physical realism and visual quality compared to existing methods, while TTCO improves texture consistency of moving foreground objects.
Takeaways & Limitations
PSIVG is limited by MPM's handling of complex agents and articulated structures, perception quality during object reconstruction, and inherited limitations of the GwtF video model.
Abstract
from arXiv · showhide
Recent advances in diffusion-based video generation have achieved remarkable visual realism but still struggle to obey basic physical laws such as gravity, inertia, and collision. Generated objects often move inconsistently across frames, exhibit implausible dynamics, or violate physical constraints, limiting the realism and reliability of AI-generated videos. We address this gap by introducing Physical Simulator In-the-loop Video Generation (PSIVG), a novel framework that integrates a physical simulator into the video diffusion process. Starting from a template video generated by a pre-trained diffusion model, PSIVG reconstructs the 4D scene and foreground object meshes, initializes them within a physical simulator, and generates physically consistent trajectories. These simulated trajectories are then used to guide the video generator toward spatio-temporally physically coherent motion. To further improve texture consistency during object movement, we propose a Test-Time Texture Consistency Optimization (TTCO) technique that adapts text and feature embeddings based on pixel correspondences from the simulator. Comprehensive experiments demonstrate that PSIVG produces videos that better adhere to real-world physics while preserving visual quality and diversity. Project Page: https://vcai.mpi-inf.mpg.de/projects/PSIVG/
1. Introduction
PSIVG addresses the difficulty of generating videos that obey physical laws by placing a physical simulator inside the video-generation loop. It combines simulator-derived motion guidance with test-time texture optimization to improve physical and temporal consistency.
- Motivation: Modern video generators often produce objects that lack 3D consistency, vanish abruptly, or violate gravity, inertia, and collision constraints.These failures persist despite major improvements in visual quality from diffusion-based generation.
- Motivation: Denoising and reconstruction objectives provide no explicit mechanism for enforcing physical constraints or learning physics.This motivates incorporating physics-aware information directly into the diffusion process.
- PSIVG: PSIVG generates a template video, then uses a physical simulator to produce physically consistent foreground-object trajectories for guiding video generation.The template supplies the scene, camera movement, objects, and initial movements, while the simulator enforces physically consistent trajectories.
- TTCO: TTCO adapts the generation process at test time to reduce texture flickering and discoloration during object movement and rotation.The method targets texture consistency because simulator conditioning alone can leave object appearance inconsistent across frames.
- Contributions: PSIVG is presented as a training-free inference-time pipeline that reconstructs 3D meshes and 4D scene motion, while TTCO uses simulator correspondences to refine moving-object textures.The framework requires no additional training data for TTCO.
2. Related Work
Prior work spans diffusion-based video generation, physics-aware generation, and physical simulation, but these components have distinct limitations. PSIVG combines simulator-based physical accuracy with diffusion-model visual realism and texture refinement.
- Video Generation Models: Diffusion-based video generators support text-to-video and image-to-video synthesis, while controllable methods add masks, depth, edges, trajectories, or optical flow.Physical consistency remains difficult, likely because reconstruction losses inadequately capture physical principles.
- Physically Consistent Generation: Earlier physics-aware image-generation methods rely on simplified 2D rigid-body assumptions, limiting 3D understanding and temporal texture coherence.Other approaches generate videos from images and user actions, including methods focused on animating cartoons.
- Physically Consistent Generation: PSIVG reconstructs scene representation and geometry from a generated template video, using diffusion generation and TTCO to tolerate some reconstruction errors.The diffusion model refines visual details that may be inaccurate in reconstructed regions.
- Physical Simulators: Physical simulators model controlled object interactions and physically accurate dynamics, but typically lack generative capabilities and depend on predefined assets and material properties.They may also fail to capture fine-grained textures, lighting, or fluid dynamics.
- Physical Simulators: PSIVG couples a physical simulator with a video generator to combine physically grounded motion with visually compelling generated video.This pairing addresses the simulator’s rendering limitations while retaining its faithful motion physics.
3. Method
PSIVG converts a generated template video into simulator-ready scene assets, uses physical simulation to produce motion guidance, and refines video generation for physically consistent motion and foreground texture stability.
- PSIVG Pipeline: PSIVG begins with a template video and extracts 4D scene elements, object dynamics, geometries, and camera trajectories for simulation.The perception pipeline reconstructs simulator-ready assets from the generated video.
- Perception Pipeline: The perception pipeline detects and segments dynamic objects, reconstructs object-centric 3D meshes, and estimates their motion from video frames.Linear velocity uses 3D displacement over Δt, while rotational motion comes from feature matching and flow relative to matched-point centroids.
- Physical Simulation: An MPM-based simulator initializes scene geometry and physical properties within a bounded domain to generate physically accurate dynamics and visual guidance.The simulation domain is scaled and transformed so geometry, camera parameters, and physics constants map consistently into simulation coordinates.
- Physical Simulation: Simulator renders guide an optical-flow-conditioned video generator, encoding foreground trajectories, rotations, and camera movement while preserving photorealistic generation.The simulator provides faithful motion physics but its rendered RGB is not a direct replacement for generated video because of artificial style and rendering limitations.
- Texture Consistency Optimization: TTCO optimizes simulator-derived pixel correspondences at test time to reduce texture flickering and appearance drift during object movement and rotation.It adds learnable residual text embeddings and feature-wise foreground modulation, localizing adaptation while minimizing background changes.
4. Implementation Details
Template videos are generated with SD 3 followed by CogVideoX-I2V-5B or HunyuanVideo-I2V, while TTCO uses AdamW for 50 iterations and emphasizes noisier diffusion steps.
- Template videos use SD 3 for images, followed by CogVideoX-I2V-5B or HunyuanVideo-I2V with the images and prompts.
- TTCO uses AdamW with LR=2e-4 for 50 iterations and samples diffusion steps 700-1000 to guide texture generation.
5. Experiments
PSIVG is evaluated against text-to-video and controllable-generation baselines using motion controllability, video quality, qualitative comparisons, and a user study. It achieves stronger physically consistent motion, while TTCO improves texture and pixel-level consistency.
- The evaluation covers LLM-generated prompts with single- and multi-object scenes and static or dynamic camera motion.
- PSIVG consistently achieves the best SAM mIoU and Corr. Pixel MSE among the compared methods, indicating more physically consistent trajectories.
- Some baselines appear temporally stable because their frames remain nearly static, but they show poor motion diversity and physically unrealistic dynamics.
- Qualitatively, PSIVG produces physically coherent motion, whereas competing methods show floating, fading, jumping, trajectory-following, rotation, or appearance-consistency problems.
- In a 32-participant user study, PSIVG was preferred as the most physically plausible in 82.3% of comparisons.
- TTCO improves Corr. Pixel MSE, slightly improves SAM mIoU, and increases subject consistency; prompt-based optimization also outperforms test-time LoRA for preserving video quality.
6. Conclusion
PSIVG integrates physical simulation into diffusion-based video generation and uses TTCO to improve texture consistency. The method produces videos with stronger physical realism and visual quality, but remains limited by simulator coverage, reconstruction quality, and inherited generator limitations.
- PSIVG integrates physical simulation into diffusion-based video generation and enhances moving-object texture consistency through TTCO.
- The method is limited in handling complex agents, articulated structures, imperfect initial object reconstruction, and very small or thin objects.