Source-linked AI summary
Physics-Aware Video Generation via Agentic Planning and Graph-Guided Optimization
Minh-Loi Nguyen, Xuan-Vu Le, Thanh-Toan Do, Tam V. Nguyen, Minh-Triet Tran, Trung-Nghia Le
TL;DR
VDMs produce realistic videos but lack explicit physical understanding, leading to structurally hallucinated and causally illogical dynamics, while existing training-free guidance can disrupt backgrounds and miss complex state changes. PhysPlan combines iterative VLM physical planning with object-centric, dynamically scheduled test-time optimization. It significantly outperforms foundational and controllable VDM baselines on PhyGenBench and Physics-IQ for physical causality, structural integrity, and temporal coherence.
Problem
VDMs lack explicit physical understanding and often produce structurally hallucinated, causally illogical dynamics, while existing training-free methods use global gradients and rigid schedules.
Method
PhysPlan combines iterative VLM-based Chain-of-Visual-Thought planning with Object-Centric Gradient Routing and Kinetic Intensity Profiling for training-free video guidance.
Results
PhysPlan significantly outperforms foundational and controllable VDMs on PhyGenBench and Physics-IQ for physical causality, structural integrity, and temporal coherence.
Takeaways & Limitations
The framework connects VLM physical reasoning with VDM rendering to enforce more physically coherent video generation without retraining the base model.
Takeaways & Limitations
Inference is costly and currently supports only 3–5 keyframes within feasible VRAM limits, making dense frame-level supervision impractical.
Abstract
from arXiv · showhide
Video diffusion models (VDMs) have demonstrated remarkable capabilities in synthesizing high-fidelity, photorealistic video content. However, they fundamentally lack an intrinsic understanding of physical laws and frequently produce visually appealing but causally illogical sequences characterized by structural hallucinations and physically implausible dynamics. Injecting physical awareness via training-free test-time optimization is a promising alternative, yet existing methods rely on global gradient updates and rigid scheduling heuristics that inadvertently corrupt passive backgrounds and fail to model complex dynamic state changes. To address this, we propose PhysPlan, a novel training-free guidance framework that shifts the paradigm from stochastic visual interpolation to agentic physics simulation. First, a VLM operates as an iterative cognitive simulator, decomposing multimodal inputs into a Chain-of-Visual-Thought to create a multimodal representation of kinematic trajectories and 3D depth geometries. Second, these signals drives an object-centric test-time optimization. Unlike prior training-free methods that rely on global gradients and rigid scheduling heuristics, PhysPlan introduces Object-Centric Gradient Routing to isolate kinematic modifications and completely lock the passive environment. Furthermore, our Kinetic Intensity Profiling dynamically parameterizes framework hyperparameters to accommodate the varying severity of physical deformations. Extensive evaluations on the PhyGenBench and Physics-IQ benchmarks demonstrate that PhysPlan significantly outperforms both foundational and controllable VDM baselines, offering a promising approach for improving the physical understanding of video generation.
1 Introduction
PhysPlan addresses physically implausible VDM dynamics with a two-stage, VLM-guided, training-free framework that routes localized guidance and adapts optimization to deformation severity. It reports stronger physical causality, structural integrity, and temporal coherence than existing VDM baselines on PhyGenBench and Physics-IQ.
- VDMs generate realistic videos but often violate physical dynamics through structural hallucinations, illogical object changes, and failures of object permanence.
- Existing training-free guidance uses global gradients and rigid schedules, perturbing passive backgrounds and failing to capture diverse physical state changes.
- PhysPlan uses a VLM to iteratively produce a Chain-of-Visual-Thought and multimodal physical conditions for guiding reverse diffusion.
- Object-Centric Gradient Routing locks unprompted backgrounds, while Kinetic Intensity Profiling dynamically sets guidance density and learning rates according to deformation severity.
- PhysPlan significantly outperforms foundational and controllable VDMs on PhyGenBench and Physics-IQ for physical causality, structural integrity, and temporal coherence.
2 Related Work
Related work frames VDMs as visually strong but physically limited, while physics-aware and training-free methods address this gap with important trade-offs. PhysPlan extends VLM-guided generation by routing multimodal conditions to preserve spatial and depth consistency.
- Foundational VDMs achieve frame consistency and photorealistic scenes but primarily learn visual patterns rather than underlying physical rules.
- Existing physics-aware approaches use prompt refinement, physics engines, or VLM planning, but face text-to-motion ambiguity, engineering overhead, or degraded visual quality.
- PhysPlan builds on VLM-guided generation by constructing multimodal conditions that mathematically route gradients for spatial and depth consistency.
- Training-free guidance avoids retraining by manipulating latent trajectories, yet global gradients corrupt static backgrounds and rigid schedules treat complex phenomena like simple translations.
3 Proposed Method
PhysPlan combines agentic physical planning with training-free, object-centric guidance to generate videos that follow causal dynamics while preserving passive environments. A VLM constructs visual, geometric, and kinetic conditions, which dynamically control localized diffusion optimization.
- 3 Proposed Method: PhysPlan divides generation into agentic physical planning and training-free physics guidance, using a VLM for structured conditions and a VDM for localized motion synthesis.The VLM plans physical states and guidance parameters, while the VDM provides fine-grained rendering during reverse diffusion.
- 3 Proposed Method: The VLM builds a scene graph of entities, relationships, and forces, then predicts nonuniform physical events with temporal anchors and visual editing instructions.Each planned event updates the graph and timeline, and an instruction-guided image editor renders the corresponding intermediate keyframe.
- 3 Proposed Method: The framework grounds planned keyframes with subject masks and monocular depth maps, isolating active objects while encoding 3D spatial layout and volume boundaries.These conditions provide spatial and depth constraints for latent optimization.
- 3 Proposed Method: Kinetic Intensity Profiling assigns each frame a physical-severity score that dynamically controls learning rates and guidance-step density during volatile events.Higher-intensity frames receive stronger updates and more localized iterations, adapting optimization to collisions, phase changes, and other complex dynamics.
- 3 Proposed Method: Latent Slicing projects only selected temporal latent slices onto the clean data manifold, reducing the cost of evaluating spatial and depth constraints.This avoids fully decoding the spatiotemporal latent at every inference step while preserving a clean canvas for multimodal objectives.
- 3 Proposed Method: Object-Centric Gradient Routing applies optimization only to active subjects, addressing the background drift and flickering caused by global gradient updates.The method explicitly targets the limitations of vanilla Frame Guidance and its static Video Latent Optimization schedule.
4 Experiments
PhysPlan is evaluated on two physical-video benchmarks, perceptual quality, qualitative scenarios, component ablations, and human preferences. Across these evaluations, it improves physical plausibility and temporal coherence while preserving visual quality through adaptive scheduling and localized optimization.
- Quantitative Evaluation: 0.59 overall on PhyGenBench exceeds CogVideoX-I2V-5B (0.52) and Frame Guidance (0.51), with PhysPlan highest across Mechanics, Optics, Thermal, and Material.The reported domain scores are 0.55, 0.73, 0.58, and 0.51, respectively.
- Perceptual Quality Evaluation: 500.2 FVD on PhyGenBench and 495.6 on Physics-IQ substantially outperform baselines, while FID remains comparable to CogVideoX and Frame Guidance.The results associate improved FVD with reduced flickering, texture crawling, and background disruption from Object-Centric Gradient Routing.
- Qualitative Evaluation: PhysPlan produces more physically realistic and temporally coherent dropping-object sequences, preserving passive backgrounds while baselines show erratic trajectories and floating objects.The comparison targets accurate kinematics and complex structural state changes.
- Ablation Study: Removing Kinetic Intensity Profiling or Object-Centric Gradient Routing causes the largest ablation degradations, while removing either objective loss also consistently reduces performance.Uniform hyperparameters struggle with heterogeneous motion, whereas object-aware routing reduces interference; the masked spatial trajectory loss has a slightly larger effect than the depth constraint.
- User Study: PhysPlan is consistently preferred across Physical Plausibility, Frame Quality, and Temporal Smoothness in the user study.Frame Guidance is close in Frame Quality (59% vs. 60%) but trails in Temporal Smoothness (45%) and Physical Plausibility (44%).
5 Limitations
PhysPlan’s limitations arise from autoregressive visual planning, sparse structural guidance, and the computational cost of localized test-time optimization. These constraints affect long-form scaling, motion naturalness, and feasible inference length.
- 5.1 Limitations of Autoregressive Visual Planning: PhysPlan inherits constraints from the autoregressive nature of current vision-language models and image-editing APIs.These constraints underlie the pipeline’s dependence on sequential visual-state updates and external multimodal services.
- 5.1 Limitations of Autoregressive Visual Planning: Autoregressive keyframe planning can accumulate texture degradation and color shifts, occasionally undermining photorealism in later frames.Each generated keyframe becomes the visual baseline for the next image-generation step.
- 5.1 Limitations of Autoregressive Visual Planning: Conversation-history retention causes token consumption to scale quadratically with generated keyframes, while proprietary API dependence constrains highly complex, long-form video generation.The practical overhead is modest for typically 3–5 keyframes but limits scaling beyond that regime.
- 5.2 Limitations of Sparse Inference-Time Tuning: Sparse geometric guidance creates a rigidity–physics trade-off: strong static matching reduces natural motion, whereas weak guidance lets the generator override intended physics.The trade-off follows from using segmentation masks and depth maps as discrete structural constraints.
- 5.2 Limitations of Sparse Inference-Time Tuning: 3–5 keyframes are currently feasible within VRAM limits, making dense frame-level physics supervision computationally impractical.Localized gradients require decoding noisy latents through the spatial-temporal VAE at each denoising step, so memory and latency scale with constraint-frame count.
6 Conclusion
PhysPlan is a training-free framework that combines agentic physical planning with multimodal test-time optimization for video diffusion models. Its object-centric routing and kinetic profiling target background preservation and heterogeneous physical deformations, while experiments show improvements in physical causality, structural integrity, and temporal coherence.
- 6 Conclusion: PhysPlan significantly outperforms foundational and controllable video diffusion models on PhyGenBench and Physics-IQ for physical causality, structural integrity, and temporal coherence.The conclusion presents this as the framework’s benchmark-level outcome.
- 6 Conclusion: Object-Centric Gradient Routing eliminates background drift, while Kinetic Intensity Profiling addresses severe deformations that defeat static heuristic schedules.These mechanisms connect localized optimization and dynamic parameterization to the framework’s stated physical-guidance goals.
- 6 Conclusion: PhysPlan bridges agentic physical planning with test-time optimization under multimodal constraints to equip video diffusion models with VLM-based physical reasoning.The framework shifts generation toward enforcing real-world dynamics rather than relying only on stochastic visual motion.
- 6 Conclusion: Iterative gradient backpropagation and VLM routing increase inference latency and memory overhead, motivating future feed-forward adapter distillation.The proposed future direction aims to bypass inference-time backpropagation for more efficient scaling.