PaperlayerView exact revision on arXiv

Source-linked AI summary

VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior

Xindi Yang, Baolu Li, Yiming Zhang, Zhenfei Yin, Lei Bai, Liqian Ma, Zhiyong Wang, Jianfei Cai, Tien-Tsin Wong, Huchuan Lu, Xu Jia

arXiv:2503.23368v3cs.CVcs.AI

TL;DR

Video diffusion models can generate realistic videos but often fail to produce physically plausible motion because they lack understanding of physical laws. VLIPP uses physics-aware VLM planning to guide video diffusion generation; the authors report superior benchmark performance, including average gains on PhyGenBench over the best T2V and I2V methods.

  • Problem

    Video diffusion models can appear realistic but struggle to understand physical laws and generate physically plausible motion.

    Source: §1 · p. 1

  • Method

    VLIPP uses a VLM for coarse physics-aware motion planning and conditions a video diffusion model on the resulting trajectory to synthesize finer motion.

    Source: §1 · p. 2, §3.1 · p. 5, §3.2 · p. 5

  • Results

    VLIPP consistently achieves superior performance across all benchmarks; on PhyGenBench, it outperforms the best T2V method by an average of 15.3% and the best I2V method by 11.1%.

    Source: §4 · p. 6, §4.3 · p. 7

  • Takeaways & Limitations

    The results demonstrate the effectiveness of VLIPP compared with existing approaches for generating physically plausible videos.

    Source: §5 · p. 9

    The framework cannot model events not represented by image-space bounding-box trajectories, lacks 3D spatial perception, and is vulnerable to optical-flow noise for small objects.

    Source: §4.7 · p. 8

Abstract

from arXiv · show

Video diffusion models (VDMs) have advanced significantly in recent years, enabling the generation of highly realistic videos and drawing the attention of the community in their potential as world simulators. However, despite their capabilities, VDMs often fail to produce physically plausible videos due to an inherent lack of understanding of physics, resulting in incorrect dynamics and event sequences. To address this limitation, we propose a novel two-stage image-to-video generation framework that explicitly incorporates physics with vision and language informed physical prior. In the first stage, we employ a Vision Language Model (VLM) as a coarse-grained motion planner, integrating chain-of-thought and physics-aware reasoning to predict a rough motion trajectories/changes that approximate real-world physical dynamics while ensuring the inter-frame consistency. In the second stage, we use the predicted motion trajectories/changes to guide the video generation of a VDM. As the predicted motion trajectories/changes are rough, noise is added during inference to provide freedom to the VDM in generating motion with more fine details. Extensive experimental results demonstrate that our framework can produce physically plausible motion, and comparative evaluations highlight the notable superiority of our approach over existing methods. More video results are available on our Project Page: https://madaoer.github.io/projects/physically_plausible_video_generation.

1. Introduction

Video diffusion models can produce realistic videos but struggle to represent physically plausible motion. VLIPP uses vision-language reasoning to plan coarse motion and a video diffusion model to synthesize finer motion.

1. Introduction

  • VDMs produce realistic videos but still struggle to understand physical laws and generate motion that follows them.Commercial closed-source models also struggle to generate videos conforming to physical laws.Source: §1 · p. 1
  • Text descriptions alone provide an ambiguous link to real-world motion, while graphics-engine approaches are computationally costly and limited to simulatable effects.The paper also notes that scalable physical training data is difficult to gather and explicitly modeling equations for every motion is impractical.Source: §1 · p. 2
  • VLIPP uses a VLM for coarse, physics-aware motion planning and an image-to-video diffusion model for fine motion synthesis.Noise is added to the planned path during training and inference so the diffusion model can synthesize finer details such as speed, acceleration, and vibration.Source: §1 · p. 2
  • The authors report superior performance over contemporary competitors on two video-physics benchmarks and describe experiments and user studies addressing effectiveness and generalization.Their stated contributions include chain-of-thought and physics-aware VLM reasoning, plus random noise injection during video generation.Source: §1 · p. 2

2. Related Work

Prior approaches use physical simulators, motion controls, or language-model planning, but the paper identifies limits in generality, physical plausibility, or physical-event interaction. VLIPP combines physics-aware VLM planning with video diffusion generation.

2.1. Physically Plausible Visual Content Generation

  • Graphics-based methods simulate physical effects but face limits in expressiveness, efficiency, generalizability, and parameter tuning; prior VDM methods target specific motion types rather than a generalizable approach.The cited discussion also notes that graphics simulators require substantial expertise.Source: §2.1 · p. 2

2.2. Motion Controllable Video Generation

  • Bounding-box, point-trajectory, and camera controls guide motion, but these approaches often overlook physical plausibility.VLIPP instead incorporates physics as conditions for video diffusion models.Source: §2.2 · p. 3

2.3. Generation based on VLMs Planning

  • Prior language-model planning methods support layout control, temporal consistency, or text-command control, yet do not address physical interactions such as collisions, falls, and melting.VLIPP uses VLMs as coarse image-space motion planners with physics-aware reasoning and chain-of-thought prompting.Source: §2.3 · p. 3

3. Method

VLIPP turns an image and event description into physics-compliant motion guidance, then uses video diffusion to synthesize detailed dynamics. Its VLM plans image-space object trajectories, which are converted into optical-flow-based structured noise to guide generation.

3. Method

  • VLIPP takes an image and event description as input and aims to synthesize video guided by physical principles and real-world dynamics.Its pipeline separates coarse motion guidance from detailed video synthesis.Source: §3 · p. 3

3.1. VLM as a Coarse-Level Motion Planner

  • The VLM identifies applicable physical laws and predicts future object bounding boxes through stepwise reasoning about scene context, interactions, and motion.It predicts changes for 12 future frames, which are linearly interpolated to 49 frames for the subsequent generation process.Source: §3.1 · pp. 4–5

3.2. VDM Serves as a Fine-Level Motion Synthesizer

  • The planned bounding-box trajectory animates a synthetic video, from which RAFT optical flow is extracted as structured noise for the video diffusion model.The synthetic sequence moves and resizes objects according to the predicted boxes, with the background inpainted after foreground removal.Source: §3.2 · p. 5
  • Noise is injected during inference so the diffusion model can depart from the coarse trajectory when needed to generate finer motion.The paper sets γ to 0.4 for even frame indices and 0.6 for odd frame indices.Source: §3.2 · p. 5

4. Empirical Analysis and Disscusion

The evaluation compares the framework with existing methods on PhyGenBench and Physics-IQ, reporting superior benchmark performance alongside qualitative, ablation, and user-study analyses. It also identifies constraints from image-space motion representation, absent 3D perception, and small-object flow noise.

4.3. Quantitative Evaluation

  • PhyGenBench gains average 15.3% vs the best T2V and 11.1% vs the best I2V; Physics-IQ gains are 22.2% in Solid Mechanics and 9.2% in Fluid Dynamics vs second-best models.Source: Table 1 · p. 6, §4.3 · p. 7, Table 2 · p. 7

4.1. Implementation Details

  • Experiments use ChatGPT-4o as the coarse-level motion planner and Go-with-the-Flow as the fine-level motion synthesizer.Videos are generated at 720 × 480 resolution and 49 frames unless otherwise specified.Source: §4.1 · p. 6

4.2. Benchmarks and Models

  • Evaluation uses PhyGenBench’s 160 prompts covering 27 physical phenomena, adapted to image-to-video with FLUX-generated first frames and GPT-4o realism assessment, plus Physics-IQ’s 396 videos across 66 scenarios.Comparisons include text-to-video, image-to-video, and motion-controllable models; LPIPS, FID, and FVD do not account for physical realism.Source: §4.2 · pp. 6–7

4.3. Quantitative Evaluation

  • On PhyGenBench, the framework beats the best I2V method by 5.7% in Mechanics, 17.6% in Thermal, and 35.8% in Material.The paper links these domains’ larger motion, volume, or shape changes to the framework’s ability to reason about bounding-box sequences.Source: §4.3 · p. 7

4.4. Qualitative Evaluation

  • Qualitatively, the framework is reported to produce the most physically realistic videos; baselines show flawed ball falls, miss coordinated water-level changes, or fail to depict ball collisions.Source: Figure 4 · p. 6, §4.4 · p. 7, Figure 5 · p. 8

4.5. Ablation Study

  • Ablations remove the planner, in-context information, CoT, or both; planner removal yields nearly random results, and CoT matters more than in-context information, which is less effective against hallucination errors.Source: §4.5 · pp. 7–8, Table 3 · p. 8

4.6. User Study

  • In a 2AFC study with 50 participants, respondents strongly preferred the framework’s videos over competitors’ for physical realism.Source: §4.6 · p. 8, Table 4 · p. 8

4.7. Limitations

  • The approach cannot model intrinsic state changes that image-space bounding-box trajectories cannot represent, lacks 3D spatial perception, and is vulnerable to noisy small-object optical flow.The paper gives solid fragmentation and gas solidification as examples of unrepresented events.Source: §4.7 · p. 8

5. Conclusion

VLIPP addresses video diffusion models’ difficulty adhering to physical laws with a two-stage framework that incorporates physical priors. The paper reports effectiveness compared with existing approaches.

5. Conclusion

  • VDMs can achieve empirical success yet remain unable to generate physically plausible videos because they lack understanding of physical laws.Source: §5 · p. 9
  • VLIPP incorporates physical laws into video diffusion models through a vision- and language-informed physical prior in a two-stage framework.Source: §5 · p. 9
  • The paper reports that VLIPP is effective compared with existing approaches.Source: §5 · p. 9

A. Coarse-Level Motion Planning Details

The appendix gives generation settings for the compared video models and describes the reasoning template used for stage-one coarse-level motion planning.

A. Coarse-Level Motion Planning Details

B. Experiment Details

The experiments describe the two benchmarks and how they evaluate physical plausibility, then specify shared initial-frame and key-frame sampling procedures for model comparisons.

B. Experiment Details

  • PhyGenBench contains 160 prompts across Mechanics, Optics, Thermal, and Material, covering 27 physical-law types, 165 objects, and 42 actions.Source: B. Experiment Details · p. 12
  • PhyGenBench evaluates semantic alignment and physical commonsense alignment, including whether physical phenomena occur in the correct order and whether videos appear natural.Source: B. Experiment Details · p. 12
  • Physics-IQ covers five physical domains and evaluates physical comprehension and visual authenticity using generated-versus-real video comparisons.Physical comprehension uses timing, location, and action frequency to calculate pixel-level mean squared error; visual authenticity asks a VLM to identify the real video from a randomized pair.Source: B. Experiment Details · p. 12
  • For fair comparison, all image-to-video models received the same initial frame, and key frames were sampled at a uniform ratio despite different video lengths.Source: B. Experiment Details · p. 12

C. More Qualitative Results

The qualitative examples span fluid dynamics, thermodynamics, and optics, while the appendix also presents physics-aware reasoning templates for rigid-body motion and fluid and thermal phenomena.

C. More Qualitative Results

Loading 2503.23368v3…