PaperlayerView exact revision on arXiv

Source-linked AI summary

PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation

Qiyao Xue, Xiangyu Yin, Boyuan Yang, Wei Gao

arXiv:2412.00596v2cs.CVcs.AI

TL;DR

Current T2V models have difficulty adhering to physical rules, and existing approaches have limited generalizability to out-of-distribution domains. PhyT2V uses LLM-guided chain-of-thought and step-back reasoning to refine prompts iteratively. It improves physical realism over baseline T2V models and outperforms state-of-the-art prompt enhancers.

  • Problem

    Current T2V models struggle to adhere to real-world physical rules, while existing solutions have limited generalizability to out-of-distribution domains.

    Source: §1 · pp. 1–2

  • Method

    PhyT2V uses LLM-guided chain-of-thought and step-back reasoning to iteratively refine T2V prompts using physical rules and video-caption mismatch.

    Source: §1 · p. 2

  • Results

    PhyT2V achieves a 2.3x enhancement in physical realism compared to baseline T2V models and outperforms state-of-the-art T2V prompt enhancers by 35%.

    Source: §5 · p. 8

  • Takeaways & Limitations

    PhyT2V is designed to improve existing T2V models’ generalization to out-of-distribution domains without additional retraining or conditions.

    Source: §5 · p. 8

    For complicated or subtle scenes, PhyT2V may not reach physical realism after multiple iterations and stops when improvement converges.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Text-to-video (T2V) generation has been recently enabled by transformer-based diffusion models, but current T2V models lack capabilities in adhering to the real-world common knowledge and physical rules, due to their limited understanding of physical realism and deficiency in temporal modeling. Existing solutions are either data-driven or require extra model inputs, but cannot be generalizable to out-of-distribution domains. In this paper, we present PhyT2V, a new data-independent T2V technique that expands the current T2V model's capability of video generation to out-of-distribution domains, by enabling chain-of-thought and step-back reasoning in T2V prompting. Our experiments show that PhyT2V improves existing T2V models' adherence to real-world physical rules by 2.3x, and achieves 35% improvement compared to T2V prompt enhancers. The source codes are available at: https://github.com/pittisl/PhyT2V.

1. Introduction

Current T2V models struggle to maintain physical realism and temporal consistency, particularly in out-of-distribution settings where existing approaches have limited generalizability. PhyT2V uses LLM-guided reasoning to iteratively refine prompts and videos, improving physical-rule adherence without retraining.

1. Introduction

  • Current T2V models struggle with physical rules and frame-to-frame consistency, limiting their use for real-world simulation.Challenges include object interactions and temporal issues such as flickering, inconsistent motion, and object deformation.Source: §1 · pp. 1–2
  • Data-driven and physics-injection approaches rely on dataset coverage or fixed physical categories, limiting generalizability to out-of-distribution domains.The paper notes that real-world knowledge and physical rules are not explicitly embedded in the generation process of data-driven approaches.Source: §1 · p. 2
  • PhyT2V embeds real-world knowledge and physical rules in prompts using chain-of-thought and step-back reasoning, rather than expanding datasets or changing model architecture.Its LLM analyzes the prompt for objects and physical rules, evaluates video-caption mismatch, and refines the prompt using those findings.Source: §1 · p. 2
  • The prompt and generated video are refined over multiple rounds until video quality is satisfactory or its improvement converges.The evaluation also describes PhyT2V as data independent, automated, and requiring minimal engineering and manual effort.Source: §1 · p. 2, Figure 2 · p. 2
  • Up to 2.3x, PhyT2V improves existing T2V models’ adherence to real-world physical rules by refining prompts without retraining or auxiliary model inputs.The evaluation used multiple state-of-the-art T2V models and two major prompt datasets covering seven real-world domains.Source: §1 · p. 2

2. Related Work and Motivation

Prior T2V progress has not resolved failures on unfamiliar prompts, while prompt refinement and LLM reasoning offer possible ways to address them. PhyT2V’s motivation combines iterative text-only feedback with physical-rule reasoning to refine generation.

2.1. T2V Generation Models

  • Transformer-based T2V models improved temporal consistency over early approaches, but out-of-distribution prompts still produce physical illusions and artifacts.Prior work also found that dataset, architecture, or adaptation changes may be prompt-specific and lack generalizability [43] [45].Source: §2.1 · pp. 2–3
  • Detailed prompt refinement can improve out-of-distribution video quality, but complex interactions expose the need for feedback that does not rely on extra modalities.Extra inputs such as sampled frames, depth maps, or scribbles add computing overhead and are described as not generalizable [44] [52].Source: §2.1 · p. 3

2.2. Using LLM in T2V Generation

  • Existing LLM prompt methods can yield visually coherent but physically inaccurate videos, and static or simple refinements may miss nuanced physical changes.CoT breaks complex prompts into stepwise tasks, while step-back prompting uses a higher-level question to address intermediate-step errors and vagueness.Source: §2.2 · pp. 3–4
  • Because T2V requires cross-modal alignment, PhyT2V uses video captions so CoT and step-back reasoning can operate in the text domain.The passage characterizes CoT as suited to linear, single-modality processing rather than multimodal synchronization and interaction.Source: §2.2 · p. 4

3. Method

PhyT2V iteratively refines prompts by analyzing physical rules and mismatches between generated videos and prompts. Its rounds use local chain-of-thought reasoning for subproblems and global step-back reasoning to guide refinement.

3. Method

  • Each round analyzes physical rules and prompt-video mismatches in parallel, then combines them to refine the prompt using local CoT and global step-back reasoning.Unlike prior enhancers relying solely on templates or embeddings, PhyT2V incorporates semantic mismatch information and physical rules into refinement.Source: §3 · pp. 4–5

3.1. Prompting in Parallel Subproblems for Local CoT Reasoning

  • Parallel-subproblem prompts use task instructions, QA exemplars, and current-task input; a step-by-step trigger activates local CoT reasoning.The exemplars guide reasoning without fine-tuning a separate LLM checkpoint for each new task.Source: §3.1 · p. 5

3.2. Prompting in the Final Subproblem for Global Step-Back Reasoning

  • The final subproblem uses physical-rule and mismatch outputs as high-level concepts, plus evaluator feedback, to guide global step-back reasoning.The trigger is omitted from this long, reasoning-rich prompt to avoid unrelated information in the final answer.Source: §3.2 · p. 5

3.3. The Stopping Condition

  • For complicated or subtle scenes, physical realism may remain out of reach; refinement stops when iterations converge and video-quality improvement becomes small.The normal stopping condition is evaluator-measured satisfactory video quality.Source: §3.3 · p. 6

4. Experiments

The experiments evaluate PhyT2V across T2V models, benchmarks, physical-rule domains, and reasoning-step ablations. They report improved adherence, stronger performance than prompt enhancers, and contributions from both reasoning steps.

4.1. Improvement of the Generated Video Quality

  • On CogVideoX-2B, PC improvement reaches 2.2x and SA improvement reaches 2.3x; other models show 1.3x–1.9x gains over vanilla T2V.The comparisons cover VideoPhy and PhyGenBench across iterative refinement rounds.Source: Table 1 · p. 6, Table 2 · p. 6, §4.1 · p. 6
  • PhyT2V outperforms existing prompt enhancers by at least 35% on CogVideoX-5B and OpenSora.The comparison uses the VideoPhy and PhyGenBench datasets.Source: Table 3 · p. 7, Table 4 · p. 7, §4.1 · p. 7
  • Most video-quality improvement occurs in the first two refinement rounds, with little observed improvement by round four; the authors consider 3–4 rounds sufficient.VBench results also improve in many dimensions across rounds, particularly Video-Condition Consistency.Source: §4.1 · p. 7, Figure 7 · p. 7

4.2. Different Domains of Physical Rules

  • Across physical-rule domains, PhyT2V improves most categories, with larger gains for subtle interactions such as fluids and thermal-related scene changes.The domain analysis uses CogVideoX-5B and ChatGPT 4.Source: §4.2 · p. 7
  • Examples show refined prompts helping videos capture key objects and interactions in out-of-distribution cases, including juice slowly diffusing on a counter.The examples are drawn from VideoPhy and PhyGenBench prompts.Source: §4.2 · p. 7, Figure 8 · p. 7, Figure 9 · p. 7

4.3. Ablation Study

  • Physical-rule reasoning (Step 1) adds coherent descriptions of object motion, state, and deformation grounded in physical laws, while avoiding unnecessary or vague prompt text.The ablation associates this step with a higher PC score.Source: §4.3 · p. 8, Figure 10 · p. 8
  • Mismatch reasoning (Step 2) identifies discrepancies between expected and generated video details and corrects overlooked features, improving fidelity and relevance toward a higher SA score.The ablation describes this step as focusing attention on the principal object.Source: §4.3 · p. 8
  • The ablation concludes that both reasoning steps are integral to robust, semantically aligned refinement of prompts.The study evaluates each step by removing it from the refinement process.Source: §4.3 · pp. 7–8

5. Conclusion

PhyT2V is a data-independent framework for improving existing T2V models’ generalization to out-of-distribution domains. It uses chain-of-thought reasoning and step-back prompting to refine prompts, with reported gains in physical realism and over prompt enhancers.

5. Conclusion

  • PhyT2V refines T2V prompts using chain-of-thought reasoning and step-back prompting without additional model retraining or conditions.The framework targets improved generalization to out-of-distribution domains.Source: §5 · p. 8
  • PhyT2V achieves a 2.3x enhancement in physical realism over baseline T2V models and outperforms state-of-the-art T2V prompt enhancers by 35%.Source: §5 · p. 8

A. Details of Evaluation Setup

The evaluation uses four T2V models, two physical-commonsense prompt benchmarks, and metrics for physical and semantic adherence. It compares prompt-enhancer baselines and specifies generation and LLM-inference settings.

A. Details of Evaluation Setup

  • The evaluation uses four T2V models, VideoPhy and PhyGenBench prompts, and VideoCon-Physics metrics for physical common sense (PC) and semantic adherence (SA).It also evaluates generated videos with VBench metrics.Source: A. Details of Evaluation Setup · p. 11
  • PhyT2V uses Promptist and GPT-4o [47] as prompt enhancers, with baseline model hyperparameters matched for consistency; generated videos are 6 seconds long at 8 FPS and 720 × 480 resolution.Source: A. Details of Evaluation Setup · p. 11
  • For LLM inference, refined prompts are limited to 120 words, outputs use JSON format, and results from each step are stored by row in a CSV file.The first two step outputs are embedded in the third-step prompt template.Source: A. Details of Evaluation Setup · p. 11

B.1. Examples of Generated Videos in Different Categories of Physical Rules

The appendix lists additional generated-video comparisons across physical-rule categories and examples of PhyT2V prompt-refinement details for VideoPhy and PhyGenBench.

B.1. Examples of Generated Videos in Different Categories of Physical Rules

  • Additional CogVideo-5b video comparisons cover fluid-to-fluid, solid-to-fluid, and solid-to-solid cases on VideoPhy, plus force, optics, and thermal cases on PhyGenBench.Source: §B.1 · p. 11

B.2. Refinement process details

  • Prompt-refinement examples for CogVideo-5b cover the same three VideoPhy interaction categories and the force, optics, and thermal categories of PhyGenBench.Source: §B.2 · pp. 11–12

C.1. Model size

The appendix reports that PhyT2V can provide more physics-grounded generation capability on a larger CogVideo model. It also examines the role and in-context-example components of the prompt template.

C.1. Model size

  • PhyT2V can unleash more physics-grounded video-generation capability on the larger CogVideo-5b model than on CogVideo-2b.The comparison is described in Figure 24.Source: §C.1 · p. 12

C.2. Prompt template component

  • Removing the role-indicator sentence reduces output precision, while removing in-context examples prevents GPT from producing the expected format.Source: §C.2 · p. 12

D. Failure cases

PhyT2V may remain ineffective when limitations of the underlying T2V model constrain temporal consistency, extrapolation to underrepresented phenomena, or precise human-body generation.

  • Many T2V models still flicker or produce inconsistent object trajectories because they lack long-term temporal-coherence mechanisms.Even prompts emphasizing smooth transitions or continuous motion may not overcome these model limitations.Source: D. Failure cases · p. 12
  • Rare or complex physical phenomena are often underrepresented in training data, and T2V models may fail to extrapolate even with highly specific prompts.Source: D. Failure cases · p. 12
  • After several PhyT2V refinement rounds, some generations still fail to render human bodies precisely, especially hands.Source: D. Failure cases · p. 12
Loading 2412.00596v2…