PaperlayerView exact revision on arXiv

Source-linked AI summary

PhyEvolve: Multi-Agent State-Evolution Planning for Physics-Aware Video Generation

Yang-Hao Zhou, Lixing Tan, Zhiqi Li, Guang Yang, Xiao Yang

arXiv:2610.30717v1cs.CV

TL;DR

Existing video generators can struggle to represent the physical evolution implicit in prompts, particularly across unobserved intervals. PhyEvolve uses three agents to plan causal state changes and incorporate them into replacement prompts. Across four primary benchmark metrics, it gains 7.9 percentage points over NEWTON, the prior agentic baseline with the highest mean score.

  • Problem

    Existing video generators remain unreliable at making physical consequences consistent with user-described conditions and interactions.

    Source: §1 · p. 1

  • Method

    PhyEvolve uses a Director, Physical Evolver, and Composer to ground events, plan causal state transitions, and carry them into a reviewed replacement prompt.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 4

  • Results

    Across four primary benchmark metrics, PhyEvolve gains 7.9 percentage points over NEWTON, the prior agentic baseline with the highest mean score.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With H3 as the fixed generator, benchmark results support state-evolution planning as a useful way to condition existing video generators without retraining or model modification.

    Source: §5 · p. 8

    Evaluation uses short benchmark clips, and PhyEvolve’s performance on longer video sequences has not been established.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent advances in text-to-video generation have substantially improved visual fidelity, yet existing models still struggle to understand the physical evolution implicit in user prompts, making it difficult to generate videos with continuous physical state evolution. Here, we propose PhyEvolve, a multi-agent framework that enhances natural-language prompts through structured state-evolution planning. PhyEvolve consists of three agents: a Director, a Physical Evolver, and a Composer. Given a user prompt and an optional first frame, the Director grounds the scene and disentangles physical events from observation changes. The Physical Evolver uses scene evidence to identify physical mechanisms then constructs causally linked state transitions that unfold throughout the video, whether visible or not. The Composer utilizes these transitions to enhance the original prompts. Without retraining or model modification, PhyEvolve works directly with existing generators through prompt replacement. Extensive experiments on four major benchmarks show that PhyEvolve achieves state-of-the-art performance, outperforming the cutting-edge prior agentic baseline by 7.9 percentage points on average. These results demonstrate that explicit causal state-evolution planning improves physical consistency and state continuity in generated videos through natural-language prompt enhancement.

1 Introduction

Existing video generators often fail to make implicit physical mechanisms, conditions, and state changes explicit, especially when observation changes obscure ongoing events. PhyEvolve plans causal state evolution and carries it into replacement prompts, improving benchmark performance without modifying the generator.

1 Introduction

  • Brief prompts often leave mechanisms, activation conditions, and resulting state changes implicit, motivating enhancement that makes physical consequences explicit while preserving requested objects, actions, and interactions.The framework is evaluated on four physical-video benchmarks using a frozen H3 backend.Source: §1 · pp. 1, 3
  • Camera movement changes visibility, while valve closure changes the conditions governing water flow; the sprinkler’s state must evolve even while it is out of view.The final prompt must state the inferred outcome, not merely describe the camera’s return.Source: §1 · p. 1
  • The Director grounds scene entities and events, the Physical Evolver infers dependent transitions, and the Composer checks and revises a complete replacement prompt against those records.The prompt is passed to the generator with an optional initial frame, without retraining or modifying the generator.Source: §1 · pp. 1–2, Figure 2 · p. 3
  • Averaged across four primary benchmark metrics, PhyEvolve gains 7.9 percentage points over NEWTON, the prior agentic baseline with the highest mean score.With Minimax-H3 as the generation backend, it also outperforms direct generation and the H3-based versions of NEWTON and Co-Director on each benchmark’s primary metric.Source: §1 · p. 2
  • PhyEvolve frames prompt enhancement as semantic completion through structured state-evolution planning, distinguishing physical effects from observation changes and accounting for unobserved intervals.Explicit records connect scene evidence, mechanism conditions, and dependent state transitions, and help check whether the final prompt preserves planned changes.Source: §1 · pp. 1–2

2 Related Work

Related work spans physical dynamics, video world models, memory for hidden or interrupted evolution, and varied physical-generation evaluations. Agentic video systems coordinate planning and generative tools; physics-oriented approaches include tool-based planning and simulation, while PhyEvolve centers causal state-evolution plans expressed in prompts.

2.1 Physics-Aware Video Generation

  • Physics-aware video research spans intuitive and latent dynamics, world models, memory for hidden or interrupted evolution, and evaluations of physical commonsense and out-of-sight state evolution (Hafner et al., 2019; Xu et al., 2026; Ma et al., 2026).PhyEvolve focuses on explicitly planning state changes across visible and unobserved intervals.Source: §2.1 · p. 3

2.2 Agentic Video Generation

  • Agentic video work coordinates language-model planning, tools, scene layouts, storytelling, and refinement; NEWTON uses physics-aware tools and verification, while MoReGen combines multi-agent reasoning with simulation and rendering (Feng et al., 2026; Bai et al., 2026).PhyEvolve instead centers causal state-evolution plans whose consequences are expressed in a complete prompt, with review and local repair before generation.Source: §2.2 · pp. 3–4

3 Method

PhyEvolve uses three cooperating agents to ground a scene, plan causally linked state evolution, and carry reviewed physical changes into a replacement prompt.

3.1 Overview

  • The workflow grounds its plan in a prompt and optional initial frame, reviews and locally revises agent outputs, and invokes the generator only after those checks.The generated video is not fed back into the agent loop.Source: §3.1 · p. 4

3.2 Director Agent

  • The Director grounds entities and initial states, then classifies events by whether they affect physical conditions, object state, visibility, or scene constraints.Observation-only events change visibility without initiating, interrupting, or resetting physical evolution unless the input assigns them a physical effect.Source: §3.2 · p. 4

3.3 Physical Evolver Agent

  • The Physical Evolver builds mechanism records from evidence, activation conditions, mechanisms, and expected responses, then organizes them into dependent state transitions.Transitions track before- and after-states, active conditions, change processes, mechanism support, and dependencies on earlier transitions.Source: §3.3 · p. 4
  • The Physical Evolver carries each after-state forward, so mechanisms whose conditions remain active continue through occlusion and determine the state at reappearance.A separate review checks mechanism grounding, state continuity, dependencies, and consistency with Director constraints.Source: §3.3 · p. 4

3.4 Composer Agent

  • The Composer combines the original prompt, Director plan, mechanism records, and trajectory into a replacement prompt, revising only sentences flagged for physical-fidelity issues.It cannot introduce new mechanisms, and refinement stops at validation or the retry budget; the selected prompt is then sent to the unchanged generator.Source: §3.4 · p. 5

4 Experiments

Across four benchmarks, PhyEvolve outperforms the compared methods, including agentic baselines. Experiments also examine component contributions, agent backbones, user preferences, efficiency, and qualitative state evolution.

4.1 Benchmark Datasets

  • The four benchmarks assess interrupted-observation state evolution and broader physical correctness across distinct datasets and metrics.StEvo-Bench has 225 tasks with observation interruptions; PhyGenBench has 160 prompts covering 27 physical laws; PhyWorldBench has 1,050 prompts; VideoPhy-2 has 3,940 prompts derived from 197 actions.Source: §4.1 · pp. 5–6, Table 4 · p. 6, Table 5 · p. 6

4.2 Implementation Details

  • PhyEvolve is training-free; image-conditioned tasks produce 192 frames at 24 fps while preserving the input aspect ratio, whereas text-only tasks produce 124 frames at 24 fps at 1344 × 768 resolution.For text-only tasks, agents plan directly from the original prompt, and H3 generates the final video using the enhanced prompt.Source: §4.2 · p. 6

4.3 Main Results

  • PhyEvolve outperforms all listed methods on all four benchmark-level metrics, including NEWTON and Co-Director.Compared with H3, it gains 16.1 points on PhyGenBench Average, 9.2 on PhyWorldBench Overall Both, 6.2 on VideoPhy-2 All Joint, and 28 on StEvo-Bench Evolution Success.Source: §4.3 · p. 6
  • StEvo-Bench Evolution Success reaches 45.7% for PhyEvolve, versus 29.0% for NEWTON and 20.0% for Co-Director.PhyEvolve also leads on Progress (51.2%), Observation Control (89.5%), and Action Control (85.2%).Source: §4.3 · p. 6
  • PhyEvolve leads on PhyGenBench Mechanics, Thermal, and Material, and on PhyWorldBench Overall Performance at 75.8%, versus NEWTON’s 69.9%.NEWTON remains stronger on Optics (89.3% versus 84.7%); PhyEvolve’s Anti-Physics Both is 54.3%, versus 47.6% for H3 and NEWTON. On VideoPhy-2, it leads All Joint (30.1%) and Hard Joint (12.8%), though semantic adherence remains low.Source: §4.3 · pp. 6–7

4.4 Ablation Study

  • Removing Physical Evolver content causes the largest StEvo-Bench drops in Evolution, Progress, and Physics: 16.7, 13.2, and 24.5 percentage points.The qualitative comparison shows the full method depicting the glass’s fall, temporary occlusion, and shattered end state.Source: §4.4 · p. 7, Table 6 · p. 7, Figure 3 · p. 7
  • Qwen3-VL-32B has the highest PhyGenBench average score at 79.6%; user preference exceeds 60% for Video and Physics in every comparison.Against direct H3, overall Video and Physics preference rates are 74.6% and 75.3%; against NEWTON they are 75.4% and 74.7%.Source: §4.4 · p. 7, Table 7 · p. 7, Table 8 · p. 8
  • PhyEvolve uses 6.1 LLM calls and 17.0k tokens; its 452.5-second runtime is 75.2% below NEWTON and 55.2% below Co-Director.Qualitative examples show the wet front advancing during lights-off and the sandcastle washed away after a wave only with PhyEvolve among the described comparisons.Source: §4.4 · pp. 7–8, Table 9 · p. 8, §4.5 · p. 8

5 Conclusion

PhyEvolve guides generation through structured state-evolution planning, tracking physical changes across visible and out-of-sight intervals. With H3 fixed, it achieves the highest benchmark-level scores among compared methods on four benchmarks without retraining or model modification.

5 Conclusion

  • The conclusion presents PhyEvolve as a training-free multi-agent framework that guides video generation through structured state-evolution planning.Its explicit state trajectory coordinates physical actions and observation timing, tracking ongoing changes even when objects are out of sight.Source: §5 · p. 8
  • With H3 as the fixed generator, PhyEvolve achieves the highest benchmark-level scores among compared methods on PhyGenBench, PhyWorldBench, VideoPhy-2, and StEvo-Bench.The conclusion states that this requires neither retraining nor model modification.Source: §5 · p. 8

Limitations

The authors identify limits in reasoning reliability, quantitative dynamics prediction, difficult-task joint success, and evaluation duration.

Limitations

  • Evaluation uses short benchmark clips, so PhyEvolve’s performance on longer video sequences has not been established.The authors identify effectiveness over longer sequences as future work.Source: Limitations · p. 9

A.1 Agent Prompt

The appendix describes role-specific prompt templates and review procedures for the Director, Physical Evolver, and Composer.

A.1 Agent Prompt

  • The Director grounds entities and initial states, separates physical events from observation changes, and records an ordered shot plan with permitted event effects.Review checks source coverage, scene grounding, and event classification, with revisions limited to flagged fields.Source: §A.1 · p. 11
  • The Physical Evolver identifies mechanisms, activation conditions, and immediate responses, then propagates records into dependent transitions, including unobserved intervals.Separate reviews check prerequisites, responses, and state dependencies while retaining uncertainty when inputs do not determine an outcome.Source: §A.1 · p. 11
  • The Composer integrates the original request with reviewed scene and physical records into a replacement prompt, preserving requested actions, event order, and camera behavior.A fidelity review flags omissions or contradictions, and revisions are restricted to affected sentences.Source: §A.1 · p. 11

A.2 Additional qualitative results

Additional qualitative examples compare requested event progression and state changes across PhyGenBench and StEvo-Bench. Role-level templates show how the Director, Physical Evolver, and Composer structure and refine the planning process.

A.2 Additional qualitative results

  • Across five StEvo-Bench cases, PhyEvolve carries physical changes through occlusion or lighting changes so object states are updated when visibility returns.The cases include cardboard occlusion, curtain occlusion, and lighting changes; some also show failures to reveal the scene as instructed.Source: §A.2 · p. 11, Figure 9 · p. 16
  • Figure 10 presents selected PhyGenBench time-lapse examples involving bottle deformation, balloon deflation, dry ice sublimation, butter melting, and a sponge reacting to acid.These examples support visual inspection of event progression and state continuity; aggregate performance is summarized by benchmark-level metrics in the main experiments.Source: §A.2 · p. 11, Figure 10 · p. 17
  • The role-level templates assign scene grounding and action–observation disentanglement to the Director, causal state rollout to the Physical Evolver, and prompt construction and fidelity refinement to the Composer.The Director’s reviewer guides bounded local revisions; the Physical Evolver jointly reviews mechanism analysis and rollout; the Composer locks unrelated text during sentence-level corrections.Source: Figure 5 · p. 12, Figure 6 · p. 13, Figure 7 · p. 14
  • PhyEvolve completes grass ignition and rocket ascent in Figure 8, while Minimax-H3 and NEWTON fail to ignite grass, and NEWTON and Co-Director fail to show rocket lift-off.The figure samples seven frames uniformly over time; green and red boxes mark correct and failed state evolution.Source: §A.2 · p. 11, Figure 8 · p. 15

Related papers

Loading 2610.30717v1…