PaperlayerView exact revision on arXiv

Source-linked AI summary

NEWTON: Agentic Planning for Physically Grounded Video Generation

Yuxiang Feng, Juncheng Wang, Chao Xu, Yijie Qian, Huihan Wang, Wenlong Hou, Yang Liu, Baigui Sun, Yong Liu, Shujun Wang

arXiv:2605.18396v2cs.CV

TL;DR

Text prompts omit physical parameters that determine dynamics, and existing conditioning approaches do not jointly provide sufficiency, dynamism, and verifiability. NEWTON uses a trained planner to orchestrate physics tools and verifier feedback around frozen generators. On VideoPhy-2, it improves LTX-Video-2B Joint accuracy from 21.36% to 29.66% and demonstrates improvements across two frozen generators.

  • Problem

    Text prompts omit physical parameters that determine dynamics, while no existing approach jointly satisfies sufficiency, dynamism, and verifiability.

    Source: §1 · p. 2

  • Method

    NEWTON trains a planner to orchestrate physics-aware tools with a frozen generator and verifier in an iterative loop.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    29.66% versus 21.36% Joint accuracy on the full VideoPhy-2 set, with NEWTON improving both PC and SA over LTX-Video-2B; it also improves results across two frozen generators.

    Source: §5.2 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    The experiments validate that physical consistency can be engineered through agentic planning rather than modifying the generator.

    Source: §6 · p. 9

    NEWTON currently relies on three tools, and its verifier provides only scalar feedback.

    Source: §6 · p. 9

Abstract

from arXiv · show

Video generation models produce visually compelling results but systematically violate physical commonsense -- on VideoPhy-2, the best model achieves only 32.6% joint accuracy. We identify a specification bottleneck: text prompts are lossy compression of the physical world, omitting the parameters that fully determine dynamics, and no amount of model scaling can recover what was never specified. From this diagnosis we derive three properties that physics conditioning must satisfy -- sufficiency, dynamism, and verifiability -- and show that no existing approach satisfies all three. We present NEWTON, in which video generation is demoted from the system output to one action inside an agent's toolbox: a learned planner orchestrates physics-aware tools (keyframe generation, scientific computation, prompt refinement) to construct rich conditioning, and a verifier closes the loop for iterative re-planning. The planner is the sole trainable component, optimized on-policy via Flow-GRPO inside the live multi-turn loop. On VideoPhy-2, NEWTON improves joint accuracy from 21.4% to 29.7% on LTX-Video and from 30.7% to 37.4% on Veo-3.1, without modifying either generator. Our project page: https://Newton026.github.io/newton

1. Introduction

Video generators produce visually compelling videos but still violate physical commonsense, and text prompts omit parameters that determine dynamics. NEWTON uses a trained planner to orchestrate physics tools and a verifier in an iterative loop, improving physical commonsense on frozen generators.

1. Introduction

  • Video generators violate physics across domains, and scaling model size or training data has not closed the gap.Reported failures include changing speeds without contact, falling objects ignoring gravity, and collisions violating momentum conservation.Source: §1 · p. 1
  • Text prompts omit trajectory-determining parameters such as mass, friction, table height, and initial velocity, leaving the generator to hallucinate them.The paper characterizes this as an ill-posed problem that can yield visually plausible but physically incoherent dynamics.Source: §1 · p. 2
  • Physics conditioning must be sufficient, dynamic across scenes, and verifiable, but existing approaches do not satisfy all three properties.End-to-end training lacks sufficient specification, fixed-modality signals are not dynamic, and one-shot methods lack feedback.Source: §1 · p. 2
  • NEWTON makes video generation one action in a planner’s toolbox, with an Executor and Verifier supporting iterative tool selection and re-planning.Only the Planner is trainable; it is optimized on-policy via Flow-GRPO inside the live multi-turn loop.Source: §1 · p. 2
  • NEWTON substantially improves physical commonsense on two frozen generators without modifying them, using scene-dependent tool scheduling.The planner computes projectile trajectories, generates keyframes for spatial constraints, and refines prompts for material properties.Source: §1 · p. 2
  • The planner discovers scene-dependent tool-use strategies that generalize across unseen physical scenarios.The contributions also report substantial VideoPhy-2 improvements across two generators.Source: §1 · pp. 2–3

2. Related Work

Prior work uses simulators, modifies generators, or applies agentic systems to visual generation, but the paper says no method jointly satisfies sufficiency, dynamism, and verifiability. NEWTON differs from earlier video-agent approaches through its external tools, feedback loop, and on-policy planner training.

2.1. Physics-Grounded Video Generation

  • Simulator-based approaches deliver strong continuum-mechanics behavior but commit to fixed simulator families, while NewtonGen is limited by construction on collisions and multi-object interactions.The cited simulator methods include PhysMotion, PhysCtrl, and PhysChoreo; NewtonGen embeds linear physics-informed Neural ODEs with a residual MLP.Source: §2.1 · p. 3
  • Other approaches modify generators to internalize physics, while reward-based post-training such as PhyGDPO uses one-shot refinement without per-sample verification.The section discusses VideoREPA, WISA, and ProPhy as generator-modification approaches.Source: §2.1 · p. 3
  • No existing method jointly satisfies the sufficiency, dynamism, and verifiability properties identified by the paper.The paper identifies these as the properties addressed by NEWTON.Source: §2.1 · p. 3

2.2. Agentic Systems for Visual Generation

  • Unlike CECT’s enriched textual event chain, NEWTON uses heterogeneous tools, verifies and re-plans for up to five iterations, and trains its planner on-policy.The comparison says CECT plans once and uses a frozen generic LLM; NEWTON’s tools include keyframe generation, Python physical computation, and prompt refinement.Source: §2.2 · p. 3

3. Preliminary and Motivation

Diffusion Transformers accept conditioning signals and generate video by integrating from noise to data. The paper argues that text underspecifies physics, motivating scene-specific physical conditioning rather than generator retraining.

3.1. Video Generation with Diffusion Transformers

  • Modern text-to-video generators encode video into VAE latents, patchify them into tokens, and process them with transformer blocks.They are trained via flow matching, and an ODE solver integrates from noise at t=0 to data at t=1 during inference.Source: §3.1 · pp. 3–4
  • The same text prompt can correspond to multiple physically distinct trajectories that the generator cannot disambiguate without additional conditioning.The figure describes prompts as a lossy compression of physics.Source: Figure 3 · p. 4

3.2. Motivation: The Specification Bottleneck

  • On VideoPhy-2, even the best model achieves only 32.6% joint performance, while conservation-law violations approach 40%.Joint performance requires both Semantic Adherence and Physical Commonsense scores of at least 4.Source: §3.2 · p. 4
  • Text omits physical parameters such as mass, friction, table height, and initial velocity, which jointly determine a trajectory.The paper describes recovering these parameters from one sentence as an ill-posed task that can produce physically incoherent dynamics.Source: §3.2 · p. 4
  • The paper argues that generators learn physics implicitly from raw video, a data-inefficient process limited by training coverage.It notes that structured physical laws can determine trajectories when the relevant parameters are given.Source: §3.2 · p. 4
  • Rather than retraining the generator, the proposed strategy enriches conditioning with physically grounded keyframes, quantitative constraints, and precise prompts.The remaining challenge is automatically acquiring and structuring the physical knowledge needed for each prompt.Source: §3.2 · p. 4

4. NEWTON: Neural Agentic World-Aware Tool-Orchestrated Navigation

NEWTON improves physical plausibility from a frozen generator by enriching its conditioning through an iterative planner–executor–verifier system. Only the planner is trained on-policy via Flow-GRPO.

4. NEWTON: Neural Agentic World-Aware Tool-Orchestrated Navigation

  • NEWTON uses a Planner to choose physics-aware tools, an Executor to run them with a frozen generator, and a Verifier to score physical plausibility and feed back results for re-planning.The Planner is the only trainable component and is optimized on-policy via Flow-GRPO in the live multi-turn loop; the tools, generator, and Verifier remain frozen.Source: §4 · p. 4

4.1. System Pipeline

NEWTON uses a Planner, Executor, and Verifier to orchestrate physics-aware tools alongside a frozen video generator. Across fixed cycles, verifier feedback informs replanning, while keyframes, scientific computation, and prompt refinement add complementary physical conditioning.

4.1.1. Three-Role Architecture

  • The Planner selects actions from memory, the Executor dispatches tools alongside a frozen generator, and the Verifier scores videos on SA and PC and feeds results back.The Planner is the trainable role; the framework is generator-agnostic.Source: §4.1.1 · pp. 4–5

4.1.2. Iterative Cycle

  • Across T fixed cycles, memory updates from actions and observations; the system returns the video with the highest verifier score.Cycles may focus first on computation or prompt refinement before generating video.Source: §4.1.2 · p. 5

4.1.3. Physics-Aware Tools

  • Three tools enrich conditioning: keyframes anchor physical states, Python computation supplies numerical results, and prompt refinement adds physical details or constraints.Keyframes constrain interpolation; computation can inform later keyframe prompts or constraints.Source: §4.1.3 · p. 5

4.2. In-the-Flow Optimization

NEWTON trains its Planner within live system rollouts rather than offline expert trajectories. Flow-GRPO updates the Planner from complete multi-cycle outcomes, using a composite reward that combines quality and tool-use incentives with interface constraints.

4.2.1. Why In-the-Flow

  • Offline supervised training misses the Planner’s own mistakes, tool failures, and verifier feedback; NEWTON instead trains through rollouts of the live system under its current policy.The paper cites a 19% average accuracy drop versus a frozen baseline for SFT on expert trajectories in agentic settings.Source: §4.2.1 · p. 5

4.2.2. Flow-GRPO

  • Flow-GRPO runs full T-cycle rollouts before assigning trajectory rewards, then uses group-normalized advantages in a clipped surrogate update.The update applies to the Planner policy, with a KL penalty against a fixed reference policy.Source: §4.2.2 · p. 6

4.2.3. Reward Design

  • The reward combines quality, keyframe, and computation components, with format or length violations triggering a fixed penalty.Quality rewards tiered maximum SA and PC scores; the computation bonus requires a valid computation and positive quality reward.Source: §4.2.3 · p. 6
  • An independent keyframe bonus rewards newly generated keyframes when the resulting video meets a semantic-adherence threshold, while tool-use bonuses help densify achievable rewards.The keyframe bonus encourages exploration early in training; together, tiered quality and independent tool-use bonuses support group-normalized advantage estimation.Source: §4.2.3 · p. 6

5. Experiments

On VideoPhy-2, NEWTON improves joint accuracy over its LTX-Video-2B backbone, and it also improves average scores on PhyGenBench. Ablations report gains from larger planners, more planning cycles, on-policy training, and attaching NEWTON to a stronger frozen generator.

5.1. Experimental Setup

  • The evaluation uses VideoPhy-2’s 590 captions and 180-caption HARD subset, plus PhyGenBench’s 160 prompts across four physical categories.VideoPhy-2 reports PC, SA, and their joint threshold; PhyGenBench uses its official VLM-judged [0, 1] protocol.Source: §5.1 · p. 6, §5 · p. 6

5.2. Main Results on VideoPhy-2

  • Qualitative examples show NEWTON accumulating a salt mound and progressively separating grapefruit rind, unlike the described baseline outputs.The comparisons cover real-world samples and animation samples.Source: §5.2 · pp. 6–7, Figure 5 · p. 7, Figure 6 · p. 7
  • 21.36% to 29.66% Joint accuracy on the full VideoPhy-2 set and 4.44% to 12.22% on HARD; NEWTON improves both PC and SA over LTX-Video-2B.The HARD-set change is a 2.75× relative gain; VideoREPA and WISA trade higher PC for sharply lower SA.Source: §5.2 · p. 6, Table 1 · p. 7
  • In a 197-prompt human preference study, volunteers preferred NEWTON for both physics plausibility and overall video quality.Twenty volunteers made two forced-choice judgments per triplet.Source: §5.2 · p. 7, Figure 7 · p. 7

5.3. Cross-Benchmark Generalization on PhyGenBench

  • On PhyGenBench, NEWTON raises the average from 0.510 to 0.560, above Wan2.2-TI2V-5B’s 0.544; gains concentrate in Optics (+0.067) and Material (+0.092).Mechanics is essentially unchanged (−0.008).Source: §5.3 · pp. 7–8, Table 2 · p. 8

5.4. Ablation Studies

  • HARD-Joint reaches 12.22% with a 9B planner and 12.22% at T=5, versus 7.22% with 2B and 4.44% at T=2.The scale sweep also includes 4B; the cycle sweep trains a separate planner for each budget and evaluates it at that budget.Source: §5.4 · p. 8, Table 3 · p. 8, Table 4 · p. 8
  • Across cycles 1–5, Flow-GRPO gains about twice as much as offline SFT: PC +0.21 versus +0.11 and SA +0.22 versus +0.13.For HARD-Joint, Flow-GRPO improves +3.3 versus SFT’s +1.7; the figure reports best-so-far PC and SA over five refinement cycles.Source: §5.4 · p. 8, Table 5 · p. 8, Figure 8 · p. 8
  • On a 100-caption test subset, NEWTON lifts Joint by +8.4 on LTX-Video-2B and by +6.7 on Veo-3.1, from 30.74 to 37.41.The generator-backbone ablation keeps the planner, tools, and verifier fixed.Source: Table 6 · p. 8, §5.4 · p. 8

6. Conclusion

NEWTON addresses the specification bottleneck with an agentic framework that enriches generator conditioning with scene-specific physical knowledge, improving physical consistency across two frozen generators. Its current tool set and scalar verifier feedback limit the system's scope and re-planning detail.

6. Conclusion

  • NEWTON identifies lossy text prompts as a specification bottleneck and derives sufficiency, dynamism, and verifiability as requirements for physics conditioning.The framework demotes video generation to one action in a planner's toolbox.Source: §6 · pp. 8–9
  • NEWTON substantially improves physical consistency across two frozen generators on VideoPhy-2 without modifying either generator.The planner orchestrates physics-aware tools and a verifier in an iterative loop to enrich conditioning with scene-specific knowledge.Source: §6 · p. 9
  • NEWTON currently relies on three tools, while its verifier provides only scalar feedback.The authors suggest broader tool coverage and language-form diagnostics as future directions.Source: §6 · p. 9

Related papers

Loading 2605.18396v2…