PaperlayerView exact revision on arXiv

Source-linked AI summary

VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement

Wenzhuo Xu, Yuchen Zhu, Chongjian Ge, Xuan Shen, Jing Shi, Jason Kuen, Yongxin Chen, Molei Tao, Christopher McComb, Noelia Grande Gutiérrez, Jiuxiang Gu

arXiv:2609.03153v1cs.CV

TL;DR

Physically plausible video can still violate prompt-grounded physical obligations without revealing which obligation failed or when. VeriPhy addresses this with a pre-observation typed planning and evidence-traceable verification system, and reports 228 of 304 defects on a 149-clip core versus 164 for question decomposition. Its main scope boundary is that simulator-derived controls constrain scene diversity and the critic’s refinement interface was not exercised.

  • Problem

    Visual plausibility and scalar quality scores do not establish physical validity or identify the violated obligation and failure time.

  • Method

    VeriPhy compiles prompts into typed obligations before observation, scopes frozen specialist tools through validated execution, and deterministically composes provenance-carrying evidence into three-valued verdicts.

  • Results

    228 of 304 annotated defects were accounted for on 149 clips, compared with 164 for a published question-decomposition evaluator and 222 for monolithic prompting.

  • Takeaways & Limitations

    The evidence chain makes individual verdicts auditable and provides an interface for future physical refinement.

  • Takeaways & Limitations

    Simulator-derived controls bound scene diversity, the simulation-to-appearance mapping is unverified for physical consistency, and the corpus is recall-only.

Abstract

from arXiv · show

Visual fluency in generated video does not imply physical reliability, and a scalar quality score alone is incapable of indicating the obligation a clip violates or the moment it fails. We present VeriPhy, an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations and a statically validated execution plan before any frame is observed. During execution, observations gate and scope only declared calls to frozen low-level experts (e.g., segmentation and tracking, counting, eleven typed physical measurements over the resulting tracks, depth, OCR, and audio-event detection). Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state. Typed resolvers and fixed composition map usable records to a three-valued state (supported, contradicted, or unknown, surfaced as plausible, implausible, or abstain) with full provenance, so that every verdict is traceable to the evidence that produced it. We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time. On a 149-clip core carrying 304 such records, VeriPhy accounts for 228, against 164 for a published question-decomposition evaluator given the same clips and the same claims. Recall alone does not separate it from prompting the same backbone monolithically, which reaches 222; what separates them is that each decision retains its evidence record and provenance, making the traces auditable one verdict at a time and usable as the interface through which a critic verdict could be written back into generation.

1 Introduction

VeriPhy targets the gap between visually plausible video and physically defensible, auditable diagnosis. It compiles prompt-grounded obligations into an evidence-traceable critic and evaluates it against localized human flaw records.

  • Physical failures can involve entities, trajectories, contact, causality, audio timing, and cross-modal consistency despite perceptual plausibility.
  • Existing evaluators provide complementary judgments, localization, measurements, and multimodal checks rather than one auditable evidence chain.
  • VeriPhy compiles prompts into typed obligations and statically validated plans before observing frames, then scopes specialist calls through observations.
  • Usable evidence records feed typed resolvers and fixed composition, producing supported, contradicted, or unknown states with claim-level provenance.
  • 1,500 clips yield 2,582 human-written flaw records, while a 149-clip core contains 304 defects for recall analysis.
  • VeriPhy accounts for 228 of 304 defects, compared with 164 for a published question-decomposition evaluator and 222 for monolithic prompting.

2 Related Work

Related work has advanced structured video judgment, physical evaluation, and tool-using execution, but these capabilities remain distributed across separate protocols and evidence sources.

  • Video evaluation has progressed from aggregate metrics toward structured judgments covering perceptual quality, temporal consistency, prompt alignment, and explanations.
  • Prompt-decomposition methods make checks explicit, but their supporting evidence is generally still produced by the judging model.
  • Physics-oriented benchmarks add phenomenon-, rule-, trajectory-, and related physical tests to evaluation.
  • Execution harnesses separate orchestration from model judgment through explicit planning, dependencies, routing, scope, validation, termination, and trace logging.

3 Physics-Guided Simulation Pipeline

The simulation pipeline factors physical motion out of appearance by validating solver-generated trajectories and rendering them as geometric controls for video generation. This creates a known physical target while retaining prompt-driven appearance synthesis.

  • A trusted physics solver fixes the motion while the appearance prompt supplies only visual content, separating physical targets from generator appearance.
  • The generation setup treats prompt-conditioned generation and auditable criticism as separate maps over clips.
  • Figure 2 depicts a physics request becoming a simulated, validated scene whose motion becomes depth control for the generator.
  • The authoring map creates a structured scene specification, simulation produces a trajectory, and rendering converts solver state into depth control.
  • Validation checks event-specific predicates on simulated trajectories, including arcs, hit or miss relations, settling, and visibility.
  • Failed simulations trigger specification revision for at most three revisions, after which the system marks an unvalidated fallback.
  • The registered control shares the solver, camera, and playback frame clock, preserving alignment with the intended motion.
  • VACE injects the geometric control into a frozen Wan 2.2 backbone alongside appearance prompt, mask, and conditioning strength.

4 VeriPhy

VeriPhy is a constrained reason–act–observe critic that plans typed physical checks before seeing video, executes scoped specialist measurements, and composes provenance-bearing evidence into three-valued verdicts.

  • VeriPhy evaluates clips against prompts with a three-valued verdict whose decisions trace back to produced evidence.
  • A text-only planner creates typed claims and an executable measurement plan before observation, while specialist and semantic backends return targeted evidence.
  • Observations may gate, scope, or localize only nodes declared in the pre-observation plan, preserving obligation-to-measurement traceability.
  • Specialist measurements and learned semantic states remain distinct from deterministic adjudication, which yields supported, contradicted, or unknown.
  • The planner grounds claims in verbatim prompt spans and compiles grouped plans against a shared entity and event namespace.
  • Planning has no video input: each line specifies evidence needed to settle a claim rather than its truth value.
  • Surface operations expand into required execution closures, including occurrence, timing, co-visibility, masks, depth, and audio dependencies.
  • Static validation checks parsing, references, types, acyclicity, and claim coverage before execution.

5 Experiments

The experiments test control fidelity, trajectory commitment, scalability, and auditable flaw detection. VeriPhy finds more annotated flaws than question decomposition while revealing category-specific coverage and limitations in multi-object physics.

  • Experimental setup: 1,500 clips contain 2,582 human-written, prompt-grounded flaw records, narrowed to a 149-clip core with 304 annotated flaws for recall analysis.The core is the denominator for all reported recall comparisons.
  • Control fidelity under prompt variation: The control suppresses prompt-induced motion variation: clause-attributable range falls from 0.875 unconditioned to 0.001 with control throughout.With control withdrawn after the first fifth of sampling, clause range is 0.060 while seed range is 0.427.
  • Trajectory commitment under control withdrawal: Early control withdrawal preserves airborne flight but reverses post-contact motion on 9 of 16 seeds, compared with 0 of 16 when control is never released.At σ = 0.952, whole-clip horizontal and vertical retention fall 0.13 and 0.18, respectively.
  • Trajectory commitment under control withdrawal: Across three scenes, the thrown ball’s after-contact horizontal retention is −0.63 at σ = 0.952 but recovers to 0.98+ by σ = 0.903, while other scenes remain above threshold.The commitment band is σ ≈0.94–0.96, with vertical recovery later than horizontal for the thrown ball.
  • Scaling beyond single-body probes: The authoring, simulation, and conditioning path produces 1,314 scenes across 66 kinds, but multi-object interactions do not uniformly reproduce specified motion.Single-object trajectories match throughout, whereas billiards-break collisions fail across the deployed and alternate backbones.
  • Flaw recall on the evaluation core: VeriPhy finds 228 of 304 annotated flaws (75.0%), versus 164 (53.9%) for question decomposition and 222 for the same backbone prompted monolithically.VeriPhy finds every annotated flaw on 91 of 149 clips; question decomposition finds fewer flaws on 70 clips, ties on 63, and exceeds it on 16.
  • Flaw recall on the evaluation core: Recall varies from 90% for camera and style to 49% for counting, while physical motion covers four of seven flaws and deformation remains outside trajectory measurements.Subgroup recall stays within about ten points of the overall rate across severity, duration, flaw count, and difficulty strata, except for small-sample variation.

6 Conclusion

VeriPhy connects controllable generation, localized flaw evidence, and auditable physical criticism into a loop that can support both evaluation and refinement. Its evidence traces remain inspectable, while critic-guided rewriting shows promise but leaves the effect of text-only edits on physics unresolved.

  • Integrated system: VeriPhy combines a simulation-driven generator, a human-annotated defect benchmark, and a critic that compiles prompts into typed physical obligations.The system links generation, measurement, and ground truth in one evaluation loop.
  • Evaluation: 228 of 304 annotated defects were accounted for by VeriPhy, versus 164 for a published question-decomposition evaluator on the same 149 clips and claims.The comparison is made at flaw level on the development core.
  • Refinement: Auditable evidence records can be distilled into readable lessons that improve recall without retraining or absorbing the lessons into model parameters.The resulting state remains inspectable and editable.
  • Rewrite examples: Figures 15 and 16 show critic-guided rewrites repairing object warping and duplication defects in regenerated clips.The examples compare baseline and rewritten generations frame by frame.
  • Open problem: The rewrite loop improves many targeted clips, but independent evaluation finds no corresponding physical or semantic movement, leaving text-only physics correction unresolved.The open question is whether correction should instead enter through geometric control.
  • Future direction: Audio is architecturally supported but not exercised in this report, whose measured claims are visual.Future audiovisual verification would bind visual contact windows to audio onsets on a shared event clock.

7 Limitations

The work is bounded by its simulator-based control setting, recall-only and weakly annotated corpus, unresolved semantic binding, and incompletely exercised refinement safeguards. These limits constrain how broadly the reported results and mechanisms should be interpreted.

  • Scope: Simulator-derived control bounds scene diversity, and simulation-to-appearance mapping is not verified for physical consistency.This is identified as the primary limitation of the work.
  • Data: The corpus is recall-only, uses single-annotator flaw records, derives scope and category tags automatically, and lacks recorded source generators.Precision and inter-annotator agreement therefore are not directly established.
  • Evaluation risk: Repeated design against one frozen set risks overfitting even without training on that set.The concern follows from repeatedly using the same evaluation environment.
  • Verification limits: Spatial-relation and negative-event predicates remain limited, while symbolic identifier checks cannot guarantee that heterogeneous tools localized the same physical instance.Reasoner-mediated sub-checks reduce but do not remove semantic opacity.
  • Distillation: The acceptance gate is not exercised, so the reported distillation results do not test the admission discipline intended to reject bad lessons.Fresh misses also lack proxy history for defining the needed Uval.
  • Distillation: The taught planner reads 1.76 times as many claims as the untaught planner, so recall gains combine targeting improvements with a larger plan.Because flaw recall rewards asking more, the effect is not isolated to lesson quality.

A.3 Trajectory-agreement metrics

The trajectory-agreement metrics compare generated tracked paths with simulation paths over valid windows, splitting analysis at first ground contact and reporting explicit disagreement modes. Correlation is reported only when enough jointly defined frames are available.

  • Retention: Trajectory retention is the per-axis Pearson correlation between generated and control tracked positions over the valid intersection of a window and observed frames.The metric is reported only when the intersection meets a minimum frame count.
  • Windowing: The analysis splits windows at the first ground-contact frame, producing airborne and after-contact intervals.Contact is located from the control trajectory’s apex and subsequent downward movement.
  • Failure modes: A whole-clip disagreement is flagged when full-clip horizontal retention is below 0.7.This is one of two prefixed failure modes accompanying retention.
  • Failure modes: A late-window disagreement is flagged when full-clip horizontal retention is below 0.7 while airborne horizontal retention exceeds 0.9.The condition distinguishes late divergence from otherwise strong airborne agreement.

A.4 Critic dataflow and procedure

The critic procedure places learned planning and verification inside deterministic compilation, observation-gated scoping, typed resolution, and claim-level roll-up. It returns plausible, implausible, abstain, or infrastructure-failure outcomes from evidence records rather than an opaque score.

  • Planning and compilation: The planner compiles claims and a surface plan into a typed directed acyclic graph before any frame is read.Compilation validates parsing, references, typing, acyclicity, and operation coverage.
  • Execution: Observation-gated scope activates only declared operations, whose frozen backends run on scoped video grounded by SAM 3 tracks.Operations execute in topological order after validation or fallback handling.
  • Fallback: Unknown claims receive one learned fallback-verification step before clip-level roll-up.The fallback operates only after deterministic composition.
  • Final verdict: Clip-level output is implausible for any contradicted claim, plausible when at least one claim exists and all are supported, and abstain otherwise.Infrastructure failure is returned separately as ⊥infra.
  • Resolution: Typed resolvers map operation evidence to supported, contradicted, or unknown claim states, with deterministic rules for structured checks and a learned asymmetric gate for semantic checks.Abstention yields unknown, while infrastructure errors remain distinct.
  • Composition: Claim roll-up marks a claim contradicted if any associated obligation is contradicted, supported if all are supported, and unknown otherwise.This deterministic composition preserves obligation-level traceability.

A.5 Benchmark details

The benchmark combines human-authored flaw records with structured coverage and evaluation protocols for prompt-grounded physical failures. Its evaluation vocabulary includes typed checks for counts, ordering, and depth relations.

  • Core composition: The 150-clip core contains 306 flaws, with actions absent or wrong the largest category at 139.Other categories include wrong counts (36), wrong or missing objects (30), garbled text (25), and wrong spatial arrangement (19).
  • Core composition: Sixty-six clips contain more than one flaw kind, and prompts often combine lists, counts, ordering, text, and sound.The core therefore includes clips with multiple interacting obligation types.
  • Typed checks: Typed checks express obligations such as counting entities, ordering event windows, and testing depth relations through compiler-derived dependencies.The depth example consumes localization, masks, and depth evidence.
  • Annotation protocol: Human flaw records provide six fields, with prompt grounding and rationale always present while timing and boxes are optional.The optional fields capture temporal spans and object localization when annotators supply them.
  • Evaluation protocol: Table 9 reports benchmark comparisons, while its measures distinguish evidence-conditioned flags, anomaly-phrased flags, and matched reports.The benchmark rows are transcribed from the referenced benchmark figure and evaluated under its reported definitions.

A.8 Recall by clip property

Recall is evaluated against localized human flaw records on a fixed core, with comparisons using the same clips and claims. The core is balanced and contains both localized and near-whole-clip failures.

  • Recall by property: Table 10 groups recall by severity, duration, annotated flaw count, and difficulty stratum, with subgroup recall staying within about ten points of overall recall.The reported grouping indicates no listed property predicts whether a defect is found.
  • Annotation protocol: One human flaw record includes the quoted prompt fragment, visible-failure interval, object boxes, severity, confidence, and rationale.The record localizes the failure across the interval’s start, middle, and end.
  • Recall setup: The fixed core is balanced by severity, flaw count, and clip length, while 711 additional disjoint annotated clips are reserved for training.Only the first filtering step removes clips; removing audible-but-not-visible records leaves the clip count unchanged.

B Framework: Learning as State Optimization

The framework treats physical-reasoning improvement as optimization of a readable, human-supervised state rather than model-weight updates. It connects generation, benchmarking, and critic adjudication through this state abstraction.

  • Framework abstraction: The framework unifies a physics-guided generator, a human-authored defect benchmark, and a tool-using physical critic.These three artifacts are connected as the framework’s central abstraction and roadmap.
  • State optimization: Learning is defined as optimizing a human-readable, typed, provenance-carrying state that people can read, version, and veto.The framework’s thesis is that each improvement edits an inspectable object.
  • State optimization: The framework culminates in self-evolution without changing model weights.The proposed evolution operates through state changes rather than weight updates.
  • Related foundations: The framework draws on ReAct-style reason–act–observe agents, program-of-thought tool composition, and agent-memory approaches.These traditions motivate the separation of procedural, experiential, and external knowledge in the proposed state.

B.2 The unifying abstraction

The unifying abstraction represents the system as readable state, deterministic episode maps, and write channels connecting auditable physical verification to refinement. Frozen experts provide measurements, while typed obligations and provenance constrain interpretation.

  • State representation: VeriPhy’s state is S = (O, M, K): operator library, experience library, and external knowledge channel.These components store reusable procedures, defeasible experience-derived priors, and candidate-pruned external knowledge.
  • State representation: Operators are reusable measurement procedures with explicit cost nodes, while experience memory preserves evidence separately from defeasible belief.The operator library grows by packaging recurring reasoning sub-procedures; fresh measurements can override beliefs.
  • Read and write maps: A single-episode read operator consumes state to emit an auditable trace, while write channels update the state or generator from that trace.The six design pillars are expressed as read-S, write-S, write-generator, and supervisory operations.
  • Physicalized ReAct: Typed obligations specify what a prompt requires, and programs over O compose frozen experts for tracking, counting, physical measurements, depth, OCR, and audio events.This specializes the ReAct control-flow harness to explicit physical measurement.
  • Physicalized ReAct: Operators abstain when scope preconditions fail, and the adjudicator maps evidence to supported, contradicted, or unknown obligations plus localized contradiction packets.Unknown and abstain remain first-class outcomes rather than forced judgments.
  • Cost and scope: A program budget c(G) lets the reasoner trade measurement breadth against spending, but scaling the operator library remains roadmap work beyond the fixed O0 and compiler.The current instantiation uses a fixed operator set and compiler.
  • Refinement: The outward channel maps localized contradiction packets into local edits of simulation and control inputs before regeneration.The generator uses MuJoCo simulation and Wan 2.2 through VACE control conditioning.
  • Auditability: Auditability comes from deterministic roll-up over provenance-carrying measurements and human-readable diffs of O, M, or K.Reviewers inspect readable objects rather than gradients, while deterministic roll-up does not remove evaluator drift in operators or the compiler.

B.6 Weight-free context-training self-evolution

VeriPhy’s inward channel enables critic self-evolution by distilling trace feedback into readable state updates rather than changing model weights. Strict proxy-based acceptance, external knowledge separation, and human oversight constrain these updates.

  • Weight-free self-evolution: VeriPhy improves its critic by writing lessons from verification episodes into human-readable memory M without updating model weights.Both successful and failed episodes can be distilled into plans, operators, physics-failure priors, or scope restrictions.
  • Guarded updates: Candidate state edits are accepted only when a held-out proxy strictly improves, otherwise do-nothing elitism leaves the state unchanged.The edit applies component-wise to O, M, and K, while unsuccessful candidates are pruned.
  • External knowledge: External knowledge K is designed to break a closed self-reinforcing loop in which critic-derived priors condition future reasoning and generate more priors.Retrieval, physical facts, human corrections, and domain conventions enter K under the same candidate-and-pruning discipline.
  • Measured effect: 375 of 502 held-out cases achieved recall after distilling lessons from recorded misses, compared with 340 untaught cases.The reported intervention appends lessons to the planner’s rubric while leaving the operator library and external channel untouched.
  • Human oversight: Human reviewers may audit traces, veto or edit state updates, and approve deployment of refined generations or new states.Readable state diffs make oversight consist of reading and approving proposed updates, although the tooling and deployment protocol remain roadmap items.

C An Executable Roadmap

The executable roadmap separates VeriPhy’s implemented Phase 0 read path from future phases that add held-out evaluation, operator growth, self-evolution, control feedback, external knowledge, and human supervision. Only Phase 0 artifacts and measurements are reported results; later phases specify targets and protocols.

  • Scope of evidence: No number below the Phase 0 line is a result; Phases 1 and later state targets and protocols rather than outcomes.Table 12 distinguishes implemented or measured Phase 0 rows from unmeasured roadmap rows.
  • Phase 1 — Evaluation: Phase 1 targets held-out precision-and-recall evaluation with complete provenance coverage for every verdict.The planned protocol compares plan-based and per-claim adjudication and audits frame-level localization for contradicted verdicts.
  • Phase 2 — Operators: Phase 2 targets a growing versioned operator library with cost accounting and improved obligation coverage per unit cost.Planned experiments compare the fixed O0 pipeline with promoted operators at matched budgets.
  • Phase 3 — Context training: Phase 3 distills successful and failed traces into versionable priors in M, accepted through held-out proxy pruning and do-nothing elitism.The roadmap requires a distillation function, disjoint validation split, and diffable memory store.
  • Phases 4–6 — Write paths: Phase 4 proposes targeted critic-to-generator control edits, while Phases 5 and 6 add external knowledge and human-supervised deployment.These phases are roadmap targets, including re-adjudication after regeneration, provenance-tagged K facts, trace review, state-diff approval, and rollback.
Loading 2609.03153v1…