PaperlayerView exact revision on arXiv

Source-linked AI summary

Asking the World: Generalist Physical Reasoning through Agentic World Modeling and Probing

Shenxiang Zeng, Chen Yang, Peiyao Chen, Guohui Zhang, Jiansheng Fan, Chen Wang

arXiv:2609.39135v2cs.CV

TL;DR

Physical reasoning from video requires evidence about latent properties and dynamics, while existing VLM and fixed-pipeline approaches leave a need for adaptable physical modeling and inquiry. ATW constructs and probes task-relevant executable worlds, reporting improved accuracy across benchmarks and real-world scenarios.

  • Problem

    Physical questions often require evidence beyond video, while direct VLM inference and predefined tool pipelines offer limited reliability or adaptability across materials, dynamics, and question types.

    Source: §1 · p. 2

  • Method

    ATW adaptively builds a task-relevant executable world from video, then queries, simulates, and intervenes on that world to gather evidence for each question.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    ATW improves accuracy across CLEVRER, ContPhy, and three real-world scenarios, including 80.82% overall per-question accuracy on CLEVRER.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    The findings suggest that actively building and interrogating executable worlds is an effective approach to generalist physical reasoning, with complementary contributions from physical execution and feedback-guided adaptation.

    Source: §4.3 · p. 8, §5 · p. 9

    The method should not be used in safety-critical settings without independent validation because perception or simulation errors can produce incorrect conclusions.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Physical reasoning from video requires inferring latent physical properties and dynamics beyond direct observation. Direct VLM inference remains unreliable on complex physical tasks without explicit modeling and validation, while predefined tool pipelines rely on task- and domain-specific priors that limit generalization across materials, dynamics, and reasoning tasks. We introduce Asking the World (ATW), a generalist agent that constructs and interrogates task-relevant executable worlds through two adaptive stages: World Modeling calibrates a world from video, while World Probing queries, simulates, and intervenes on it to obtain question-relevant evidence. Rather than prescribing the operations in either stage, ATW determines how to model and probe according to the scene and question. We develop PolyWorld Engine, a lightweight and highly programmable Warp-based multiphysics simulator for constructing and probing worlds with rigid bodies, soft bodies, cloth, ropes, fluids, and their coupled interactions. CEM-based system identification recovers task-relevant dynamics during World Modeling. The resulting world becomes an active workspace for question-directed physical experiments rather than a predetermined downstream tool. We evaluate ATW on CLEVRER, ContPhy, and three real-world scenarios. Using Gemini-3-Flash as its base VLM, ATW achieves 80.82% overall per-question accuracy on CLEVRER, improving direct Gemini-3-Flash by 46.50 points, GPT-5.5 by 13.58 points, and PhysMind by 8.27 points. On ContPhy, it reaches 70.56% overall accuracy, surpassing Gemini-3-Flash by 28.10 points and GPT-5.5 by 3.53 points. Across the three real-world scenarios, ATW achieves 71.67% accuracy, 28.33 points above GPT-5.5. These results establish agentic world modeling and probing as an effective, execution-grounded approach to generalist physical reasoning. Project page: https://asking-the-world.github.io/

1 INTRODUCTION

Physical reasoning often needs evidence beyond video, but existing approaches either rely on VLM inference or commit to predefined physical representations and procedures. ATW instead builds and probes task-relevant executable worlds, with broad reported gains across benchmark settings.

1 INTRODUCTION

  • Existing VLM and physics-grounded approaches leave a gap for an agent that can construct an appropriate executable world and decide how to interrogate it for each scene and question.VLM approaches face failures in physical consistency and causal evolution, while physics-grounded methods often commit in advance to a representation, model, or inference procedure.Source: §1 · p. 2
  • ATW organizes physical reasoning into adaptive World Modeling, which constructs task-relevant executable worlds, and World Probing, which gathers question-directed evidence through queries, simulations, and interventions.The modeling stage commits an executable world before the probing stage uses it to answer questions.Source: §1 · p. 2, Figure 2 · p. 3
  • 80.82% overall per-question accuracy on CLEVRER, 70.56% on ContPhy, and 71.67% across three real-world scenarios are reported for ATW.The paper reports gains over GPT-5.5 of 13.58 points on CLEVRER, 3.53 points on ContPhy, and 28.33 points across the real-world scenarios.Source: §1 · p. 2
  • PolyWorld Engine supports on-demand construction and probing of scenes with rigid bodies, soft bodies, cloth, ropes, fluids, and coupled interactions.CEM-based system identification recovers task-relevant parameters from visual evidence.Source: §1 · p. 2

2 RELATED WORK

Related work spans visual physical reasoning, executable-world construction, and agents that reason through engines. ATW distinguishes itself by using execution feedback to select what to model and test for physical questions.

Visual Physical Reasoning

  • Visual physical-reasoning methods use structured object models, targeted VLM training, tools, or simulator search, while ATW uses execution feedback to select what to model and test.Prior approaches’ physics is described as implicit or procedurally fixed.Source: §2 · p. 3

Executable Physical Worlds

  • Executable-world methods optimize within specified dynamics, reconstruct scenes, or use reconstructed dynamics for question answering, whereas ATW builds video-calibrated worlds with heterogeneous materials and coupled interactions.Prior systems typically fix representations or material families, or infer dynamics from static inputs.Source: §2 · p. 3

Agents Reasoning through Engines

  • Prior agents use tools, programs, or interactive environments for observable scenes or prespecified goals, while ATW conditions world construction on physical questions about hidden, future, or counterfactual dynamics.ATW chooses which physical hypotheses to instantiate and probe.Source: §2 · p. 3

3 METHOD

ATW builds a task-sufficient executable world from video, then probes that committed world for question-relevant evidence. Its adaptive stages, programmable multiphysics engine, and video-grounded calibration support this process.

3.1 ATW: TWO-STAGE AGENTIC PHYSICAL REASONING

  • ATW proceeds through World Modeling, which commits a task-sufficient executable world, followed by World Probing, which adapts question-directed experiments within that fixed world.Feedback can change decisions within either stage, but probing does not reopen modeling or alter the committed base world.Source: §3 · p. 3, §3.1 · p. 4
  • The modeling agent selects question-relevant entities, geometry, motion, physical representations, and parameters, revising assumptions when intermediate evidence or simulated trajectories mismatch.The stage commits object identities, geometry, initial states, physical models, and fitted parameters.Source: §3.1 · p. 4

3.2 POLYWORLD ENGINE: AGENT-OPERABLE MULTIPHYSICS WORLDS

  • PolyWorld Engine combines rigid bodies, soft bodies, cloth, ropes, and fluids in a shared executable scene with coupled interactions and reusable simulation data.Its semantic operations preserve the source world, objects, temporal scope, and execution status of evidence.Source: §3.2 · pp. 4–5, Table 1 · p. 5
  • Probing reuses the calibrated base world for predictions and creates separate branches for interventions, allowing multiple questions and candidate interventions to share fitted parameters and initial conditions.Branch trajectories and events remain separately addressable.Source: §3.2 · p. 5

3.3 GROUNDING AND CALIBRATING EXECUTABLE WORLDS

  • World Modeling grounds executable scenes in masks, motion tracks, and geometry, requesting or refining only the evidence needed by the selected representation.Masks delimit objects and visible deformation, tracks constrain motion, and geometry establishes spatial configuration and initial conditions.Source: §3.3 · p. 5
  • CEM fits exposed physical parameters by comparing simulated, model-specific readouts with visual evidence, while persistent motion or shape discrepancies can prompt a representation or object-binding revision.The fitting procedure updates a smoothed diagonal Gaussian proposal from the lowest-loss elite samples and progressively incorporates longer observation windows.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments compare ATW with VLM and physical-reasoning baselines on CLEVRER, ContPhy, and real-world videos, then ablate adaptive reasoning, simulation, and system identification. ATW leads the reported overall results across all three settings.

4.1 EXPERIMENTAL PROTOCOL

  • Evaluation covers CLEVRER, ContPhy, and three real-world scenarios, comparing foundation VLMs, trained video reasoners, and training-free methods.ATW uses Gemini-3-Flash and PolyWorld Engine without benchmark training or fine-tuning; controlled variants isolate structured perception, physical execution, and adaptive reasoning.Source: §4 · p. 6, §4.1 · p. 6

4.2 MAIN RESULTS

  • ATW leads all three reported settings, scoring 80.82% CLEVRER overall per-question accuracy, 70.56% ContPhy overall accuracy, and 71.67% real-world accuracy.On CLEVRER, ATW improves on Gemini-3-Flash by 46.50 points, GPT-5.5 by 13.58, and PhysMind by 8.27; on ContPhy, it exceeds Gemini-3-Flash by 28.10 and GPT-5.5 by 3.53. Real-world accuracy compares with 43.33% for GPT-5.5 and 33.33% for Gemini-3-Flash.Source: §4.2 · p. 6, Figure 4 · p. 6, Table 2 · p. 7
  • Real-world modeling segments task-relevant objects, tracks poses across key frames, and reconstructs executable dynamics for targeted probes.These steps provide geometry, trajectories, and interactions for prediction and counterfactual probes.Source: §4.2 · p. 7, Figure 5 · p. 8

4.3 ABLATION STUDIES

  • Across CLEVRER and ContPhy, accuracy rises from video-only through perception-only and single-pass worlds to Full ATW.Full ATW adds 6.90 CLEVRER points and 8.92 ContPhy points over Single-pass; executable worlds provide substantial gains despite category variation.Source: Figure 6 · p. 8, §4.3 · pp. 8–9
  • With Warp fixed, CEM raises CLEVRER per-question accuracy from 43.45% to 82.76%; with CEM fixed, Warp has the lowest trajectory error and highest accuracy among tested backends.CEM also raises per-option accuracy from 69.49% to 93.19%; both comparisons associate lower fitting error with stronger downstream reasoning.Source: Figure 7 · p. 9, §4.3 · p. 9
  • With Qwen3-VL-235B-A22B, ATW raises overall per-question accuracy from 38.85% to 76.32% on CLEVRER and from 45.23% to 62.46% on ContPhy.Paired comparisons improve in the same direction across the two tested backbones and settings.Source: Figure 8 · p. 9, §4.3 · p. 9

5 CONCLUSION

ATW uses feedback-guided world modeling and question-directed probing to reason through executable worlds, with PolyWorld Engine supporting diverse materials and coupled interactions. Across the evaluated benchmarks and real-world scenarios, it improves reasoning accuracy; the authors caution against safety-critical use without independent validation.

5 CONCLUSION

  • ATW constructs task-relevant executable worlds and uses observations and execution feedback to revise models and guide question-directed probes.PolyWorld Engine supports diverse materials and coupled interactions, while system identification grounds dynamics in visual evidence.Source: §5 · p. 9
  • Controlled ablations show complementary contributions from physical execution and feedback-guided adaptation.The reported evaluations cover CLEVRER, ContPhy, and three real-world scenarios.Source: §5 · p. 9
  • Generative AI assisted with code implementation and manuscript drafting, but not synthetic-data generation or mathematical claims; authors reviewed the code and verified results.Source: AI USE STATEMENT · p. 10
  • Errors in perception or simulation can produce incorrect conclusions, so safety-critical use requires independent validation.Source: ETHICS STATEMENT · p. 10

A MORE IMPLEMENTATION DETAILS

The implementation combines visual foundation modules with a programmable multiphysics simulator and question-conditioned tools. System identification fits executable worlds to video evidence, while agents inspect, refine, and probe those worlds.

A.1 MODELS AND VISUAL FOUNDATION MODULES

  • Gemini-3-Flash selects operations and interprets evidence in both stages; neither it nor the visual modules are trained or fine-tuned on CLEVRER or ContPhy.The cross-backbone experiment substitutes Qwen3-VL-235B-A22B in both stages.Source: §A.1 · p. 14
  • The visual stack combines segmentation, 3D geometry and tracking, object reconstruction, and pose alignment to retain object identities and motion evidence.The agent can inspect segmentation and request another pass when masks do not capture relevant objects.Source: §A.1 · p. 14

A.2 WARP-BASED SIMULATION AND CEM FITTING

  • Warp-based models simulate rigid bodies, soft bodies, cloth, ropes, and fluids from scene geometry, initial conditions, and candidate parameters.A readout converts simulated states into quantities comparable to visual observations.Source: §A.2 · p. 14
  • System identification fits model parameters to retained visual evidence, adapting motion or shape discrepancies to each physical representation and its observable dynamics.Observations can include tracked rigid-body poses, soft-body deformation, cloth points, rope centerlines, or fluid distributions; validity and confidence affect which measurements count.Source: §A.2 · p. 15
  • CEM samples bounded parameter candidates, updates its proposal from the lowest-loss elite set, and extends the fitting horizon as later interactions become available.Fitted parameters are retained with geometry and initial conditions for replay and intervention.Source: §A.2 · p. 15

A.3 QUERY-CONDITIONED PHYSICAL ROLLOUT TOOLS

  • Question-conditioned tools inspect world assets and properties, continue unedited dynamics, or execute interventions such as object removal.Predictive questions use the post-observation interval, while counterfactual and goal-directed questions test modified worlds.Source: §A.3 · p. 15
  • Readouts return events, trajectories, timelines, or predicate results for explicit worlds and intervals; comparisons quantify how interventions change motion and events.The agent can also inspect observed motion and simulated frames, revisit evidence, and update object bindings.Source: §A.3 · p. 16

A.4 SAM 3 AGENT FOR GENERAL-PURPOSE VIDEO SEGMENTATION

  • The SAM 3 agent selects object masks from prompted candidates, revises rejected seeds using visibility and feedback, then propagates accepted masks bidirectionally across video.Its output includes indexed masks, object-to-track associations, and temporal quality summaries without requiring a mesh or rigid-motion assumption.Source: §A.4 · p. 16

B.1 CLEVRER

The evaluation setup spans rigid-body video reasoning, continuum systems, and recorded real-world interactions. Each benchmark tests distinct question types, with real-world answers drawn from multi-view recordings and ambiguous cases excluded.

B.1 CLEVRER

  • CLEVRER evaluates causal reasoning over synthetic videos of interacting rigid objects, using 1,000 validation videos, 4,280 questions, and 14,228 answer options.Source: §B.1 · pp. 16–17
  • Its explanatory, predictive, and counterfactual questions test event causes, post-video outcomes, and changes under interventions such as object removal.Overall per-question accuracy requires every answer option associated with a question to be correct.Source: §B.1 · p. 17

B.2 CONTPHY

  • ContPhy evaluates continuum-system reasoning across 600 videos and 1,950 questions about physical properties and changes to scenes.Source: §B.2 · p. 17
  • Its questions cover properties, prediction, counterfactuals, and goal-directed interventions across fluids, cloth, ropes, and deformable soft balls.Overall accuracy weights each question equally.Source: §B.2 · pp. 17–18

B.3 REAL-WORLD

  • The real-world dataset contains 60 videos and questions across billiards counterfactuals, bouncing-ball prediction, and toy-car prediction, with 20 examples per category.Source: §B.3 · p. 18
  • Predictive reference answers use future footage from synchronized views, while counterfactual questions are retained only when observers can answer confidently; category and overall accuracy use question counts.Cases with unresolved spatial ambiguity or viewpoint occlusion are excluded.Source: §B.3 · p. 19

C.1 DETAILED REAL-WORLD RESULTS

Table 4 reports detailed accuracy for 60 real-world videos and questions, with each scenario contributing 20. It distinguishes per-question, per-option, and per-judgment reporting.

C.1 DETAILED REAL-WORLD RESULTS

  • Overall per-judgment accuracy aggregates 80 billiards options, 20 bouncing-ball judgments, and 60 toy-car options, totaling 160 judgments.Source: §C.1 · p. 19
  • Table 4 covers three scenarios with 20 videos and questions each, and marks the best result in each column.Source: Table 4 · p. 19
  • Per-question accuracy requires all associated option judgments to be correct; per-option accuracy is additionally reported for billiards and toy-car tasks.Source: Table 4 · p. 19, §C.1 · p. 19

C.2 ADDITIONAL ABLATION RESULTS

The ablations show that identity binding improves accuracy across question categories, while perception-module choices produce distinct accuracy profiles. Full ATW achieves the highest overall score among the reported module variants.

Effects of identity binding and perception modules

  • Removing appearance cards and identity bindings lowers overall accuracy from 82.30% to 70.11% on CLEVRER and from 68.00% to 62.46% on ContPhy, with declines across every question category.The ablation compares the full system with a version lacking both appearance cards and identity bindings.Source: §C.2 · p. 20
  • FoundationPose tracking reaches 80.46% overall accuracy and 97.14% predictive accuracy on CLEVRER, but its rigid-pose formulation does not capture evolving cloth, fluid, or soft-body shapes.Its tracking uses an input object mesh and rigid-body prior; ATW uses Track4World for trajectory tracking across diverse physical systems.Source: §C.2 · p. 20
  • GeoCalib achieves 78.62% overall accuracy, while full ATW obtains the highest overall score among the reported perception-module choices.The GeoCalib variant estimates camera intrinsics and gravity direction using learning-based single-image camera calibration.Source: §C.2 · p. 20

C.3 DETAILED PHYSICAL-GROUNDING ABLATIONS

The physical-grounding ablations compare system-identification methods and physics backends, while the system prompts specify how ATW routes scenes, probes worlds, and answers from retained evidence.

C.3 DETAILED PHYSICAL-GROUNDING ABLATIONS

  • On ContPhy, CEM has the strongest aggregate performance among the three configurations, improving physical-property, predictive, goal-driven, and overall accuracy over Random Search while matching its 63.01% counterfactual accuracy.Random Search and CEM use equal simulation budgets.Source: §C.3 · p. 20, Table 5 · p. 20
  • On CLEVRER, trajectory RMSE decreases from default parameters through Random Search to CEM as overall per-question and per-option accuracy rises; with CEM fixed, Warp also follows the lower-error, higher-accuracy trend.The largest reasoning gains occur in explanatory and counterfactual questions, while predictive accuracy remains high across configurations.Source: §C.3 · pp. 20–21, Table 6 · p. 20

D SYSTEM PROMPTS

  • Routing selects planar or three-dimensional representations from visible interaction geometry and uncertainty, while object discovery assigns stable identities and physical types from temporal evidence.The prompts distinguish out-of-plane states and occluded contact surfaces from cases adequately described by planar outlines, and direct segmentation refinement when candidates are inadequate.Source: D SYSTEM PROMPTS · p. 21
  • Probing instructions require question-matched tool calls and inspection of returned evidence; answer generation treats unknown evidence as unresolved rather than negative.Answers must address the requested entities, intervention, and time window, cite supporting evidence, and identify missing evidence when the answer cannot be determined.Source: D SYSTEM PROMPTS · p. 21
Loading 2609.39135v2…