PaperlayerView exact revision on arXiv

Source-linked AI summary

Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning

Hanyang Wang, Yimo Cai, Weiliang Chen, Jiawei Chi, Haowen Sun, Qiyu Dai, Yi-Hsin Hung, Xingzhuo Guo, Jinshan Ren, Runmao Yao, Ziwei Liu, Mingsheng Long, Yueqi Duan, Jun Gao, Jiangran Lyu, Fangfu Liu, Jialong Wu

arXiv:2608.27549v1cs.CV

TL;DR

Physical reasoning across changing situations calls for compact representations of what exists, how it evolves, and what governs behavior. Code-as-World encodes these structures as executable code and uses an agentic discovery loop, while Code-as-World-VL achieves state-of-the-art performance on QuantiPhy [11] and outperforms leading proprietary models.

  • Problem

    Physical reasoning across changing objects and observations calls for compact representations of what exists, how it evolves, and which factors determine behavior.

    Source: §1 · p. 3

  • Method

    Code-as-World represents physical composition, dynamics, and appearance as executable code, which an agent proposes, executes, renders, verifies, and iteratively refines from multimodal observations.

    Source: §1 · p. 3

  • Results

    Code-as-World-VL achieves state-of-the-art performance on QuantiPhy [11], and Code-as-World-VL-9B outperforms substantially larger models, including Gemini-3.1-Flash.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Verified executable worlds provide scalable supervision for vision-language models' quantitative physical reasoning.

    Source: §1 · p. 4

    QuantiPhy covers only a limited subset of physical understanding, emphasizing monocular scale calibration under relatively constrained motion settings.

    Source: §5.5 · p. 15

Abstract

from arXiv · show

Physical understanding and reasoning depend on forming compact and generalizable representations of the world. While modern vision-language models can recognize and explain diverse physical events, they often lack explicit representations of the underlying mechanisms-such as object states, physical parameters, and governing dynamics-needed for reliably reasoning how the world evolves and responds to interventions. In this work, we introduce Code-as-World, a paradigm that represents physical worlds through executable world representations. By expressing physical composition, dynamic evolution, and visual appearance as executable code, Code-as-World provides a compact, quantitatively grounded, and controllable abstraction of the physical world. To construct such representations from multimodal observations, such as natural-language descriptions or real-world videos, we develop an agentic discovery loop inspired by abductive reasoning, where an agent proposes, executes, renders, verifies, and iteratively refines executable world hypotheses. As a concrete application, we use verified executable worlds to provide scalable physical supervision for training vision-language models on quantitative physical reasoning. Experiments show that Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and surpasses leading proprietary models, highlighting the potential of executable world representations as a scalable foundation for physical intelligence.

1 Introduction

Physical intelligence requires generalizing across unfamiliar situations, which calls for compact representations of what exists, how it evolves, and what determines its behavior [1] [2]. Code-as-World represents task-relevant physical structure as executable code and uses verified worlds to supervise quantitative physical reasoning.

1 Introduction

  • Generalization across changing objects, configurations, or observations is central to physical intelligence, motivating representations that compress sensory experience into a compact world abstraction [1] [2].Humans reason with concepts such as objects, mass, motion, gravity, and friction rather than memorizing individual observations.Source: §1 · p. 3
  • Vision-language models can describe physical phenomena, but physical reasoning also requires recovering mechanisms—object states, physical parameters, dynamics, and responses to interventions—that explain and predict them [4] [5] [8] [9] [10] [11] [12] [13].The paper distinguishes phenomena, which describe what happens, from mechanisms, which explain why it happens and predict outcomes under different conditions.Source: §1 · p. 3
  • Code-as-World expresses task-relevant physical structure as executable code specifying physical composition, dynamic evolution, and visual appearance.Objects and relations are organized compositionally, while states, physical parameters, and governing dynamics remain explicit.Source: §1 · p. 3
  • Its agentic discovery loop proposes, instantiates, executes, renders, and verifies world hypotheses, using discrepancies with language or visual evidence to refine them.The loop treats representation as an inverse problem rather than a one-shot prediction.Source: §1 · pp. 3–4
  • On QuantiPhy, Code-as-World-VL-9B outperforms Gemini-3.1-Flash, while Code-as-World-VL-27B surpasses the 9B variant and all evaluated baselines.The models are trained with measurement supervision and physical supervision derived from verified executable worlds.Source: §1 · p. 3

2 In Search of Physical World Representations

Pixel prediction, 3D reconstruction, and language each capture important aspects of the physical world, but none alone provides a complete representation for semantic understanding, structural composition, and temporal prediction. The section motivates an executable representation that explicitly captures entities, states, and mechanisms while combining these strengths.

Pixels

  • Pixel-prediction models can generate plausible futures without distinguishing underlying causes, so visual plausibility alone does not ensure physically correct reasoning.A physically meaningful representation must disentangle persistent object identity, state transitions, interactions, and viewpoint changes.Source: §2 · p. 4

3D

  • 3D reconstruction constrains geometry, viewpoint, and appearance, but faithful reconstruction may leave dynamic variables entangled and fail to explain causal physical behavior.Recovering an object's geometry does not by itself explain why it falls after support is removed or how it interacts with other objects.Source: §2 · p. 4

Natural language

  • Language captures compact, compositional, transferable semantic concepts, but discrete tokens poorly express precise geometry, trajectories, contacts, and physical parameters.It serves as an effective semantic interface, not a complete physical-state representation.Source: §2 · p. 4

Representation synthesis

  • Code complements pixels, 3D, and language with an abstract, quantitatively grounded, executable representation that can explicitly capture entities, states, and mechanisms.The desired representation integrates semantic abstraction, structural composition, and temporal evolution rather than merely describing or reproducing observations.Source: §2 · p. 4, Figure 2 · p. 5

3 Code as Worlds: Executable World Representations

Code-as-World represents physical worlds in code, separating structured physical states from visual appearance while enabling execution and rendering. Its executable world representation specifies what exists, how it evolves, and how it is observed.

3 Code as Worlds: Executable World Representations

  • Code-as-World represents entities, relations, physical parameters, and events as operations over states, separating compositional, editable physical structure from rich visual appearance.Execution and rendering generate observations consistent with the represented world, prioritizing physical equivalence over pixel-level duplication.Source: §3 · p. 5

3.1 Practical Implementation

  • An executable world representation couples a physical simulation engine with composition (entities and properties), dynamic evolution (states and events), and visual appearance (camera and rendering).Environmental structures that support or collide with objects belong to composition; backgrounds and other visual-only context belong to appearance.Source: §3.1 · pp. 5–6
  • EWR components can be inspected, modified, and executed independently, enabling changes to objects, physical parameters, dynamic conditions, or camera settings while preserving the remaining structure.This interface supports downstream reasoning, simulation, verification, and data generation.Source: §3.1 · pp. 5–6

4 Agentic Discovery of World Representations

Code-as-World recovers executable world representations from text or video through an iterative propose–instantiate–execute–render–verify process. These representations expose physical mechanisms for querying and intervention, while the discovery process is limited by what the simulator can model.

4 Agentic Discovery of World Representations

  • The discovery process externalizes physical mechanisms as executable interfaces that can be queried, intervened on, and verified.The resulting worlds support physical data generation, quantitative supervision, verifiable reward construction, and downstream physical reasoning.Source: §4 · p. 6

4.1 Evidence Construction

  • Modality-specific evidence adapters constrain a shared EWR space, using semantic evidence from text or visual evidence from real video.Text hypotheses use extracted entities, relations, events, and outcomes plus physical priors and defaults; video hypotheses use depth, masks, tracks, and generated object meshes [47].Source: §4.1 · p. 6, Figure 3 · p. 7

4.2 Agentic Discovery Loop

  • The agent iteratively refines a parsimonious EWR by instantiating it in a simulator, inspecting its trajectory, rendering predictions, and verifying them against input evidence.Text verification checks semantic and physical constraints; video verification compares RGB, depth, masks, and trajectories, with discrepancies guiding revisions until acceptance or rejection.Source: §4.2 · p. 7

4.3.1 Experimental Setup

  • The evaluation tests video reconstruction over up to five rounds against one-shot generation and Best-of-5, using visual, overlap, trajectory, velocity, and accuracy metrics.Candidate videos come from motion-filtered WISA-80K [48] and are manually reviewed; text specifications are LLM-generated and human-reviewed. Across input modalities, the same simulation interface produces synchronized videos and physical states.Source: §4.3.1 · pp. 7–8

4.3.2 Qualitative Analysis

  • Qualitative examples show text descriptions becoming physically grounded videos and real videos becoming executable worlds, which can be edited for coherent counterfactual resimulation.Changing initial velocity alters trajectories, while changing camera configuration renders the same collision from different views; video synthesis preserves the specified physical evolution and viewpoint.Source: §4.3.2 · p. 8, Figure 4 · p. 9, Figure 6 · p. 10

4.3.3 Agentic Discovery Loop

  • At a matched five-evaluation budget, iterative discovery outperforms Best-of-5 sampling on most aspects of video reconstruction.Static quality and most motion-fidelity metrics improve overall across successive rounds; Figure 5 reports Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D.Source: §4.3.3 · p. 9, Figure 5 · p. 9

4.4 Limitations

  • When real-world dynamics exceed a simulator’s modeling scope, the loop may converge on a locally plausible EWR without recovering a mechanistically accurate explanation.Small variations in terrain, contact geometry, material properties, or other latent factors can affect even apparently simple rigid-body motion.Source: §4.4 · p. 9

5 Application: Learning Quantitative Physical Reasoning

Code-as-World uses verified executable worlds to provide scalable supervision for quantitative physical reasoning, where models must infer real-world quantities hidden in monocular video. The section describes a two-phase training approach and evaluates its image-space grounding and world-space reasoning.

5.1 Problem Formulation

  • For world-space queries, a reference quantity with known value calibrates pixel measurements into metric size, displacement, velocity, or acceleration; 3D depth adds geometric cues.The model predicts a numerical answer from a monocular video and question, with questions specifying the object, timestamps, quantity, and output unit.Source: §5.1 · p. 11

5.2 Image-Space Measurement Grounding

  • Bounding boxes, masks, and tracks from visual datasets generate pixel-level questions about extent, position, displacement, velocity, and acceleration, teaching localization, measurement, and tracking.Object extent and position are read from boxes or masks, while motion quantities are derived from tracks; this stage requires no world-space calibration.Source: §5.2 · p. 11

5.3 World-Space Physical Calibration from Verified Executable Worlds

  • Text-driven worlds provide exact simulator-state supervision, while video-driven worlds better match real-world appearance and motion; joint training combines physical accuracy with visual generalization.Both sources yield synchronized videos, states, and world-level VQA examples, optimized with GRPO and rewards for numerical accuracy, units, and response format.Source: §5.3 · pp. 11–12

5.4.1 Experimental Setup

  • Training uses image-space supervised fine-tuning followed by GRPO on world-level VQA, with direct-answer 4B and 9B variants and a reasoning-enabled 27B variant.All variants uniformly sample 16 temporally ordered video frames.Source: §5.4.1 · p. 12

5.4.3 Quantitative Physical Reasoning

  • Code-as-World-VL-9B achieves the best average among direct-answer variants, while 4B beats larger open-weight baselines and remains competitive with leading proprietary systems.The models show consistent strength across benchmark subsets, connecting visual measurements with metric scale and motion.Source: Table 1 · p. 13, §5.4.3 · pp. 14–15
  • Code-as-World-VL-27B exceeds the direct-answer 9B variant and strongest proprietary baseline, but its larger scale and reasoning protocol change together.Its average MRA is highest among the reported variants; the comparison does not isolate the effect of reasoning alone.Source: Table 1 · p. 13, §5.4.3 · p. 15

5.4.2 Image-Space Measurement

  • Both 4B and 9B Image-Space variants perform strongly, and full Code-as-World-VL models improve over them on every image-space benchmark.World-Space training preserves and strengthens the underlying image-space grounding capability.Source: §5.4.2 · p. 14

5.5 Limitations

  • Evaluation covers constrained monocular scale calibration, while models learn verified-world supervision rather than the external process of constructing, simulating, diagnosing, and revising hypotheses.The benchmark omits much real-world physics, including camera motion, contact, collisions, fluids, and long-horizon multi-object dynamics.Source: §5.5 · p. 15

6 Related Work

The related work positions Code-as-World at the intersection of physical reasoning, executable world representations, and agentic discovery, targeting the gap of abstracting physical mechanisms from real-world observations.

Physical understanding and reasoning

Code as world representations

Agentic optimization and discovery

  • Agentic discovery has improved algorithms [22], automated experiments, robotics skills [98], and visual-environment representations [91] [93]; this work refines physical-world hypotheses through simulation and verification against language or visual evidence.Source: §6 · p. 16

7 Conclusion · Appendix

Code-as-World uses executable world representations and agentic discovery to construct and verify mechanism-grounded hypotheses, and these worlds can provide scalable supervision for quantitative physical reasoning. The conclusion identifies broader physical regimes and capabilities as directions for future work.

7 Conclusion

  • Code-as-World combines structured code and agentic discovery to construct, verify, and improve executable hypotheses that capture how physical worlds are composed, evolve, and can be manipulated.Verified executable worlds also provide scalable physical supervision for training vision-language models on quantitative physical reasoning.Source: §7 · p. 16

7.1 Broader Physical Phenomena

  • Although the implementation focuses on rigid-body dynamics, code can interface with simulators for fluids, deformable bodies, combustion, fracture, elasticity, plasticity, and gas dynamics [100].Future work must teach coding agents to invoke and compose simulators and develop evidence extraction and verification across physical regimes.Source: §7.1 · p. 16

7.2 Broader Physical Capabilities

  • Executable worlds could support grounded physical reasoning beyond question answering by supervising entity grounding, relation inference, interaction simulation, and self-verification against explicit states [1] [85] [86].Source: §7.2 · p. 16
  • Persistent executable states that maintain object identity, geometry, and temporal evolution could enable more coherent long-horizon video generation and controllable interventions [109].Separating world dynamics from visual rendering may improve physical consistency.Source: §7.2 · p. 16
  • Executable worlds could support deliberate embodied agents that model possible futures and candidate actions before acting, rather than relying solely on reactive policies [110].Source: §7.2 · p. 16
  • A shared abstraction across passive observations, simulation, and real-world interaction could transfer physical knowledge across settings and scale embodied learning beyond direct real-world experience [110].Source: §7.2 · pp. 16–17

A Dataset Details … B Experiment Protocols

The dataset details distinguish image-plane measurement supervision from World-Space VQA examples built from verified executable worlds, and describe their sources, construction, and sample counts.

A Dataset Details

  • Image-Space datasets provide direct image-plane measurement supervision, while the executable-world dataset provides synchronized videos, physical states, and World-Space VQA examples from verified EWRs.Source: A Dataset Details · p. 23

A.1 Image-Space Datasets

  • RefCOCO, RefCOCO+ [54], RefCOCOg [72], and RefCLEF annotations yield questions on object dimensions and relative image-plane position, alongside auxiliary box-grounding supervision.Source: §A.1 · p. 23
  • GOT-10K [74] clips contribute 16 uniformly sampled ordered frames for questions on pixel-unit size, displacement, speed, acceleration, and timestamp-specific grounding.Each question includes a short semantic description of the tracked object to identify the target.Source: §A.1 · p. 23
  • After evaluation-image overlap removal, the Image-Space training set contains 73,335 question–answer pairs.GOT-10K supplies 46,763 examples: 20,858 speed, 8,399 velocity, 12,169 acceleration, 3,995 grounding, and 1,342 size-related; the four referring-expression datasets supply 26,572, including 16,989 grounding and 9,583 size-related examples.Source: §A.1 · p. 23

A.2 World-Space Executable-World Dataset

  • The World-Space executable-world dataset unifies supervision from reviewed language specifications and real-video observations, with 1,585 text-driven and 988 video-driven VQA samples.Source: §A.2 · p. 23

B.1 Executable-World Fidelity and Realism Evaluation · B.2 Executable Engine Configurations · B.3 Image-Space Measurement Evaluation

The appendices specify executable-world evaluation across visual agreement, motion fidelity, and real-video realism, describe two interchangeable simulation engines, and define image-space measurement without world-scale calibration.

B.1 Executable-World Fidelity and Realism Evaluation

  • Visual Alignment combines silhouette overlap with available depth/RGB similarities, renormalizing active weights; Object IoU averages annotated object-frame mask overlaps and resolves visibility with the final z-buffer.Unavailable depth or RGB components are omitted, while silhouette remains active on every frame.Source: §B.1 · pp. 23–24
  • Trajectory fidelity compares simulator centers and CoTracker estimates on 16 sampled, 512-pixel-wide frames; it reports Traj-ADE (%D) and Velocity-ADE (%D/step), retaining absolute positions.Dataset scores weight executable worlds equally and average sim-to-real variants before the cross-world mean.Source: §B.1 · p. 24
  • JEDi [111] assesses video-feature distribution realism, while TRAJAN [113] measures track-based motion-statistic distance; lower scores indicate closer agreement with real videos.JEDi uses V-JEPA video features, and TRAJAN encodes BootsTAPIR point tracks.Source: §B.1 · p. 24

B.2 Executable Engine Configurations

  • MuJoCo supports interchangeable animation and physics engines behind the same executable-world interface, enabling evidence-engine pairings; realism-oriented re-rendering improves photorealism while preserving scene structure and rollout dynamics.The animation engine specifies time-varying poses and trajectories; the physics engine derives motion from forces, contacts, and constraints.Source: §B.2 · p. 25

B.3 Image-Space Measurement Evaluation

  • Image-Space evaluation measures boxes and central-difference motion directly in pixels, pixels per second, or pixels per second squared without world-scale priors; scalar MRA is averaged by benchmark, excluding grounding examples.Questions are generated only at timestamps where all required boxes are valid; the five dataset scores are reported in Table 3.Source: §B.3 · p. 25

B.4 World-Space QuantiPhy Evaluation … C.2 Additional Agentic Discovery Loop

The appendices specify World-Space QuantiPhy scoring and a separate reasoning-model evaluation, then report sim-to-real fidelity and iterative-discovery performance with a different physics engine.

B.4 World-Space QuantiPhy Evaluation

  • 159 QuantiPhy validation examples pair a monocular video and question with a physical prior, requiring inference of a scalar World-Space quantity under the official protocol.The open-source validation set contains question–answer pairs with visible ground-truth answers.Source: §B.4 · p. 25
  • The four evaluation subsets are 2S, 2D, 3S, and 3D, where numerals denote planar or depth-aware settings and letters denote static or dynamic source priors.The queried target may itself be static or dynamic; the subset is determined by the source prior and spatial setting.Source: §B.4 · p. 25
  • Mean Relative Accuracy scores predictions across ten increasingly strict relative-error thresholds, then reports an unweighted macro-average of the four subset scores.Unparseable or non-finite responses receive zero; reported main-paper MRA values are multiplied by 100.Source: §B.4 · pp. 25–26

B.5 Reasoning-Model Setting

  • Code-as-World-VL-27B (Reasoning) is a separate outcome-supervised variant, using the direct-answer models’ inputs without access to generating EWRs, simulator states, tracks, tools, retrieval, or ground-truth measurements.It receives 16 ordered frames, the question, unit, and benchmark physical prior, plus the same depth context for depth-aware questions.Source: §B.5 · p. 26
  • The reasoning variant is trained with GRPO, and reward and evaluation use only the final numerical answer rather than supervising or scoring the free-form reasoning trace.The reported Table 1 result uses a step-60 validation run with one sampled response per example, no voting or tool calls, and parsing after the final </think> tag.Source: §B.5 · p. 26

C.1 Text-Driven Sim-to-Real Evaluation

  • Generated videos are closer to authentic videos in video-feature and motion-feature distributions while retaining motion agreement comparable to simulator renders.The evaluation measures distributional realism and motion fidelity; the reported result indicates improved appearance without materially changing EWR-specified physical evolution.Source: §C.1 · pp. 26–27, Table 2 · p. 26

C.2 Additional Agentic Discovery Loop

  • By round five, the agentic discovery loop outperforms matched-budget Best-of-5 on all five metrics, with gains persisting under a different physics engine.Across five rounds, Visual Alignment, Object IoU, and Accuracy@2%D increase, while Traj-ADE and Velocity-ADE decrease.Source: Figure 11 · p. 27, §C.2 · p. 27

C.3 Image-Space Measurement Results

Code-as-World-VL combines image-space grounding and pixel-level measurement with World-Space supervision. The full models improve over their Image-Space variants on all five Image-Space benchmarks, while the two stages reinforce grounding and quantitative measurement.

C.3 Image-Space Measurement Results

  • The comparison covers object extent and grounding in images, plus object extent and motion in video, against open-weight models on four referring-expression datasets and GOT-10K.Table 3 reports the quantitative comparison.Source: §C.3 · p. 27, Table 3 · p. 28
  • Figure 12 shows Code-as-World-VL identifying intended targets and measuring length, speed, and acceleration close to ground truth, while comparison models make larger localization or numerical errors.The examples use pixel-space units specified by each question.Source: §C.3 · p. 27, Figure 12 · p. 29
  • After World-Space supervision, the full 4B and 9B models improve on all five Image-Space benchmarks over their corresponding Image-Space variants.Table 3 evaluates four referring-expression datasets and GOT-10K; the Image-Space variants use only the first training phase.Source: §C.3 · p. 28, Table 3 · p. 28
  • The Image-Space variants establish reliable visual grounding and pixel-level measurement as the perceptual basis for subsequent World-Space learning.Image-Space supervision grounds physical reasoning in observable measurements, while World-Space supervision feeds physical consistency back into grounding and quantitative measurement.Source: §C.3 · p. 28

C.4 Ablation of Data Sources

The ablation tests Image-Space measurement data alongside text- and video-driven executable worlds in a two-phase curriculum. Results show complementary benefits from both World-Space sources at 4B and 9B scales.

C.4 Ablation of Data Sources

  • The ablation uses compact 4B and 9B models, all first trained on Dpix before introducing Dtext and Dvideo separately or jointly.Source: §C.4 · p. 29
  • At 4B, combining text- and video-driven worlds with Image-Space data achieves the best macro-average, 50.6, showing complementary benefits.Adding either World-Space source improves the 4B Image-Space model.Source: §C.4 · p. 30, Table 4 · p. 30
  • At 9B, the full model improves from 50.9 to 56.8, supporting complementary contributions from simulator supervision and real-video alignment.Source: §C.4 · p. 30, Table 4 · p. 30

Related papers

Loading 2608.27549v1…