Source-linked AI summary
Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning
Hanyang Wang, Yimo Cai, Weiliang Chen, Jiawei Chi, Haowen Sun, Qiyu Dai, Yi-Hsin Hung, Xingzhuo Guo, Jinshan Ren, Runmao Yao, Ziwei Liu, Mingsheng Long, Yueqi Duan, Jun Gao, Jiangran Lyu, Fangfu Liu, Jialong Wu
TL;DR
Modern vision-language models can describe physical phenomena but lack explicit representations of the mechanisms needed for physical reasoning. Code-as-World represents physical composition, dynamics, and appearance as executable code and discovers such worlds through iterative proposal, execution, rendering, and verification. Verified executable worlds provide scalable supervision, and Code-as-World-VL achieves state-of-the-art performance on QuantiPhy while outperforming leading proprietary models.
Problem
Modern vision-language models can recognize and explain physical events but often lack explicit object states, physical parameters, and governing dynamics for reasoning about world evolution and interventions.
Method
Code-as-World expresses physical composition, dynamic evolution, and visual appearance as executable code, while an agent proposes, executes, renders, verifies, and refines world hypotheses from multimodal evidence.
Results
Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and outperforms leading proprietary models using verified executable worlds as physical supervision.
Takeaways & Limitations
Verified executable worlds provide scalable physical supervision for training vision-language models on quantitative physical reasoning.
Takeaways & Limitations
QuantiPhy covers only a limited subset of physical understanding, focusing on monocular scale calibration under relatively constrained motion settings.
Abstract
from arXiv · showhide
Physical understanding and reasoning depend on forming compact and generalizable representations of the world. While modern vision-language models can recognize and explain diverse physical events, they often lack explicit representations of the underlying mechanisms-such as object states, physical parameters, and governing dynamics-needed for reliably reasoning how the world evolves and responds to interventions. In this work, we introduce Code-as-World, a paradigm that represents physical worlds through executable world representations. By expressing physical composition, dynamic evolution, and visual appearance as executable code, Code-as-World provides a compact, quantitatively grounded, and controllable abstraction of the physical world. To construct such representations from multimodal observations, such as natural-language descriptions or real-world videos, we develop an agentic discovery loop inspired by abductive reasoning, where an agent proposes, executes, renders, verifies, and iteratively refines executable world hypotheses. As a concrete application, we use verified executable worlds to provide scalable physical supervision for training vision-language models on quantitative physical reasoning. Experiments show that Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and surpasses leading proprietary models, highlighting the potential of executable world representations as a scalable foundation for physical intelligence.
1 Introduction
Code-as-World addresses the gap between describing physical phenomena and representing the mechanisms that govern them. It introduces executable world representations and an agentic discovery loop, then uses verified worlds to supervise quantitative physical reasoning.
- Physical intelligence requires representations of object states, physical parameters, governing dynamics, and responses to interventions, beyond visible descriptions.
- Code-as-World represents physical composition, dynamic evolution, and visual appearance as compact, quantitatively grounded executable code.
- The agentic discovery loop proposes, instantiates, executes, renders, and verifies world hypotheses against language or visual evidence.
- Verified executable worlds provide scalable physical supervision for vision-language models performing quantitative physical reasoning.
- Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and outperforms leading proprietary models.
2 In Search of Physical World Representations
Existing physical-world representations preserve different aspects of observations but do not alone combine semantic understanding, structural composition, temporal prediction, and explicit physical mechanisms. Code is presented as a complementary representation that integrates these properties through an executable, quantitatively grounded abstraction.
- Pixels: Pixel prediction can produce visually plausible futures without explicitly representing the causes of visual changes or ensuring physical correctness.
- 3D: 3D reconstruction preserves geometry, viewpoint, and appearance, but reconstructability does not necessarily reveal the causal factors governing physical behavior.
- Natural language: Language captures compact, compositional semantic abstractions but struggles to encode continuous physical states, precise geometry, trajectories, contacts, and parameters.
- Pixels, 3D representations, and language offer complementary detail, geometric structure, and semantic abstraction, respectively.
- The proposed representation should integrate semantic compactness, structural organization, and temporal evolution while explicitly capturing the mechanisms underlying the world.
3 Code as Worlds: Executable World Representations
An executable world representation organizes task-relevant physical structure as code while separating underlying state from visual presentation. Its components can be inspected, modified, and executed to support simulation, verification, reasoning, and data generation.
- Code-as-World represents physical worlds through executable code rather than solely pixels, latent features, or natural-language descriptions.
- The representation separates structured physical state from continuous appearance, preserving compositionality, editability, explicit constraints, visual detail, and uncertainty.
- Executable world representations are coupled with programmatic calls to a physical simulation engine.
- Physical composition: Physical composition specifies entities, geometry, metric dimensions, and properties such as mass, friction, and gravity.
- Dynamic evolution: Dynamic evolution specifies initial states, temporal changes, events, and simulation duration, producing complete state trajectories during execution.
- Visual appearance: Visual appearance specifies cameras, backgrounds, materials, lighting, frame rates, resolutions, and rendering configurations without altering the underlying physical process.
- Different representation components can be independently inspected, modified, and executed while preserving the remaining structure.
4 Agentic Discovery of World Representations
Code-as-World recovers executable world representations from text or video through an iterative propose–instantiate–execute–render–verify process. The resulting worlds support synchronized physical states, controllable resimulation, and improved reconstruction quality under a matched evaluation budget, while remaining limited by simulator coverage.
- 4.1 Evidence Construction: Code-as-World formulates recovery from partial, heterogeneous observations as agentic discovery over executable world representations rather than direct prediction.The agent searches for hypotheses consistent with semantic or visual evidence while favoring parsimonious explanations.
- 4.1 Evidence Construction: Text and video adapters convert inputs into semantic or visual evidence that constrains a shared executable world representation space.Text processing extracts entities, relations, events, and outcomes, whereas video processing uses depth, masks, tracks, and generated object meshes.
- 4.2 Agentic Discovery Loop: Each iteration proposes or updates p = (C, E, A), compiles it into simulator parameters θ, executes a trajectory τ, renders observations, and verifies discrepancies against selected evidence.The trajectory records object states, contacts, collisions, and event outcomes; verification compares semantic and physical constraints for text or RGB, depth, masks, and trajectories for video.
- 4.3 Results: A shared simulation interface produces synchronized videos and physical states across both input modalities, enabling executable worlds to support downstream analysis and generation.Examples show semantic descriptions becoming physically grounded videos and authentic visual observations becoming executable worlds.
- 4.3 Results: At a matched five-evaluation budget, the agentic loop outperforms Best-of-5 independent sampling on most reconstruction aspects, including Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D.Editable representations also permit changes to physical quantities, initial conditions, and camera configurations while preserving specified physical evolution in realistic videos.
5 Application: Learning Quantitative Physical Reasoning
The application trains vision-language models for quantitative physical reasoning using image-space measurements and verified executable-world supervision. Code-as-World-VL improves quantitative reasoning while preserving and strengthening image-space grounding, but evaluation remains limited in physical scope and the discovery loop stays external to the model.
- 5.1 Problem Formulation: Quantitative physical reasoning asks VLMs to infer measurable quantities such as size, displacement, velocity, and acceleration from monocular videos.World-space questions use a reference quantity for metric calibration, while 3D settings additionally use depth information.
- 5.2 Image-Space Measurement: Pixel-level supervision converts visual annotations and object tracks into questions about extent, position, displacement, velocity, and acceleration.Bounding boxes and masks provide extent and position, while tracks provide motion quantities.
- 5.2 Image-Space Measurement: The first curriculum phase teaches localization, measurement, and tracking, providing the visual foundation for subsequent world-space physical reasoning.The model is trained on the resulting pixel-level dataset with supervised fine-tuning.
- 5.3 World-Space Physical Calibration from Verified Executable Worlds: Verified executable worlds provide synchronized videos and simulated state trajectories for generating quantitative question-answer pairs with exact world-space labels.Text-driven and video-driven worlds are combined into unified world-level training data and optimized with GRPO using numerical, unit, and format rewards.
- 5.3 World-Space Physical Calibration from Verified Executable Worlds: Text-driven worlds supply exact simulator supervision, while video-driven worlds better match real observations; joint training combines physical accuracy with visual generalization.The two sources provide complementary benefits for world-level training.
- 5.4.1 Experimental Setup: Both 4B and 9B Image-Space variants perform strongly, and complete Code-as-World-VL models improve over them on every benchmark.The models are trained with a two-phase curriculum, with 4B, 9B, and 27B reasoning variants evaluated under uniformly sampled video frames.
- 5.4.3 Quantitative Physical Reasoning: Code-as-World-VL-9B achieves the best average performance among direct-answer variants, while Code-as-World-VL-4B outperforms larger open-weight baselines and remains competitive with leading proprietary systems.The gain is consistent across benchmark subsets and is reported as more reliable connection of visual measurements with metric scale and motion.
- 5.5 Limitations: QuantiPhy covers only monocular scale calibration for four quantities under relatively constrained motion, while Code-as-World-VL does not internalize agentic world discovery.Hypothesis construction, simulation, diagnosis, and iterative revision remain external to the model.
6 Related Work
Related work represents physical worlds through increasingly rich spatial, physical, code-based, and agentic abstractions. These lines of research motivate executable representations that can support physical content, interaction, and iterative discovery.
- Physical understanding and reasoning: Physical-understanding research spans spatial reasoning and physical question answering over object properties, interactions, and measurable quantities.These approaches study geometric structures as well as physical reasoning from images and videos.
- Code as world representations: Code-based world models use executable programs as environments that can be generated, modified, and evaluated through interaction.Related work also investigates code-based representations for physical content creation and simulation.
- Agentic optimization and discovery: Agentic optimization systems iteratively propose, evaluate, and refine external artifacts, including algorithms, academic experiments, and robotics skills.This line of work provides the broader context for agentic discovery of executable representations.
7 Conclusion
The paper presents executable code as a mechanism-grounded representation for physical worlds and uses verified worlds to supervise quantitative physical reasoning. It also outlines extensions to broader reasoning, video generation, embodied interaction, and physical regimes.
- 7 Conclusion: Code-as-World represents physical worlds through executable code that captures composition, dynamics, and appearance.The representation is positioned as a bridge between visual observations and mechanism-grounded reasoning.
- 7 Conclusion: Verified executable worlds provide scalable physical supervision for training vision-language models on quantitative physical reasoning.The paper applies this supervision to measurable physical quantities and quantitative reasoning tasks.
- 7.1 Broader Physical Phenomena: The current implementation focuses primarily on rigid-body dynamics, while broader physical regimes require new simulation composition and evidence-verification capabilities.Suggested regimes include fluids, deformable bodies, combustion, fracture, elasticity, plasticity, and gas dynamics.
- 7.2 Broader Physical Capabilities: Future directions include grounded physical reasoning, physically consistent video generation, and deliberate embodied interaction.These directions involve explicit world states, persistent object properties, possible futures, and candidate actions.
Authors
The paper lists Hanyang Wang, Yimo Cai, Weiliang Chen, Jiawei Chi, Haowen Sun, Qiyu Dai, Yi-Hsin Hung, Xingzhuo Guo, Jinshan Ren, Runmao Yao, Ziwei Liu, Mingsheng Long, Yueqi Duan, Jun Gao, Jiangran Lyu, Fangfu Liu, and Jialong Wu as authors.
- Authors: The author list begins with Hanyang Wang, Yimo Cai, Weiliang Chen, and Jiawei Chi.
- Authors: The middle of the author list includes Haowen Sun, Qiyu Dai, Yi-Hsin Hung, Xingzhuo Guo, Jinshan Ren, Runmao Yao, and Ziwei Liu.
- Authors: The author list concludes with Mingsheng Long, Yueqi Duan, Jun Gao, Jiangran Lyu, Fangfu Liu, and Jialong Wu.
Affiliations
The listed affiliations are MirroS, Tsinghua University, Peking University, and Nanyang Technological University.
- Affiliations: MirroS is listed as one of the paper’s affiliations.
- Affiliations: Tsinghua University and Peking University are listed among the affiliations.
- Affiliations: Nanyang Technological University is listed among the affiliations.
A Dataset Details
The dataset details distinguish Image-Space measurement supervision from World-Space supervision built from verified executable worlds. Evaluation covers visual alignment, object and trajectory fidelity, and video realism, while training uses image- and video-derived quantitative questions.
- Dataset construction: The Image-Space datasets provide direct image-plane measurements, whereas the executable-world dataset provides synchronized videos, physical states, and World-Space VQA examples.The executable-world dataset contains 1,585 text-driven and 988 video-driven VQA samples.
- Dataset construction: Image- and video-based examples ask about object dimensions, positions, displacement, speed, acceleration, and grounding.Image examples derive measurements from referring-expression boxes, while video examples use 16 sampled frames and tracked bounding boxes.
- Evaluation: The evaluation measures agreement with source observations, object-mask overlap, trajectory fidelity, and proximity to real-video distributions.Visual Alignment and Object IoU assess visual agreement; Traj-ADE and Velocity-ADE assess trajectories, while JEDi and TRAJAN assess video realism and motion realism.
- Execution: MuJoCo supports interchangeable animation and physics execution engines under a shared executable-world interface.The animation engine specifies kinematic poses and trajectories, while the physics engine derives motion from forces, contacts, and constraints.
- Image-Space evaluation: Image-Space scalar targets remain in raw pixels, pixels per second, or pixels per second squared without object-size priors or world-unit scale.Sample-level MRA is averaged independently within each benchmark, while grounding outputs are evaluated separately.
B.4 World-Space QuantiPhy Evaluation
The World-Space evaluation measures quantitative physical reasoning on QuantiPhy and examines model scope, training, inference, reconstruction, and sim-to-real fidelity. It also evaluates iterative discovery and complementary grounding results.
- Evaluation setup: QuantiPhy contains 159 monocular-video question–answer pairs requiring scalar World-Space quantities from a question and physical prior.
- Evaluation setup: The benchmark reports four kinematic subsets: 2S, 2D, 3S, and 3D, defined by spatial setting and source-prior type.
- Metrics: Mean Relative Accuracy averages sample scores across increasingly strict relative-error thresholds, then computes an unweighted macro-average across the four subsets.Unparseable non-finite responses receive zero credit, and main-paper MRA values are multiplied by 100.
- Reasoning-model setting: Code-as-World-VL-27B receives benchmark inputs but has no test-time access to generating EWRs, simulator states, tracks, tools, retrieval, or ground-truth measurements.
- Sim-to-real generation: Sim-to-real videos improve overall video-feature and motion-feature distributional realism while retaining motion agreement comparable to simulator renders.The evaluation separates distributional realism from motion fidelity and preserves simulator-specified physical evolution.
- Image-Space measurement: World-Space supervision improves all five Image-Space benchmarks over corresponding Image-Space variants, linking observable measurement with physical consistency.
C.4 Ablation of Data Sources
The ablation tests Image-Space measurement data alongside text-driven and video-driven executable worlds. Adding either World-Space source helps, while combining both yields the strongest reported averages.
- Data sources: The two-phase curriculum uses Image-Space measurement data Dpix, text-driven executable worlds Dtext, and video-driven executable worlds Dvideo.All variants first use Dpix before introducing World-Space sources separately or jointly.
- Ablation results: 50.6 is the best 4B average when both World-Space sources are combined after Image-Space training.Adding either World-Space source improves the 4B Image-Space model, while the combined sources provide complementary benefits.
- Ablation results: 56.8 is the full 9B model’s improved average, rising from 50.9 under the same data-source ablation.
- Interpretation: The ablation indicates that exact simulator supervision and real-video alignment contribute complementary signals beyond Image-Space grounding.The comparison reports MRA on QuantiPhy subsets and their macro-average.