PaperlayerView exact revision on arXiv

Source-linked AI summary

ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling

Xinge Peng, Zhen Fang, Ruibing Wang, Yongxuan Lv, Haodong Li, Guohui Zhang, Xiaoxiao Ma, Yiting Lu, Lin Chen, Zehui Che, Feng Zhao

arXiv:2610.15166v1cs.CV

TL;DR

Existing benchmarks assess physical reasoning, visual or geometric fidelity, and behavior, but provide limited evidence about integrating world understanding with executable realization. ExeWorldBench evaluates that integration across three progressively challenging tasks, and experiments reveal substantial limitations in executable world modeling. The authors hope the benchmark encourages progress toward physically coherent executable worlds.

  • Problem

    Existing benchmarks provide fragmented assessments of physical reasoning, visual or geometric construction, and behavior, leaving the integration of world understanding and executable realization insufficiently evaluated.

    Source: §1 · pp. 1–2

  • Method

    ExeWorldBench evaluates executable 3D world modeling through mechanism reconstruction, multi-stage causal composition, and intent-driven synthesis.

    Source: §1 · p. 2

  • Results

    Experiments show that high code executability does not necessarily translate into strong structural, mechanical, or causal correctness, and performance generally declines as causal complexity and execution horizon increase.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors hope ExeWorldBench encourages progress toward models that construct physically coherent executable worlds.

    Source: §5 · p. 10

Abstract

from arXiv · show

Coding agents are increasingly evolving beyond reproducing what a world looks like toward constructing executable worlds that capture how a world works. This shift calls for executable world modeling, which integrates two core capabilities: world understanding, reasoning about the underlying structural, mechanical, and causal principles of a world, and executable realization, translating that understanding into a code-based system whose behavior emerges from physical interactions. However, existing benchmarks provide only fragmented evaluations centered on physical reasoning, visual fidelity, or behavioral correctness, leaving the integration of these capabilities insufficiently assessed. To bridge this gap, we introduce ExeWorldBench, to our knowledge the first benchmark to comprehensively evaluate world understanding and executable realization through code-based 3D world construction. ExeWorldBench comprises three progressively challenging tasks: Mechanism Executability, which evaluates the realization of individual mechanisms from visual observations; Compositional Executability, which extends to multi-stage systems with spatial, mechanical, and causal dependencies; and Intent-Driven Executable World Synthesis, which requires constructing executable worlds from high-level goals and physical requirements. Extensive experiments on frontier models reveal substantial limitations in executable world modeling, offering insights into the development of more physically grounded and causally coherent world construction systems.

1 INTRODUCTION

Executable world modeling requires both understanding a world’s structure, mechanics, and causal principles and realizing that understanding in executable code. ExeWorldBench evaluates this integration across three increasingly autonomous construction tasks, while experiments reveal substantial gaps between code executability and physically coherent world construction.

1 INTRODUCTION

  • Existing benchmarks assess physical reasoning, visual or geometric construction, and behavior separately, providing limited evidence that models integrate understanding with executable realization.Behavioral tests may involve simple or isolated motions, and plausible trajectories can sometimes be reproduced without implementing the underlying mechanisms.Source: §1 · pp. 1–2
  • Executable 3D world modeling integrates reasoning about a world’s geometric, mechanical, and causal organization with translating that understanding into an executable code-based system.World understanding alone does not guarantee faithful realization, and runnable code alone does not show that the world was correctly understood.Source: §1 · pp. 1–2
  • ExeWorldBench evaluates executable 3D world modeling through Mechanism Executability, Compositional Executability, and Intent-Driven Executable World Synthesis.The tasks progress from visual-observation-based mechanism reconstruction to multi-stage systems and high-level textual specifications.Source: §1 · p. 2
  • Frontier models show substantial gaps between code executability and physically coherent world construction, so runnable code alone does not establish strong executable-world modeling.The benchmark motivates tighter integration of world understanding and executable realization.Source: §1 · p. 2

2 RELATED WORK

Prior code-based world modeling and benchmark work covers specification-guided construction, observation-based discovery, executable representations, geometry, and dynamics. The paper characterizes these capabilities and evaluations as fragmented, with limited assessment of coherent 3D mechanisms whose interactions produce complex causal behavior.

Approaches to Code-Based World Modeling

  • Code-based world modeling includes specification-guided construction, persistent executable representations, and observation-based discovery, but the paper describes existing methods as offering fragmented capabilities.The paper distinguishes these approaches from executable world modeling, which requires both world understanding and translation into executable code.Source: §2 · p. 3

Benchmarks for Code-Based World Modeling

  • Existing benchmarks assess physical specifications, browser-native environments, executable geometry, dynamics reconstruction or prediction, and simulation-code adaptation.The paper names PhysCodeBench, SimuScene, WorldCoder-Bench, 3DCodeBench, BenchCAD, VisPhyWorld, MPMWorlds, and PACE-Bench across these areas.Source: §2 · p. 3
  • The paper says prior benchmarks mainly emphasize behavior, geometry, parameter recovery, or dynamics adaptation, offering limited evaluation of coherent 3D mechanisms with interacting geometry, constraints, contacts, and physical interactions.These mechanisms jointly produce complex causal behavior.Source: §2 · p. 3

3 EXEWORLDBENCH

ExeWorldBench progresses from reconstructing individual mechanisms to composing multi-stage systems and synthesizing worlds from intent. Its data pipelines select or construct tasks and verify that their motion, causal connections, or specifications are sufficiently inferable and feasible.

3.1 TASK FORMULATION

  • The benchmark progresses from visual reconstruction of individual mechanisms, through multi-stage causal composition, to synthesis from high-level textual specifications.Its data construction pipeline filters real mechanisms, composes systems from atomic components, and creates component- and goal-constrained synthesis tasks with feasibility verification.Source: §3.1 · p. 3, Figure 1 · p. 4
  • For Mechanism Executability, a model must infer and realize a mechanism’s complete motion from a reference image, despite coupled component interactions and limited motion observations.The model must infer constraints, contacts, and motion propagation from structural evidence rather than follow sampled motion frames.Source: §3.1 · p. 3
  • Compositional Executability requires linking multiple mechanisms into a causal chain, including reasoning about stage-to-stage effects and arranging components so each stage can connect to and trigger the next..Source: §3.1 · p. 3
  • Intent-Driven Executable World Synthesis removes visual references; its component-constrained setting supplies components, while its goal-constrained setting leaves mechanical realization to the model.The latter specifies goals, objects, states and events, initial states, physical rules and dependencies, and final states.Source: §3.1 · pp. 3–4

3.2 DATA CONSTRUCTION

  • For Mechanism Executability, selected public animations show identifiable interacting components and motion cycles inferable from visible static structure; degraded, occluded, or ambiguous cases are filtered.A visual quality model and GPT-6 Astra support filtering, followed by manual review of retained samples.Source: §3.2 · p. 4
  • Mechanism motion inferability is checked by generating image-grounded evidence, asking a model without motion-sequence access to infer motion, validating against reference motion, and manually reviewing retained samples.Only dynamics consistent with the reference motion pass the cross-modal validation step.Source: §3.2 · pp. 4–5

3.2.2 T2: COMPOSITIONAL EXECUTABILITY

  • Compositional Executability uses a library of atomic components to generate ordered systems of 4 to 13 steps, then manually reviews transitions and assembles reference animations from the predefined implementations.The assembly process primarily organizes the components and adjusts their spatial layout.Source: §3.2.2 · p. 5

3.2.3 T3: INTENT-DRIVEN EXECUTABLE WORLD SYNTHESIS

  • Intent-driven synthesis samples 4 to 12 components for component-constrained tasks and generates goal-constrained specifications at three difficulty levels, retaining tasks that pass feasibility and solution checks.Retained tasks are also manually reviewed for specification consistency, mechanical feasibility, and solution validity.Source: §3.2.3 · p. 5

4 EVALUATION

Across ExeWorldBench’s evaluation, runnable code does not ensure structurally, mechanically, or causally sound worlds. Performance also generally declines as causal complexity and execution horizon increase.

4.1 METRICS

  • Frontier models are evaluated in API-based and harness-based settings using penalized metrics that include execution failures and assess structure, motion, causal execution, and goal satisfaction.Goal satisfaction evaluates initial state, intermediate behavior, and final state.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 CODE EXECUTABILITY

  • Harness-based models reach nearly 100% executability on most tasks, yet high executability coexists with only modest gains in world-modeling metrics after repair.The gap indicates that runnable Blender code is necessary but insufficient for structurally valid and physically coherent worlds.Source: §4.2 · p. 6

4.3 T1: MECHANISM EXECUTABILITY.

  • On T1, Fable 5.1 scores V = 4.04 but M = 3.24, illustrating that strong structural fidelity does not guarantee accurate mechanical motion.Models with strong motion scores also achieve high structural scores, but the reverse does not always hold.Source: §4.3 · p. 6, Table 1 · p. 7

4.4 T2: COMPOSITIONAL EXECUTABILITY

  • On T2, Seed-2.1 ranks fifth in T1 Mechanical Motion but seventh in T2 ECM after repair, showing that multi-stage execution requires more than individual motion.Table 2 reports Visual Structure and Executable Causal Mechanism for T2.Source: §4.4 · p. 6, Table 2 · p. 7
  • On T3-1, only harness-based models achieve relatively strong ECM after constrained repair; with high-level intent alone, models perform substantially worse.Among API-based models on T3-1, GPT-6 Astra scores 3.34 and Fable 5.1 scores 2.75; all remaining models score at most 0.40. The added difficulty of T3-2 suggests that inferring a suitable mechanism is harder than organizing and connecting predefined components.Source: §4.4 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8
  • Across API-based models, initial-state requirements are satisfied more reliably than intermediate-behavior and final-state requirements.Models often instantiate a plausible initial configuration but struggle to sustain the required causal process through to the intended outcome.Source: §4.4 · p. 8

4.6 LONG-HORIZON ANALYSIS

  • On T2 and T3-1, performance declines as execution stages increase; GPT-6 Astra (Codex CLI) shows the smallest drop, while GPT-6 Astra falls below 1.0 ECM at eight T3-1 steps.On T2, GPT-6 Astra (Codex CLI) remains comparatively stable; GPT-6 Astra and Fable 5.1 (Claude Code) fall to roughly one-fifth of their short-horizon scores at 13 steps.Source: Figure 2 · p. 9, §4.6 · p. 9
  • Motion references modestly improve T1 performance for all models except GPT-6 Astra; on T2, most benefit, while GLM-5.3-Flash and GPT-6 Astra decline marginally.Overall improvements remain limited.Source: Figure 3 · p. 9, §4.7 · p. 9, Figure 4 · p. 10

5 CONCLUSION

ExeWorldBench evaluates executable 3D world modeling across mechanism execution, causal composition, and intent-driven synthesis; results show that runnable code does not ensure structural, mechanical, or causal correctness.

5 CONCLUSION

  • ExeWorldBench evaluates executable 3D world modeling across mechanism execution, multi-stage causal composition, and intent-driven synthesis.The authors report that correctness generally declines as causal complexity and execution horizon increase.Source: §5 · p. 10
  • The authors used generative AI for benchmark data construction and evaluation and manuscript polishing, but not research ideation, core methodological design, or scientific conclusions.The authors reviewed AI-assisted outputs and take responsibility for the final paper.Source: AI USE STATEMENT · p. 10

A.1 DATA CONSTRUCTION DETAILS

The appendix reports component usage statistics for T2.

A.1 DATA CONSTRUCTION DETAILS

  • Tab. 5 presents component usage statistics for T2.Source: §A.1 · p. 13

A.2 MORE RESULT

The appendix reports T3-2 goal satisfaction across initial state, intermediate behavior, and final state, alongside conditioned API-model results and component usage statistics.

A.2 MORE RESULT

A.3 RESULT VISUALIZATIONS

The appendix shows representative qualitative results for four tasks across harness-based and API-based models. Its T3-2 specification tests priority and sequencing in a shared region.

A.3 RESULT VISUALIZATIONS

  • Figures 6–9 show representative results for T1, T2, T3-1, and T3-2, contrasting the best-performing harness-based model with API-based models at high, medium, and low performance levels.Source: §A.3 · p. 15, Figure 6 · p. 16, Figure 7 · p. 17
  • T3-2 requires payload B to start before payload A enters, while A enters and crosses first; A’s clearance must mechanically release already-moving B without payload contact or simultaneous shared-region overlap.Source: §A.3 · pp. 20–21
  • The T3-2 specification requires the shared-region boundary to be declared before simulation and both payloads and the mechanism to meet defined final states.Each payload must remain inside its own target, and object_A must remain in state_A1 during the final hold.Source: §A.3 · pp. 20–21

Related papers

Loading 2610.15166v1…