PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis

Shuo Lu, Yinuo Xu, Kecheng Yu, Siru Jiang, Yongcan Yu, Yubin Wang, Haitao Yang, Yuxiang Zhang, Bin Wang, Ran He, Jian Liang

arXiv:2606.01869v2cs.AI

TL;DR

Existing web-generation benchmarks and evaluators largely observe pixels or DOM elements, while important Three.js runtime behavior remains hidden. WORLDCODER-BENCH pairs expert-curated executable tasks with STATEPROBE, which checks hidden behavioral contracts. Across nine frontier models, the best reported coverage is 27.8% on WORLDCODER-CORE and 19.9% on WORLDCODER-ROBUST.

  • Problem

    Three.js worlds expose only rendered pixels to screenshots and DOM inspection, while their physics, animation, and interaction logic determine whether they work correctly.

    Source: §1 · p. 1

  • Method

    WORLDCODER-BENCH provides executable Three.js tasks, and STATEPROBE checks generated programs’ runtime states and transitions against hidden, mutation-hardened contracts.

    Source: §1 · p. 2

  • Results

    27.8% Verification Coverage on WORLDCODER-CORE and 19.9% on WORLDCODER-ROBUST are the best system’s results across nine frontier models.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Failures lie mainly in executable behavior, especially state-schema drift and broken interaction chains, rather than missing scene elements.

    Source: §5 · p. 9

    The paper identifies preserving exact action-synchronous runtime-state contracts and improving physical and algorithmic reasoning for the long tail of executable 3D worlds as open challenges.

    Source: §4.5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly asked not only to write static interfaces, but to construct executable interactive worlds from natural language. Browser-native 3D, commonly built with Three.js, is a natural next frontier: generated programs must integrate assets, obey spatial and physical constraints, and keep user-facing controls synchronized with hidden runtime state. Existing web-generation benchmarks and evaluators, however, largely observe only pixels or DOM nodes, while the mechanics of a Three.js world unfold inside an opaque <canvas>. We introduce WorldCoder-Bench, a benchmark for autonomous, physically grounded 3D world synthesis. WorldCoder-Bench contains 2,026 expert-curated tasks across Simulation, Rendering, and Application scenarios, with optional .glb assets and hidden behavioral contracts. We further propose StateProbe, an execution-based protocol that probes generated programs in a sandboxed browser and verifies hidden, mutation-hardened contracts over runtime states and transitions. Beyond verification coverage, we report Return on Automation and Time Efficiency Multiplier to measure correctness-adjusted cost and time savings. Across nine frontier models, the best system reaches only 27.8% verification coverage on WorldCoder-Core and 19.9% on WorldCoder-Robust, with failures dominated by state-schema drift and broken interaction chains rather than missing scene elements. Utility metrics further show that cheap or fast models can still provide substantial value on easier domains. WorldCoder-Bench is available at https://anonymous.4open.science/r/WorldCoder-Bench/.

1 Introduction

Browser-native 3D generation requires worlds to work, not merely look right, yet their runtime behavior is hidden inside the Three.js canvas. WORLDCODER-BENCH and STATEPROBE target this evaluation gap with executable tasks and hidden behavioral contracts.

1 Introduction

  • As LLMs move from static interfaces to executable applications, browser-native 3D is a next step involving simulations, scientific visualizations, configurable products, and games.The paper argues that visual plausibility is not a useful proxy for whether a world works correctly.Source: §1 · p. 1
  • Three.js worlds render into a single canvas, leaving the geometry, physics, animation, and interaction logic that determine correctness invisible to screenshots and DOM inspection.Existing benchmarks under-test physical correctness, asset integration, and state synchronization in generated 3D worlds.Source: §1 · p. 1, Figure 2 · p. 3
  • 2,026 expert-curated tasks span Simulation, Application, and Rendering, including primitive-only scenes and asset-dependent worlds with optional .glb resources.Models must produce a single HTML page that loads, renders, and responds to user actions in a standard browser.Source: §1 · p. 2
  • STATEPROBE runs generated programs in headless Chromium, snapshots runtime state around scripted actions, and checks state changes against hidden, mutation-hardened contracts.Contracts are tested against injected defects such as deleted state updates, scaled physical constants, and swapped event targets.Source: §1 · p. 2
  • Across nine frontier models, no system exceeds 30% Verification Coverage, while even the strongest external evaluation paradigm misclassifies 45.6% of severely defective outputs, underscoring the necessity of STATEPROBE.The benchmark introduces Return on Automation and Time Efficiency Multiplier, task-level utility metrics combining inference cost and latency with public developer labor rates and discounting cheap-but-incorrect outputs.Source: §1 · pp. 2–3

2 WORLDCODER-BENCH

WORLDCODER-BENCH evaluates executable Three.js worlds against hidden behavioral contracts, covering diverse tasks, assets, and runtime behavior. Its curation and split design target rigorous evaluation and mechanism-level generalization.

2 WORLDCODER-BENCH

  • WORLDCODER-BENCH evaluates autonomous, physically grounded 3D world synthesis by requiring models to generate self-contained Three.js programs from natural-language specifications, targeting physical correctness, asset integration, and state synchronization.STATEPROBE is an execution-based state-verification protocol that scores generated worlds against hidden behavioral contracts.Source: §2 · p. 3

2.1 Task Formulation

  • Each task pairs a natural-language instruction with optional assets; models produce one self-contained Three.js HTML file evaluated through hidden runtime assertions.Models see neither the contracts nor the action sequences, thresholds, or test scripts.Source: §2.1 · p. 3

2.2 Data Curation

  • Curation proceeds from expert seeds through LLM expansion and filtering to runtime validation, expert-authored contracts, and randomized variants.The final 2,026 canonical tasks exclude unstable or unevaluable candidates; variants perturb physical constants, object counts, initial states, and asset choices.Source: §2.2 · pp. 3–4, Figure 3 · p. 4

2.3 Dataset Composition and Splits

  • Tasks are organized as Simulation, Rendering, or Application according to user intent and the runtime contracts needed for dynamics, visual states, or interaction logic.CORE has 205 hidden tasks; ROBUST adds three randomized variants per CORE task, 615 in total, to probe mechanism-level generalization.Source: §2.3 · pp. 4–5
  • Difficulty levels D1–D6 span basic initialization to multi-step logic, strict physical constraints, and state synchronization; asset tiers distinguish primitive-only, single-.glb, and multi-asset tasks.Source: §2.3 · p. 5

3 Evaluation

STATEPROBE addresses the limits of external observation by checking generated worlds through runtime state and hidden behavioral contracts. Its protocol combines mutation-hardened assertions, scripted actions, and coverage and utility metrics.

3.1 From External Observation to State Verification

  • Screenshots, VLMs, DOM checks, and visual exploration cannot recover key hidden runtime quantities; STATEPROBE reads state, applies controlled actions, and checks task conditions.Three.js worlds render inside a canvas, leaving the DOM with only peripheral elements such as buttons or labels.Source: §3.1 · p. 5

3.2 Mutation-Hardened Contracts

  • Expert-authored contracts specify required affordances, states, transitions, tolerances, and task-specific invariants, then mutation testing checks whether they reject injected failures.Contracts that fail to reject mutations are revised or the task is discarded.Source: §3.2 · p. 5

3.3 Execution Protocol

  • STATEPROBE executes generated programs in headless Chromium, applies deterministic scripted actions, and checks hidden assertions against before-and-after runtime-state snapshots.The interface schema is visible, but action sequences, thresholds, expected transitions, and assertion scripts remain hidden.Source: §3.3 · pp. 5–6, Figure 5 · p. 6

3.4 Metrics

  • Affordance, State, and Transition Coverage diagnose failures, while Verification Coverage is the primary leaderboard metric and system diagnostics include crashes, missing outputs, cost, and latency.Source: §3.4 · p. 6
  • Return on Automation measures quality-adjusted human-labor value per model-cost dollar, while Time Efficiency Multiplier measures human-hours saved per model-generation hour.Both metrics discount outputs that fail behavioral verification.Source: §3.4 · pp. 6–7

4 Experiments

The experiments compare nine frontier models on behavioral correctness, robustness, practical utility, and failure modes. Although models often produce plausible scenes, their hidden-contract performance remains limited, especially on action-driven behavior.

4.1 Setup

  • The evaluation benchmarks nine proprietary and open-weights models zero-shot under a shared task and asset format, scoring outputs against hidden mutation-hardened contracts.Capability is reported as V-Cov and its Affordance, State, and Transition components; practical utility uses RoA and TEM, with a human-labor estimator anchored to seven expert annotations.Source: §4.1 · p. 7, §4 · p. 7

4.2 Performance on WORLDCODER-CORE

  • GPT-5.4 reaches 27.8% V-Cov on WORLDCODER-CORE, and the top four models cluster at 23.0–27.8%.No system exceeds 30%; the proprietary–open gap is 2.5 points, with Qwen3.6-Plus at 25.3%.Source: §4.2 · p. 7
  • A-Cov is 40.6–67.4% and S-Cov 30.7–46.1%, while T-Cov is only 15.8–26.3%, locating many failures at transitions.External evaluators do not reproduce the contract-based ranking: DOM-only has τb=−0.021, Screenshot+VLM τb=−0.068, and an inspection agent τb=+0.111 while passing 45.6% of low-V-Cov outputs.Source: §4.2 · pp. 7–8, Table 2 · p. 8

4.3 Robustness on WORLDCODER-ROBUST

  • On WORLDCODER-ROBUST, GPT-5.4 leads V-Cov at 19.9% and T-Cov at 18.3%, while Qwen3.6-Plus falls from 25.3% to 13.5%.The proprietary–open gap widens to 6 points; the robust split perturbs task parameters and assets.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 Practical Benefits

  • DeepSeek-V4-Flash leads RoA at $19,150/$, while GPT-5.4 leads TEM at 68.9×; Materials, Shaders, and Postprocess top both metrics across models.The top five of 15 domains contribute 55–65% of total TEM, while Complex System, Creative, and Graphics yield near-zero RoA for the three plotted models.Source: §4.4 · pp. 8–9, Figure 6 · p. 8

4.5 Error Analysis

  • State Schema Drift and Broken Interaction Chain account for 42.8% and 40.8% of zero-T-Cov failures, respectively.Crash rates are 3.4% for GPT-5.4 and 2.0% for Gemini; Physics Violation is 0.4% and Semantic Misunderstanding 2.7% in this subset.Source: §4.5 · p. 9, Figure 7 · p. 9

5 Conclusion

The paper presents a benchmark and execution-based protocol for evaluating behaviorally correct Three.js worlds, reporting low verification coverage and failures concentrated in runtime behavior. It positions this work against benchmarks that rely on external observations of 3D scenes.

5 Conclusion

  • Across nine frontier models, the best reaches 27.8% Verification Coverage on WORLDCODER-CORE and 19.9% on WORLDCODER-ROBUST.The main failures involve state-schema drift, broken interaction chains, and runtime state synchronization; utility metrics also show value in easier domains.Source: §5 · p. 9

A.1 Benchmarks for Web, Frontend, and Game Code Generation

  • Existing web, frontend, and game-code benchmarks cannot reliably determine physical correctness in browser-native 3D scenes because key state is encapsulated inside the canvas and rendering or physics engine.The cited benchmark scope includes screenshots, DOM structures, browser actions, page assertions, Godot projects, and 2D Pygame environments.Source: §A.1 · p. 13

A.2 3D Evaluation Paradigms

Prior 3D evaluation uses visual assessment or physics-oriented tasks, but neither directly evaluates browser-native Three.js code generation. The paper proposes probe-first verification of runtime state and interactions.

A.2 3D Evaluation Paradigms

  • Visual 3D evaluation assesses rendered geometry and plausibility, while physics-oriented benchmarks test physical understanding through motion prediction, puzzles, or conservation constraints.Examples include multi-view rendering and video observation for visual evaluation, and intuitive-physics or mechanics tasks for physics-oriented evaluation.Source: §A.2 · p. 13
  • Probe-First Evaluation exposes a standardized runtime-state interface, captures before-and-after engine snapshots, and checks interaction transitions against physical and task-specific assertions.Mutation testing injects controlled faults to test evaluator detection and calibrate task-level confidence.Source: §A.2 · p. 13

B Dataset Details and Statistics

The benchmark contains 2,026 canonical tasks organized by user intent and fine-grained domain. Its main results use a 205-task hard subset selected for low reference-model transition coverage.

B.1 Top-Level Statistics and Domain Taxonomy

  • WORLDCODER-CORE contains 205 hidden tasks and is the hard main-results subset selected for low reference-model T-Cov.Eighty percent of CORE tasks are difficulty D5 or D6, and 25 tasks (12%) require external .glb assets.Source: Table 4 · p. 14, §B.1 · p. 14
  • The 2,026-task benchmark is organized into Simulation, Rendering, and Application macro-categories, with 15 fine-grained domains retained for diagnostic analysis.The taxonomy follows the primary intent of observing system evolution, controlling visual presentation, or completing an interactive goal.Source: Table 5 · p. 14, §B.1 · p. 14

B.2 Dataset Splits and Difficulty Distribution

The appendix describes the benchmark splits and WORLDCODER-CORE’s domain–difficulty distribution, then lists protections against leakage of task contracts and hidden references.

B.2 Dataset Splits and Difficulty Distribution

  • CORE’s difficulty annotations use D3–D6; easier D1/D2 tasks were filtered out by bottom-T-Cov sampling.Application and Simulation tasks dominate the D5–D6 tail, while creative and graphics rendering tasks concentrate at D4–D5.Source: §B.2 · p. 14, Table 7 · p. 15
  • WORLDCODER-CORE, WORLDCODER-EXTENDED, and WORLDCODER-DEV are subsets of the 2,026 finalized canonical tasks.Table 6 lists four released hidden/public splits.Source: Table 6 · p. 15

B.3 Anti-Leakage Protections

  • Anti-leakage protections keep behavioral contracts and hidden-split references private, while randomizing task parameters and asset files across variants.Tasks are expert-authored rather than reproductions of public Three.js tutorials or example galleries.Source: §B.3 · p. 15

C Extended Experimental Results

The appendix details how per-task API cost, latency, and estimated human effort support RoA and TEM accounting, and examines sensitivity to developer hourly rates.

C.1 Cost / Time Accounting and Hourly-Rate Sensitivity

  • RoA and TEM in Table 1 are computed from per-task evidence rather than aggregate estimates.The accounting uses cost, latency, and human-effort components.Source: §C.1 · p. 16
  • Per-call API cost uses logged prompt and completion tokens with public May-2026 rates, except DeepSeek-V3.2 uses a documented V4 Pro promotional-price proxy.Table 8 lists the rates used.Source: §C.1 · p. 16, Table 8 · p. 16
  • Per-call latency is the generation pipeline’s measured end-to-end time; median latency ranges from 79 s for Claude Sonnet 4.6 to 244 s for DeepSeek-V3.2.Latencies are converted to hours for accounting.Source: §C.1 · p. 16
  • Human effort uses expert-authored estimates where available and a difficulty-conditioned fallback otherwise; the seven explicit estimates range from 120–480 min.Their mean is 284 min and median is 280 min; the fallback is intentionally conservative.Source: §C.1 · p. 16, Table 9 · p. 16
  • Changing the hourly rate scales every model’s RoA equally, preserving rankings, while TEM remains unchanged.Practitioners can rescale Table 1 RoA values by R/60.Source: §C.1 · p. 16, Table 10 · p. 17

C.2 Full Model Leaderboard with System-Level Diagnostics

The full leaderboard adds crash, missing-probe, token, and latency diagnostics to WORLDCODER-CORE results; domain-level patterns show that Simulation performance varies substantially across models.

C.2 Full Model Leaderboard with System-Level Diagnostics

  • DeepSeek-V4-Flash crashes on 42.4% of tasks and Claude Sonnet 4.6 on 13.7%, despite the latter having V-Cov comparable to other mid-tier models.GPT-5.4 and Gemini-3.1-Pro each keep RUNTIME_CRASH below 4%.Source: §C.2 · p. 17, Table 11 · p. 17
  • PROBE_MISSING reaches 7.3% for Claude Opus 4.6 and 10.2% for Kimi-K2.5; both models omit the runtime-state interface in those runs.RoA and TEM require at least 90% report coverage, so Kimi-K2.5 and MiniMax-M2.7 report only V-Cov columns.Source: §C.2 · p. 17, Table 11 · p. 17
  • Median completion-token counts span 5,418 for Claude Sonnet 4.6 to 15,843 for DeepSeek-V4-Flash, primarily explaining why RoA inverts the main-text V-Cov ranking.Table 11 also reports median end-to-end API latency.Source: §C.2 · p. 17, Table 11 · p. 17
  • Materials, shaders, and postprocess carry Rendering for every model, creative is universally zero, and game is harder than every Simulation domain except complex_system.Simulation shows the greatest cross-model variance: molecular ranges 6.0–31.9% and soft_body 5.0–32.3%, pointing to physics and analytic correctness rather than static rendering as the source of adjacent-model V-Cov gaps.Source: §C.2 · p. 17, Table 12 · p. 18

C.3 Per-Domain Economic Profile (All Seven Models)

Per-domain results show consistent strengths and weak tails in economic metrics across seven models, while external evaluation paradigms can distort model rankings relative to runtime-state verification.

C.3 Per-Domain Economic Profile (All Seven Models)

  • Materials, Shaders, and Postprocess rank in the top five for per-domain RoA and TEM across all seven models, while Complex System, Creative, and Graphics form a near-zero-RoA tail.The two open-weights DeepSeek variants reach roughly $22,000–$33,000 RoA peaks per API dollar, versus roughly $2,000–$6,600 for the other five models; GPT-5.4 and Gemini-3.1-Pro reach per-domain TEM peaks of roughly 140–150× versus roughly 40× for DeepSeek panels.Source: §C.3 · p. 18, Figure 8 · p. 18

C.4 Per-Model Paradigm Comparison

  • DOM-only’s aggregated Kendall τb is −0.238 across seven models, and DeepSeek-V3.2’s DOM score of 54.7 exceeds GPT-5.4’s 50.0 despite lower V-Cov values of 21.9 and 27.8, respectively.Pure Agent also gives the highest score to DeepSeek-V4-Flash, the second-weakest model by V-Cov.Source: §C.4 · p. 19, Table 13 · p. 19

D.1 Failure-Mode Breakdown by Model and Domain

Failure patterns are broadly consistent across models, while domain-specific interaction and UI demands shape where other errors cluster. Representative cases show that apparently correct scenes can still fail runtime contracts, and the broader-impact discussion describes diagnostic uses and sandbox mitigations.

D.1 Failure-Mode Breakdown by Model and Domain

  • The breakdown classifies 596 zero-T-Cov records across nine models by primary and secondary failure modes, with domain tables counting primary causes only.An LLM grader classifies generated HTML and contract traces.Source: §D.1 · p. 19
  • Physics Violation averages 1% across models, peaking at 5% for DeepSeek-V4-Flash; Semantic Misunderstanding occurs in only 0–2%.Physics tasks more often fail from incorrect field names or unwired keyboard listeners than from the integration scheme.Source: §D.1 · p. 19
  • State Schema Drift occurs in 93–98% and Broken Interaction Chain in 88–98% of records across models; proprietary frontier models are not safer on these modes.The comparison covers zero-T-Cov records, with behavioral-mode columns counting primary and secondary causes non-exclusively.Source: §D.1 · p. 19, Table 14 · p. 20
  • Physics and visualization failures are at least 90% Drift-driven, while Chain rises in chemistry (39%), animation (26%), and game tasks (25%).Missing UI Elements peaks in molecular (41%), soft_body (36%), and chemistry (35%).Source: §D.1 · p. 19, Table 15 · p. 20

D.2 Representative Case Walk-through

  • Five outputs on P253 exhibit distinct failures, including asset-path errors, sandbox escape attempts, absent rendering, a missing global reference, and frame-rate-dependent physics.The contract tests phase transitions, projectile trajectory, and synchronization of score, shots, and last-shot result; screenshots or DOM inspection would miss these failures.Source: §D.2 · p. 22

E Broader Impact

  • The authors frame the benchmark as diagnostic for catching state-level defects before deployment and run generated programs in a network-locked Chromium sandbox.They also identify possible misuse of realistic interactive 3D worlds for deceptive simulations or disinformation.Source: E Broader Impact · p. 25

Related papers

Loading 2606.01869v2…