PaperlayerView exact revision on arXiv

Source-linked AI summary

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos

Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou, Lingjun Mao, Ruobing Han, Jinglin Cao, Wenpeng Xu, Lukun He, Minghao Fu, Fan Feng, Biwei Huang

arXiv:2610.04432v3cs.CVcs.AIcs.RO

TL;DR

Conventional Real-to-Sim pipelines require substantial engineering effort, and existing benchmarks provide fragmented evidence about agents’ ability to construct interactive worlds. Video2World evaluates this process end to end, finding that agents can build executable simulations but reliable task completion remains difficult and geometric fidelity does not determine success.

  • Problem

    Real-to-Sim pipelines require substantial engineering effort, while existing benchmarks provide only fragmented evidence about agents’ ability to integrate capabilities for autonomous interactive-world construction.

    Source: §1 · pp. 1–2

  • Method

    Video2World formulates video-to-simulation as an end-to-end coding task and evaluates 222 reconstructions from 189 videos across geometric fidelity, dynamic fidelity, and functional correctness.

    Source: §1 · pp. 2–3

  • Results

    Across nine frontier coding-agent systems, executable worlds are possible but reliable task completion remains challenging, and stronger geometric accuracy does not necessarily correspond to task success.

    Source: §1 · p. 3, §4.2 · p. 10

  • Takeaways & Limitations

    Interactive world reconstruction is a joint scene-and-behavior problem, so evaluation beyond individual fidelity metrics is warranted.

    Source: §6 · p. 12

    Functional correctness evaluates the reconstructed scene and agent-generated behavior jointly, so a failed rollout may arise from the world, the behavior, or their interaction.

    Source: G Limitations and Future Work · p. 79

Abstract

from arXiv · show

Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate \emph{autonomous video-to-simulation} as a software engineering task in which an agent observes an embodied video, constructs the corresponding simulated environment and robot behavior, and iteratively refines the result through execution feedback. To evaluate this capability, we introduce \textbf{Video2World}, a benchmark comprising 222 reconstruction instances derived from 189 robot and human demonstration videos. Video2World measures reconstructed worlds along geometric fidelity, dynamic fidelity, and functional correctness, capturing spatial perception, physical reasoning, and executable interaction. Evaluating 9 frontier coding-agent systems reveals a sharp improvement in Task success beginning with Claude Opus 5, rising from below 5\% to over 15\%, while substantial gaps to human-assisted reconstruction remain. We further find that worlds that look better could work worse: better visual fidelity does not always lead to higher task success. This echoes the broader gap between perceptual realism and factual correctness observed in generative models.

1 Introduction

Real-to-Sim reconstruction remains labor-intensive, while existing benchmarks offer only fragmented evidence about whether agents can integrate the capabilities needed to build interactive worlds. Video2World frames this as an end-to-end coding task and evaluates reconstruction fidelity and task execution.

1 Introduction

  • Interactive simulation can scale embodied data, but conventional Real-to-Sim pipelines still require substantial engineering effort to reconstruct and calibrate environments.Existing benchmarks often assess isolated capabilities rather than whether agents can integrate them to build, execute, and refine interactive worlds.Source: §1 · pp. 1–2
  • Agents must construct an environment and robot behavior from embodied video, then refine both through execution feedback.The benchmark pairs videos with simulator environments and aligned configurations, states, and interaction trajectories.Source: §1 · p. 2, Figure 1 · p. 3
  • Video2World comprises 222 reconstruction instances from 189 videos and measures geometric fidelity, dynamic fidelity, and functional correctness.The three perspectives assess scene structure, physical evolution, and successful task execution.Source: §1 · p. 3
  • Across nine frontier coding-agent systems, executable worlds are possible but reliable task completion remains challenging, and stronger geometric accuracy does not necessarily mean greater task success.Human-assisted reconstruction is a stronger reference, but it also falls short of perfect task completion.Source: §1 · p. 3

2 Preliminary: Autonomous Video-to-Sim via Coding Agents

Autonomous video-to-simulation asks a coding agent to turn an embodied video into an executable environment and robot behavior. The agent iteratively revises both using simulator execution feedback.

2 Preliminary: Autonomous Video-to-Sim via Coding Agents

  • Given an embodied video, target simulator, and general-purpose tools, the agent produces a simulation program specifying an environment and robot behavior.The program is evaluated through a rollout of simulated states and robot actions.Source: §2 · p. 4
  • The rollout follows the policy’s actions through simulator dynamics, and functional correctness means reproducing the demonstrated task through robot execution.The simulator dynamics are instantiated by the reconstructed environment.Source: §2 · p. 4
  • The agent inspects observations and resources, edits and executes its program, then uses feedback to revise the environment and behavior until submission or budget exhaustion.The submitted program is fixed for evaluation.Source: §2 · p. 4

3 Video2World: Benchmarking Coding Agent Video-to-Sim

Video2World pairs embodied videos with target simulator configurations and evaluates autonomous reconstruction through a standardized agent protocol. Its object-centric annotations and complementary metrics support comparison across embodiments, reconstructed scenes, physical motion, and task execution.

3 Video2World: Benchmarking Coding Agent Video-to-Sim

  • Video2World is a benchmark for end-to-end coding-agent reconstruction of interactive worlds from embodied videos.The benchmark collects videos and corresponding world assets and annotations, standardizes the agent protocol, and defines evaluation metrics.Source: §3 · p. 4

3.1 Property Construction

  • The benchmark contains 189 clips forming 222 reconstruction instances, with object-centric annotations for scene geometry, object motion, and task requirements.Annotations record object shape, scale, articulation, spatial relations, motion and state changes, as well as task roles, initial conditions, events, goals, and milestones.Source: §3.1 · pp. 4–5
  • Object-centric alignment supplies shared task-level targets across human and robot demonstrations while allowing embodiment-specific trajectories.Reference behavior for each target robot is executed under contact physics; on 40 human-assisted instances from 35 videos, six effector-motion statistics had Spearman correlations of 0.74–0.99, including 0.80 for net rotation.Source: §3.1 · pp. 5–6

3.2 Agentic Protocol Design

  • The standardized protocol controls agent resources, reconstruction budget, and execution setting while leaving each system’s internal construction strategy autonomous.Agents work in a sandbox with videos, simulator APIs, documentation, assets, and coding tools; evaluators execute submitted environments and behaviors under assigned settings.Source: §3.2 · p. 6

3.3 Evaluation Metrics Design

  • Evaluation reports executability, geometric fidelity, dynamic fidelity, and functional correctness, with task success and milestone-based task progress measuring demonstrated interaction.Geometry includes scene- and object-level reconstruction; dynamic fidelity measures physical execution against observed object motion, accounting for rotation, symmetry, and deformable objects.Source: §3.3 · pp. 6–8
  • Dynamic metrics compare reconstructed and demonstrated object motion while allowing different control strategies to realize the same physical interaction.The translation discrepancy is measured relative to each trajectory’s initial state, removing global translation offsets while preserving motion magnitude and temporal evolution.Source: §3.3 · pp. 7–8

4 Experiments

Experiments assess autonomous video-to-simulation across coding-agent systems and find that executable worlds often fall short of reliable task completion. Geometric fidelity, functional success, and motion correspondence provide related but distinct views of reconstruction quality.

4.2 Main Results

  • Lower geometric errors are generally associated with higher Task Success, but accurate geometry does not guarantee success and some functional reconstructions differ spatially from the source.GPT-6 Astra has the strongest object-shape fidelity, whereas Fable-5.1 completes more tasks; geometry and functionality should therefore be evaluated separately.Source: Figure 4 · p. 9, §4.2 · p. 11

4.1 Experimental Setup

  • The experiment evaluates nine coding-agent models, each making one autonomous attempt per instance with simulation and rendered-rollout feedback under a 180-minute limit.Human-assisted reconstruction is included as a reference; evaluation aggregates applicable metrics by task family with equal weight.Source: §4.1 · p. 9

4.2 Main Results

  • Successful simulated reference executions show broadly similar effector-motion characteristics to source demonstrations across duration, path length, peak speed, lift height, displacement, and rotation.The comparison concerns human-assisted reference instances selected for task success in the original evaluation.Source: Figure 5 · p. 10, §4.2 · p. 11
  • Coding agents often build executable worlds but remain unreliable at full task completion, and stronger object-shape accuracy does not necessarily yield better task performance.Task Progress consistently exceeds Task Success; GPT-6 Astra leads in object shape and size accuracy, while Fable-5.1 performs better on scene-level and task measures.Source: §4.2 · p. 10
  • Human-assisted reconstruction leads the reported automatic metrics and human preference, yet does not achieve uniformly reliable task completion.Successful human-assisted rollouts also show that target tasks are feasible under the benchmark configuration.Source: §4.2 · p. 10

5 Related Work

Related work spans real-to-sim reconstruction, foundation-model agents for embodied tasks, and benchmarks for coding and scene construction. Video2World is situated among approaches that generate or refine executable environments and behaviors from observations.

Real-to-Sim Reconstruction for Robot Manipulation

  • Real-to-sim methods reconstruct environments or object models for robot learning, while generative systems create tasks, environments, and robot behaviors [Torne et al., 2024, Han et al., 2026, Jiang et al., 2025, Wang et al., 2024a,b].Video2Policy and Agentic Real-to-Sim are especially related for video-based task reconstruction and executable episode twins [Ye et al., 2025, Chen et al., 2026a].Source: §5 · p. 11

Foundation-Model Agents for Embodied Tasks

  • Foundation-model agents connect semantic reasoning with embodied action through planning, code generation, and execution feedback [Ahn et al., 2022, Driess et al., 2023, Liang et al., 2023, Huang et al., 2023, Wang et al., 2023].Examples include executable programs for perception and control APIs and program revision through feedback.Source: §5 · p. 11

Benchmarks for Coding and Agentic Construction

  • Coding and agentic-construction benchmarks cover repository work, embodied tasks, 3D modeling, scene operations, and physical-scenario code generation [Jimenez et al., 2024, Fu et al., 2026, RLE-Bench contributors, 2026, You et al., 2026, Gao et al., 2026, Zhao et al., 2026, Wang et al., 2026].BVB is particularly related in evaluating video reconstruction as animated Blender scenes through spatiotemporal facts and perceptual similarity.Source: §5 · pp. 11–12

6 Conclusion

Video2World evaluates interactive-world reconstruction from embodied videos across geometric, dynamic, and functional dimensions. Its interface fixes the simulator and embodiment while agents infer the scene and behavior from permitted video observations and public resources.

6 Conclusion

  • Video2World frames video-to-simulation as an end-to-end software-engineering task and evaluates geometric fidelity, dynamic fidelity, and functional correctness.The paper concludes that coding agents can often build executable simulations, but reliable completion remains challenging; geometric fidelity does not fully determine functional correctness, and human-assisted reconstruction remains imperfect.Source: §6 · p. 12

A Environment and Agent Interface

  • A benchmark instance fixes the video, target embodiment, execution backend, and task brief; the submitted scene and robot behavior form an executable package interpreted under the target simulator's fixed controls and rules.The agent supplies a finite control-reference sequence and interval, which may be revised during construction but is frozen for final evaluation.Source: A Environment and Agent Interface · p. 16

A.1 Public Inputs and Information Boundary

  • Agents infer task objectives, object roles, and interaction sequences from video without additional task descriptions or success criteria; reference geometry, calibration, trajectories, controls, and benchmark scores are withheld.The public-input protocol restricts episode-specific access and sets limits on video-frame sampling and image observations for OpenCode configurations.Source: §A.1 · pp. 17–18

A.2 Simulation Configurations and Control Conventions

This section documents the benchmark’s simulation and evaluation setup, including configuration-dependent controls, agent instructions and execution feedback, candidate construction and model procedures, and the organization of videos, target configurations, hidden annotations, and task families.

A.2 Simulation Configurations and Control Conventions

  • Camera registration and control semantics vary by interface: some camera-frame submissions use withheld evaluator registration, while native RoboDojo and household submissions estimate camera-to-world transforms.The 222 instances include two human-video configurations that use the same 33 source clips with different target embodiments. Local replay checks are not universal benchmark admission thresholds and do not certify benchmark outcomes.Source: §A.2 · pp. 18, 20
  • Execution uses configuration-specific clocks, initialization, waits, and settling; dexterous hands run force-limited at 300 Hz, while native RoboDojo and household interfaces use distinct command and physics rates.Parallel-gripper commands use one for open and zero for closed, whereas household grippers use zero for open and one for closed.Source: §A.2 · pp. 19–20

A.3 Submission, Validation, and Execution

  • Submissions package scene and control specifications; structural validation checks the assigned contract but does not certify task success, which is judged from executed states rather than declared trajectories.The evaluator executes its own controls, and candidate object-pose predictions cannot drive bodies or replace measured rollouts.Source: §A.3 · pp. 21–22

A.4 Agent Tools and Public Operations

  • Agents can inspect video, replay their own physical execution, validate, checkpoint, and submit; these operations expose neither reference annotations nor benchmark scores, and acceptance is distinct from task success.A validated protected copy becomes the accepted candidate, preventing subsequent workspace edits from changing it.Source: §A.4 · p. 23, Table A5 · p. 24

A.5 Construction Budget and Feedback

  • Construction feedback is limited to the candidate’s own scene, controls, renders, and diagnostics; managed episodes allow 10,800 s and a $60 observed API-cost threshold before a selected package is frozen for separate evaluation.Delayed usage reporting may cause billed cost to exceed the threshold, and there is no fixed total number of model turns or tool calls.Source: §A.5 · pp. 24–26

A.6 Agent Initialization and Model Configurations

  • Trials begin from the selected video, embodiment, interface, and model configuration; seven configurations use OpenCode, while Astra uses Codex CLI and Fable uses Claude Code.A fresh agent session and persistent workspace are used during construction, after which the selected candidate is transferred to a separate evaluation process.Source: §A.6 · p. 27
  • Native coding-agent sessions use a 180-minute hard limit and select the last successful submission or a preserved checkpoint on timeout.Unlike the benchmark-managed OpenCode runs, their local submission service does not enforce the complete staged-deadline and refinement policy; the OpenCode generation cutoff is 165 minutes.Source: §A.6 · p. 28

A.7 Task Prompts

  • The prompts direct agents to inspect physical rollouts, rendered observations, and execution diagnostics, then revise and deliver their own reconstruction without access to reference annotations or scores.Task-specific briefs prescribe different scene and control conventions for Cartesian robot control, paired hands, native dual-arm environments, and rope or soft-object tasks.Source: §A.7 · pp. 28–30

B Benchmark Construction and Reference Annotations

  • Benchmark construction pairs each demonstration with a target robot and simulator, while hidden annotations define evaluated geometry, object motion, and interaction outcomes.Annotation provenance distinguishes source measurements, estimates fitted to observations, and reconstructed physical targets; the human-assisted reference is evaluated against the same applicable targets as agent submissions.Source: B Benchmark Construction and Reference… · p. 31

B.1 Data Sources and Video Selection

  • The inventory contains 189 source clips and 222 reconstruction instances across 39 task families, each pairing a video with a target simulator, embodiment, and control interface.Sources span robot and human demonstrations and reconstructed digital-twin demonstrations; the 33 human-demonstration clips each have both arm and bimanual-hand configurations.Source: §B.1 · p. 31
  • Video selection retains the initial configuration, task-defining interaction, and available terminal context, with furniture windows capped by assembly step and leg-insertion step.Retained-window ordering prioritizes reference execution outcome, then agreement with the official terminal assembly pose, then a deterministic sample identifier; unsuccessful selected references remain in the inventory.Source: §B.1 · p. 31

B.2 Task Organization and Dataset Composition

  • The 39 task families are fixed evaluation groups, with each reconstruction instance linked through a manifest to its source sequence, clip boundaries, entities, and annotation version.Family labels preserve task and embodiment or annotation variants rather than merging tasks that share a broad action label.Source: §B.2 · pp. 32–33

B.3 Hidden Evaluation Specifications

Each instance has a source-derived or explicitly reconstructed specification linking persistent object identities to geometry, demonstrated motion, and task requirements. Annotations distinguish observed quantities from inferred ones and define task conditions from the source interaction.

Evaluation specifications

  • The specification links persistent object identities to geometry, motion, and task requirements, while provenance, uncertainty, and visibility masks identify evidence and unsupported intervals.Human-assisted reference outputs are experimental submissions, not replacements for source observations.Source: §B.3 · p. 33

Entities and scene geometry

  • Scene records include metric dimensions, initial poses, passive articulation, and relevant entities, while inferred supports or cavities remain marked as inferred.Evaluation geometry and simulator geometry are distinguished when representation changes such as collision decomposition or articulation conversion are made.Source: §B.3 · pp. 33–34

Demonstrated object motion

  • Motion annotations preserve source-specific visibility and validity conditions, coordinate conventions, and time-indexed surfaces without assuming material-point correspondence or a unique rigid rotation.Rigid and articulated records include positions, orientations, and relevant joint coordinates when available; rope and soft-object targets may use time-indexed particles or surfaces.Source: §B.3 · pp. 34–35

Task conditions

  • Required events attach to the manipulated object and a single attempt; native rules may additionally require returning the robot to its home configuration.Incidental grasp choice and exact source robot motion are generally unrestricted, while release and stability require recorded evidence when specified.Source: §B.3 · p. 35

Illustrative annotation

  • Figure A2’s bottle-transfer annotation separates the source-derived height trajectory from terminal placement conditions, which require transfer, release, goal pose, support, and rest.The example uses 87 frames at 15 Hz, with pickup and release annotated at clip indices 15 and 80; the receiving support’s lateral extent is estimated from observations.Source: §B.3 · p. 35, Figure A2 · p. 36

Task requirements and initial conditions

  • Task conditions derive from observed object relations and events, with clip boundaries determining whether pickup or release is required and initial satisfaction alone earning no placement credit.A placement already satisfied initially must be left and newly completed within the same attempt, including required release and stability conditions.Source: §B.3 · pp. 35–36

B.4 Worked Example: From Video Input to Scored Execution

A saved bottle-uprighting reconstruction succeeds under functional evaluation despite measurable geometric and temporal discrepancies. The worked example traces how source video, submitted controls, execution records, and hidden annotations combine in scoring.

Worked example setup

  • The example re-scores an unchanged saved GPT-6 Astra submission and recorded rollout, rather than reporting a new agent construction or aggregate model result.The input is an 83-frame, 640×640 RGB clip at 7.5 Hz showing a smaller bottle picked up, turned upright, and released on a support.Source: §B.4 · pp. 36–37

Annotations and task specification

  • The hidden reference matches the moving green bottle to a human-reviewed fit and evaluates its 83 annotated poses, support model, timestamps, and withheld camera registration.The active task requires an upright, supported bottle followed by release and rest; the terminal tilt limit is 30° from the support normal.Source: §B.4 · p. 37

Recorded execution and scoring

  • Motion fidelity compares the rollout against 83 demonstration timestamps, while the functional decision also uses 41 later execution states containing release and terminal completion.A longer successful interaction can therefore still incur temporal motion error.Source: §B.4 · p. 37

Scoring procedure

  • Scoring combines the validated package and build receipt with executed poses and events, hidden source geometry and motion, camera registration, and the active task specification.The submitted object-pose sequence is a declaration rather than an animation command; physical execution records object states, gripper state, grasp/release events, and rest.Source: §B.4 · pp. 37–38

Final scores and retained evidence

  • Build and Task Success are both 1, Task Progress is 100%, and the recorded interaction succeeds despite a 7.77 cm mean translation discrepancy.The figure’s motion comparison excludes the shaded tail, but functional evaluation still uses the later release and terminal state.Source: Figure A3 · p. 38, §B.4 · p. 39
  • Scene CD, Shape CD, Size, and Centre score 2.51, 3.58, 5.76, and 24.80 cm, respectively, while rotation APE and terminal axis error are 9.27° and 1.78°.Terminal displacement error is 4.57 cm; the centre error retains initialized-placement discrepancy, so low registered Scene CD does not imply accurate absolute placement or dimensions.Source: §B.4 · p. 39

B.5 Human-Assisted Reference

The human-assisted reference (HAR) is a source-assisted reconstruction and robot-engineering comparison, not an estimate of unaided human reconstruction from RGB. Its reported success and effort records have important scope limits because workflows used privileged information, human refinement, and differing execution conditions.

Reference construction

  • HAR is developed per instance through scene construction, robot-control design, and iterative physical testing using source geometry, motion annotations, robot logs, and existing assets.The selected reference includes both the reconstructed environment and the robot behavior executed in it.Source: §B.5 · p. 39

Information access

  • Reference workflows can use source annotations, simulator states, task predicates, evaluation scores, and source-specific resources that coding agents do not receive.The references therefore measure source-assisted reconstruction and robot engineering rather than unaided reconstruction from RGB.Source: §B.5 · pp. 39–40, Table A10 · p. 40

Recorded effort and coverage

  • The furniture campaign covers 64 instances with 128 attempt records, while the audited human-video contact-search batch covers 20 of 66 arm/hand configurations.Records count simulator trials, not manual editing iterations or total historical effort; several source histories lack consolidated attempt counts.Source: §B.5 · pp. 40–41

Feedback and selection

  • Furniture and human-video reference development used evaluation feedback, while records for reused native or source executions do not fully document score consultation.The authors make no claim that any source’s reference development was evaluation-blind.Source: §B.5 · p. 41

Interpretation of the reference score

  • HAR achieves 58.8% Task Success, macro-averaged equally across 39 task families and all 222 benchmark instances.This characterizes selected source-assisted reconstruction outcomes, not isolated human visual-reconstruction ability or performance under coding agents’ information and construction budgets.Source: §B.5 · p. 41

Comparison scope

  • The comparison evaluates complete workflows with differences in information access, human refinement, and execution conditions; isolating construction-method effects would require matched conditions.Reference success is neither an annotation-acceptance condition nor a guaranteed upper bound, and the methods execute their own behavior in their own scenes.Source: §B.5 · pp. 42–43

B.6 Human Evaluation

The human evaluation compares holistic faithfulness of agent and HAR reconstructions through pairwise judgments, with ratings fit using a Bradley–Terry model. Its inference is limited to the observed cases and internal rater pool, and does not establish significant pairwise differences from point-estimate rankings.

Evaluation target

  • The study asks raters which reconstruction more faithfully captures the scene and interaction, considering layout, task semantics, motion, and visible physical plausibility rather than task success alone.Geometric simplification or a different robot embodiment does not automatically imply failure, and visual detail alone should not determine preference.Source: §B.6 · p. 43

Case sampling and coverage

  • The 195-case pool was sampled into 54 cases across 30 task families, then comparison coverage admitted 83 of 135 planned comparisons, with all ten systems connected.The remaining 52 comparisons were deferred without replacement by higher-quality or successful examples; actual coverage was less balanced than planned.Source: §B.6 · p. 44

Responses and quality checks

  • The 243 valid judgments cover 81 comparisons across 39 cases and 19 task families; 68 of 243 judgments (28.0%) rated both candidates poor.The 83 comparisons yielded 249 primary judgments, including six unable-to-judge responses excluded from the valid set.Source: §B.6 · pp. 44–45

Rater consistency

  • Across 81 valid comparisons, all three raters agreed in preference direction on 31 when equivalent and both-poor responses counted as ties.Repeat-answer consistency was 8/12, and no rater was excluded on that basis.Source: §B.6 · p. 45

Human Preference Rating

  • A Bradley–Terry model fits all 243 valid judgments, assigning half a win and half a loss to equivalent and both-poor responses; the mean HPR is 1000.Each judgment has unit weight, without additional case or family equalization; HPR is a batch relative-preference estimate, not task-success percentage.Source: §B.6 · p. 45

Interpretation and scope

  • The 95% intervals resample source-video clusters, not raters, and point-estimate rankings are not used to infer significant pairwise differences.The study measures relative holistic faithfulness under its observed case coverage and internal participant pool; HAR also uses privileged information and human refinement.Source: §B.6 · p. 45

B.7 Annotation Verification and Evaluator Refinement

This section establishes source-anchored evaluator review and a shared scoring contract for reconstructed geometry, executed motion, and task completion. It defines task-specific progress and success verification and explains how evaluator outcomes and uncertainty are aggregated and reported, distinguishing diagnostic progress and endpoint measures from complete task success and geometric fidelity from interaction success.

B.7 Annotation Verification and Evaluator Refinement

  • Review corrected two success decisions and three progress disagreements; revised bottle predicates accepted two upright, supported, released reconstructions despite separate geometric discrepancies.The revisions were grounded in task-defining events, including actual reorientation, and applied uniformly; the documented cases do not estimate benchmark-wide error prevalence.Source: §B.7 · pp. 46–47, Figure A5 · p. 47
  • 36 saved executions showed that requiring a new placement changed all three already-solved-start judgments to failure and progress 0.25; the other 33 judgments were unchanged.The original endpoint-only rule had assigned full success and progress to all three sequences.Source: §B.7 · p. 46

C Evaluation Details

  • Video2World scores initialized geometry, executed object motion, and functional completion separately, treating each method’s behavior in its own scene as part of the reconstruction.Robot trajectories provide execution evidence, not motion-matching targets.Source: C Evaluation Details · p. 48

C.1 Geometric Fidelity

  • Geometric fidelity measures scene correspondence, object shape, size, placement, and support separately, with interaction-weighted scene distances and bounded registration.Shape CD aligns rigidly without scaling, while initial placement is measured in the common frame.Source: §C.1 · pp. 48, 50

C.2 Dynamic Fidelity

  • Dynamic fidelity compares executed object motion with the demonstration using profile-specific temporal correspondence, without temporal warping; short completed executions use endpoint holding for motion scoring.States beyond the demonstration window remain available for terminal task evaluation, and motion coverage is reported.Source: §C.2 · p. 51

C.3 Functional Correctness

  • Functional correctness checks admissible initial conditions, essential interaction events, and a persistent terminal goal rather than requiring the demonstrator’s exact motion.In the shared rigid placement profile, an initially satisfied goal requires leaving that relation and completing a new placement within the attempt.Source: §C.3 · pp. 53–54
  • Progress uses task-profile-specific stages or native graded scores, so full progress is not a universal certificate of Task Success.Rope, soft-toy, and cloth profiles use distinct movement, proximity, lifting, or folding indicators; terminal success receives full progress where specified.Source: §C.3 · p. 54
  • Terminal position and orientation errors use the final executed state, including settling, as diagnostics that do not replace the complete success predicate.Some tasks have no declared terminal-position anchor, and missing candidate objects can receive explicit penalties when the reference measurement is evaluable.Source: §C.3 · p. 55

C.3.1 Task Conditions and Tolerances

  • Task Success requires the applicable task-specific events, terminal conditions, and physical-validity checks; a grasp or favorable endpoint alone cannot establish completion.Applicable validity failures preclude success, while an admissible execution may still complete in a geometrically inaccurate scene.Source: §C.3.1 · p. 57

C.4 Aggregation and Reporting

  • The evaluation covers 222 reconstruction instances across 39 task families; confirmed build failures score zero for Build Rate, Task Success, and Task Progress.The population is fixed across coding-agent methods and HAR, and each result uses a selected frozen package and resolved execution record.Source: §C.4 · p. 57
  • Continuous-error aggregates condition on successful builds and eligible reference measurements, while inapplicable metrics are not zero-filled and task-failing completed executions remain measurable.HAR intervals use 10,000 source-cluster bootstrap replicates and are pointwise estimates over the observed source population, not paired tests or repeated-run variability estimates.Source: §C.4 · p. 58

C.5 Feasibility Verification and Evaluation Denominators

This section audits human-assisted reconstruction failures, separates configuration-invalid and feasibility-uncertain cases from near misses and controller failures, and checks how evaluation denominators affect agent rankings.

C.5 Feasibility Verification and Evaluation Denominators

  • 113 of 222 instances pass under HAR; the remaining 109 failures are assigned to four machine-checkable evaluator categories.A1 marks physical-validity rejection, A2 marks families without any passing execution, B marks near misses, and C marks physically valid failures in families with a passing execution.Source: §C.5 · p. 59, Table A11 · p. 59
  • All 19 A1 instances use the dexterous-hand configuration, and evidence points to a configuration defect rather than task infeasibility.None of the 113 passing reference executions is gate-rejected, while the same videos succeed under single-arm but fail under bimanual configurations; these instances are excluded from the primary set.Source: §C.5 · p. 59
  • Eighteen A2 instances have no passing execution by any method, so conclusions about their task families are conditional on feasibility.They remain in the primary set because their inclusion affects all methods equally.Source: §C.5 · p. 59
  • Seven B near misses remain in all denominators, while the 65 remaining C failures are physically valid controller failures with median Task Progress 0.25 and mean 0.31.Twenty-five C failures record zero progress, locating substantial benchmark difficulty in executable behavior rather than scene recovery.Source: §C.5 · pp. 59–60
  • The three highest-ranked systems remain the same across S0 (215 instances), S2 (179), and S1 (112), while lower-system ordering differences are within bootstrap uncertainty.Category B is retained in every denominator; the largest shift is Fable-5.1, at +1.5 pp from S0 to S2 and +5.1 pp to S1.Source: §C.5 · p. 60, Table A12 · p. 60

C.6 Supplementary Effector-Motion Analysis

The supplementary analysis compares source and simulated effector motion while limiting its claims to coarse motion characteristics in human-assisted references. V2WScore combines functional, geometric, and dynamic quality, with successful construction acting as a gate.

C.6 Supplementary Effector-Motion Analysis

  • The human-to-robot comparison measures coarse effector motion in human-assisted reference executions, not autonomous behavior recovery or full joint and finger agreement.Human sources provide annotated hand motion rather than measured robot-arm trajectories; the simulated runs use source information.Source: §C.6 · p. 61
  • Six motion summaries capture selected characteristics, not complete trajectory equivalence; net rotation is an initial-to-end SO(3) angle, not accumulated rotation.The summaries include peak speed, lift height, path length, net displacement, and net rotation; lift height can be zero when the effector starts above the interaction.Source: §C.6 · p. 62
  • The comparison uses paired source and simulated measurements across 40 successful reference instances, with trajectories aligned to observed intervals rather than necessarily to terminal stops.Terminal quiescence is unobserved in 39 source and all 40 simulated trajectories; human-source timing follows source action cadence.Source: §C.6 · pp. 61–62

C.7 V2WScore

  • V2WScore gates the weighted functionality, geometry, and dynamics components on successful construction and averages eligible instances equally across task families.A score of 100 requires successful construction, full functional credit, and zero error in every included fidelity measurement.Source: §C.7 · pp. 62–63
  • A complete-manifest score is unavailable when applicable measurements are missing, evaluator failures remain unresolved, or evaluation versions are incompatible.Comparisons require the same instance population, normalization parameters, applicability rules, and metric versions.Source: §C.7 · p. 63

D.1 Cross-Execution Diagnostics for World Reconstruction and Behavior

Cross-execution tests show that task success often depends on compatibility between an agent’s reconstructed scene and its submitted behavior. A separate layout experiment finds high reuse within a selected set of successful reconstructions, with adaptation yielding a modest aggregate gain.

Scope and limitations

  • Cross-execution cells test controller–scene compatibility under retained benchmark configurations, not a fully matched causal decomposition; unavailable diagnostic rows are not reclassified as failures.The probe is a specific fixed state-aware controller, so its failure in an agent scene does not establish that no controller could succeed there.Source: §D.1 · pp. 64, 65, 67

Cross-execution findings

  • 13.1% of agent-built scenes support the state-aware probe on Set A, versus 24.3% own success; on Set B, HAR-action transfer is 9.9% versus 16.2% own success.The comparisons use conditioned diagnostic subsets, not the full benchmark population.Source: §D.1 · p. 65, Table A14 · p. 66
  • 2.3% of available rows succeed when agent actions replay unchanged in HAR scenes, rising to 15.5% on Set A and 15.1% on Set B after initial-position anchoring.Because anchoring uses privileged object position and leaves other differences unchanged, it does not identify object-position estimation as the unique or dominant error source.Source: §D.1 · pp. 65–66
  • Among own successes, only 23% on Set A and 19% on Set B pass both transfer tests, while 44% and 48% pass neither, indicating substantial scene–behavior co-adaptation.The tests are transfer diagnostics, not intrinsic binary labels of scene or behavior quality.Source: §D.1 · p. 66, Table A15 · p. 66

Per-model and per-family results

  • Fable-5.1 reaches 70.0% own success and 40.0% probe-in-scene success on Set A, but per-model differences are descriptive given roughly 20 rows and wide intervals.GPT-6 Astra shows the largest disagreement between the two scene-transfer diagnostics, with its cause untraced at the instance level.Source: §D.1 · p. 67, Table A16 · p. 67
  • Drawer insertion reaches 23.3% own success, but neither HAR actions nor the probe passes in the HAR scene, so the diagnostic cannot attribute those successes.At the family level, DROID pick-place is the only Set-A family whose agent scenes support the probe at an appreciable rate.Source: §D.1 · p. 67

Layout reuse beyond the source video

  • Across 540 paired layout evaluations, direct reuse succeeds in 97.22% of cases and adaptation in 99.26%, recovering 13 of 15 direct-reuse failures.At 20 cm displacement, success rises from 93.52% to 98.15%, but performance is not monotonic in displacement; adaptation also introduces two failures among 525 initially successful cases.Source: §D.1 · p. 68, Figure A6 · p. 69

Scope and limitations

  • Layout-reuse evidence is conditional on successful initial reconstruction and reference-verified translations of the same nine source tasks, not new task families or downstream policy learning.Cross-execution coverage is also limited: Set A has 20 instances across four families, while Set B remains a selected subset of the full benchmark.Source: §D.1 · pp. 67–68

E Cost Analysis

The cost analysis reports substantial aggregate API, agent-time, and evaluation requirements, while showing that cost does not track score monotonically. The accompanying case studies are diagnostic examples, not estimates of error prevalence or model rankings.

E Cost Analysis

  • Running all nine methods over 222 instances corresponds to approximately $9.7k in API spend, 1,044 agent-hours, and 286 L40S GPU-hours of evaluation.Fable-5.1 and Claude Opus 5 each cost about $14 per episode and together account for approximately 63% of estimated total API spend.Source: E Cost Analysis · p. 69
  • Cost does not track score monotonically: GPT-6 Astra has the second-highest V2WScore at about $6 per episode, while Kimi K3 averages 82 agent-minutes without a corresponding score advantage.Flash-tier configurations average less than $2 per episode.Source: E Cost Analysis · p. 69
  • No episode exceeds the three-hour wall-clock allowance; the longest lasts 164 minutes, and evaluation ranges from 7.4 to 10.1 GPU-minutes per episode.One Claude Opus 5 episode records $61.07 because delayed provider reporting can overshoot the $60 observed-spend threshold.Source: E Cost Analysis · p. 69

F Case Studies Across Demonstration Sources

  • The six case studies span robot, egocentric-human, and simulation-generated demonstrations, and were selected for diagnostic value rather than prevalence estimates or model ranking.Each result refers to a particular submitted scene, its own robot behavior, and its recorded physical execution; phase correspondence does not imply equal execution speed.Source: F Case Studies Across Demonstration… · p. 70

F.1 Robot Demonstrations: Pose and Task-Critical Geometry

Across robot and human demonstrations, task completion or favorable geometric scores can coexist with inaccurate pose, missing task-critical geometry, invalid execution, or spatial and temporal mismatch.

F.1 Robot Demonstrations: Pose and Task-Critical Geometry

  • Opus completes stacking despite 144° rotation APE; rigidly aligned Shape CD and small Scene CD do not establish the block’s annotated world pose.The submitted block’s yaw is −138.4° versus 17° in the annotation, while the task tolerates orientation differences.Source: §F.1 · pp. 70–71, Figure A7 · p. 71
  • The shape-sorter block reaches within 1.12 cm of its goal, yet Task Success is zero because the receiver lacks an apertured top and coverage is 0.23 versus the 0.50 requirement.The endpoint tolerance passes, but the task-defining receiver geometry is missing.Source: §F.1 · p. 71, Figure A8 · p. 72

F.2 Nine Models on the Same Instances

  • Opus, GPT-5.6, and DeepSeek succeed at bowl nesting, while Fable has lower Scene CD than Opus yet fails the receiving relation.Geometric fidelity and task completion can disagree within a fixed instance; the plotted positional error is only one component of the decision.Source: §F.2 · pp. 72–73, Figure A9 · p. 74
  • Qwen and Astra exceed the 10 cm lift threshold but fail inverse-kinematics and penetration checks, while Fable’s low Scene CD accompanies no lift.Height and rest tests alone do not certify sustained grasp contact.Source: §F.2 · p. 73, Figure A11 · p. 76

F.3 Egocentric Human Demonstrations under a Robot Embodiment

  • Opus completes all five package-placement stages despite a 28.4 cm scene-registration shift and a 7.2 s execution against a 1.6 s demonstration.The successful containment is relative to the submitted bowl, and the execution proceeds at a substantially different rate.Source: §F.3 · pp. 73, 76, Figure A12 · p. 77

F.4 Native Robot Demonstrations across Target Embodiments

  • Astra succeeds with xArm7 after stationary X5 execution, while Fable’s xArm7 stack passes stacking but fails robot-return and gripper-coupling checks.The recorded follower-joint residual is approximately 0.685 rad against a 0.02 rad tolerance.Source: §F.4 · pp. 77, 79, Figure A13 · p. 78

G Limitations and Future Work

The benchmark identifies limits in attributing end-to-end failures, evaluating general world validity, and isolating system components. Its current scope is reconstruction from a single embodied demonstration.

G Limitations and Future Work

  • A failed rollout may stem from the reconstructed world, the generated behavior policy, or their interaction, so end-to-end evaluation does not disentangle these sources.The paper suggests probe behaviors, independent controllers, or controlled interventions as possible evaluation extensions.Source: G Limitations and Future Work · p. 79
  • The benchmark primarily tests reproduction of the demonstrated interaction, leaving validity under unobserved actions, configurations, and goals open.Counterfactual behavior could distinguish task-specific reconstructions from reusable interactive worlds.Source: G Limitations and Future Work · p. 79
  • Measured performance reflects the complete coding-agent system, so it does not isolate the contributions of the model, scaffold, context management, tools, or execution strategy.The authors propose varying these components independently in future work.Source: G Limitations and Future Work · p. 79
  • Video2World focuses on reconstruction from a single embodied demonstration, not accumulated experience across multiple demonstrations or active information gathering.The paper identifies multi-view observations, multiple demonstrations, and accumulated experience as extensions toward persistent world models.Source: G Limitations and Future Work · p. 79

Related papers

Loading 2610.04432v3…