PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs

Zhiqi Li, Yuxuan Liao, Bo Zhu

arXiv:2609.11499v1cs.CV

TL;DR

Reconstructing complex worlds from images requires resolving structures at multiple scales while maintaining relationships among parts. RCWM uses recursive scene programs and a whole-part-whole solver, achieving the highest PSNR and lowest LPIPS across references.

  • Problem

    How should a coding model organize reconstruction across multiple scales while maintaining relationships among parts of a complex world?

    Source: §1 · p. 2

  • Method

    RCWM represents worlds as recursive scene programs and uses a solver that establishes each whole, recursively reconstructs unresolved parts, then revisits their composition.

    Source: §1 · p. 2

  • Results

    Across whole-scene and local-crop references, RCWM achieved the highest PSNR and lowest LPIPS on every reference, and the highest SSIM on nine.

    Source: §4.2 · p. 9

  • Takeaways & Limitations

    RCWM provides a recursive construction principle for reconstructing complex executable 3D worlds from a single reference image.

    Source: §5 · p. 11

    Evaluation is limited to a small set of scenes, single runs on one base model, and ablations on two scenes.

    Source: §5 · p. 12

Abstract

from arXiv · show

Code world models represent worlds as executable programs, but this representation alone does not determine how to construct a complex world. We introduce Recursive Code World Models (RCWM), a framework for reconstructing complex 3D worlds in code from a single reference image. RCWM couples a Recursive Scene Program (RSP) representation with a construction solver that recursively calls itself. An RSP represents the executable world as compositional scene code, while each solver call follows the same complete process: establish the whole, recursively reconstruct unresolved parts, and revisit the whole to refine their composition. This global-local-global recursion gives fine-scale structures their own perception-and-editing loops while preserving scene-wide geometry and relationships. Reference-aligned views propagate a shared camera projection across levels, while parent revisitation addresses boundaries, spatial relations, and shared errors that emerge after local refinement. A vision-language coding agent directly compares reference images with scene renders to guide refinement, recursive descent, and return. Across complex scenes, RCWM outperforms prior code-based image-to-scene reconstruction methods. Ablation studies further support the benefits of recursive construction and suggest that deeper calls can improve finer-scale reconstruction. RCWM provides a recursive construction principle for building complex executable worlds from visual evidence.

1 INTRODUCTION

RCWM constructs complex executable worlds through recursive scene programs, giving each unresolved subworld its own complete reconstruction process. Each level establishes a whole, recursively solves parts, and revisits their composition to refine cross-part consistency.

Introduction

  • Scene-wide reconstruction can overlook small structures, while focused local edits can introduce conflicts that only become apparent when refined parts are inspected together.The tension recurs across scales, including between buildings in a town and windows or signs in a facade.Source: §1 · p. 2
  • RCWM makes each unresolved part a subworld that receives its own complete reconstruction process, rather than treating it as an asset to insert.The same solver cycle applies at every scale, from buildings to facades and their details.Source: §1 · p. 2
  • Recursive Scene Programs represent executable worlds as compositional scene code organized over nested subworlds with explicit child references, editable structure, and shared dependencies.Source: §1 · p. 2
  • At each level, the solver establishes the whole, recursively reconstructs unresolved parts, and revisits the assembled whole to refine boundaries, spatial relations, and shared errors.Each child receives focused reference evidence and inherits the parent’s camera projection and spatial conventions; recursion determines which subproblems receive a complete solve, while parallelism only schedules compatible branches.Source: §1 · p. 2, Figure 2 · p. 3
  • The paper evaluates whole-scene and local reconstruction against image-to-scene-program baselines using the same base model, with ablations of construction order, recursive depth, and parent revisitation.The framework is instantiated with a parameterized Three.js compiler; ablations hold local visual access constant.Source: §1 · p. 2

2 RECURSIVE CODE WORLD MODELS

RCWM represents a reconstructed world as an executable program of nested subworlds and uses a recursive solver that establishes each whole, solves its parts, and revisits their composition. Shared camera context and visual comparisons connect refinement across levels.

2.1 EXECUTABLE RECURSIVE SCENE PROGRAMS

  • A Recursive Scene Program is executable, editable scene code organized into nested subworlds whose parent programs define child references, placement rules, interfaces, and shared dependencies.Execution and rendering with a reference-view camera produce an image used to align layout, local appearance, and spatial relationships with the reference.Source: §2.1 · pp. 3–4

2.2 RECURSIVE WORLD CONSTRUCTION

  • Each solver call establishes a subworld, recursively solves selected parts with the same process, then returns to refine their composition at the parent level.Visual feedback guides edits, further child calls, and the return to the parent.Source: §2.2 · pp. 4–5

2.2.4 COMPOSE, REFINE, AND RETURN

  • After children return, the parent assembles their programs and reviews the whole, refining cross-part geometry, placement, and shared structures before continuing or returning.A complete review with no new actionable discrepancy ends the cycle; budget exhaustion returns the latest executable state.Source: §2.2 · p. 5, §2.2.4 · p. 8

2.2.1 CONSTRUCTION STATE AND INITIALIZATION

  • Root initialization establishes world-coordinate and unit conventions and calibrates the reference camera; children inherit those conventions and derive cameras from its projection.Camera revisions are controlled at the root, followed by refreshed observations and affected child states.Source: §2.2.1 · p. 5

2.2.2 ESTABLISH THE WHOLE

  • The solver compares a candidate rendered within its inherited scene context against the reference to establish layout and interfaces before local construction.The parent context preserves surrounding geometry, shared structures, and placement rules while the candidate is edited and rendered again.Source: §2.2.2 · pp. 5–6

2.2.3 PREPARE AND RECURSIVELY CONSTRUCT THE PARTS

  • Children receive matched reference and camera crops that preserve parent camera pose and magnification; overlapping windows inspect boundaries, and each child repeats the solver at finer scales.Child calls receive allocated budgets that reserve resources for parent revisitation.Source: §2.2.3 · pp. 6–7, Figure 3 · p. 7

3 RELATED WORK

Prior work uses code for editable, compositional world representation and image-conditioned reconstruction, while RCWM targets executable scenes that match a specific reference image. Its recursive solver applies the full construction process to subworlds and revisits each assembled parent, framing recursion as a general world-construction principle.

Code-based world representation and generation

  • Unlike procedural or text-driven world generation, RCWM seeks an executable scene program whose rendering faithfully matches a particular reference image.Code-based approaches expose geometry, materials, and spatial relationships for editing and reuse; examples include Infinigen, SceneCraft, and WorldClaw.Source: §3 · p. 8

Image-to-code world reconstruction

  • RCWM applies its complete recursive solve to unresolved subworlds, then reinspects and refines the assembled parent rather than limiting recursion to decomposition or staged editing.Image-to-code reconstruction must match visible shapes, arrangements, occlusions, and fine details under a consistent camera despite ambiguity in 3D geometry.Source: §3 · pp. 8–9

Recursive inference and concurrent work

  • Unlike room-specialized FuncRoom-Agent, RCWM frames recursion as a general world-construction principle rather than a domain-specific generation procedure.Recursive Language Models use recursive calls to examine external context; in RCWM, child calls construct subprograms that change the executable world their parent must inspect.Source: §3 · p. 8
  • A frozen coding model applies the same visual solver across objects, interacting assemblies, and continuous terrain, spanning architectural details, buildings, towns, and terrain-rich worlds.Subworlds can invoke further solves and refine their assembled whole before returning to the parent.Source: §3 · pp. 8–9

4 EXPERIMENTS

Experiments evaluate executable-scene reconstruction on whole scenes and local crops, then ablate construction order, recursive child solves, and parent refinement.

4.1 SETUP

  • Evaluation uses five whole-scene references and five local crops, reporting PSNR, SSIM, edge F1, LPIPS, and CLIP image-embedding cosine similarity against native-resolution references.All methods receive the same raw images at the same resolution.Source: §4.1 · p. 9

4.2 RECONSTRUCTION RESULTS

  • RCWM achieves the highest PSNR and lowest LPIPS on every reference, and the highest SSIM on nine; VIGA leads valley-village SSIM, 0.41 to RCWM’s 0.38.The PSNR and LPIPS improvements span both whole-scene and local-crop references.Source: §4.2 · p. 9, Table 1 · p. 10
  • Matched-scale views expose scene layout and small structures; rotated, elevated, and close-up renders reveal encoded 3D arrangement, while call trees show nested solves and returns to parents.The rotated and close-up views render delivered programs without further optimization.Source: §4.2 · p. 9

4.3 ABLATION STUDY

  • The ablation compares flat zoom, local-to-global, global-to-local, fixed two-level recursion, and free-depth recursion; recursive variants share a solver and differ in depth restriction.Recursive calls establish a whole, construct its parts, and refine the composition before returning; all variants share the base model, representation, tools, resolution, and crop access.Source: §4.3 · pp. 10–11
  • On medieval-village, free-depth recursion uses three levels and 33 nodes, raising whole-frame PSNR from 16.8 to 19.0 and local SSIM from 0.52 to 0.60 versus fixed two-level recursion.It outperforms the other three workflows on both measures; on school-block, it uses only two levels and scores similarly to the fixed two-level variant, suggesting gains vary by task.Source: §4.3 · p. 11

5 CONCLUSION AND LIMITATIONS

RCWM combines compositional scene code with a recursive whole–parts–whole construction process to refine local detail while maintaining whole-scene consistency. The authors identify ambiguity, computational cost, and limited evaluation as constraints requiring further assessment.

Conclusion

  • RCWM applies a whole–parts–whole construction cycle to selected subworlds, combining compositional Recursive Scene Programs with reference-aligned local inspection and parent-level refinement.Parent revisitation addresses spatial relationships among returned components, supporting local detail and whole-scene consistency at every scale.Source: §5 · pp. 11–12

Limitations

  • Single-view input leaves hidden geometry and absolute scale ambiguous; visual inspection can miss errors, refinement may be costly, and convergence is not guaranteed.Evaluation covered a limited number of scenes, with single runs on one base model and ablations on two scenes; the authors call for larger benchmarks, repeated runs, and human evaluation of spatial relationships.Source: §5 · p. 12
  • Crop-based scene programs contain only content visible in the reference, so unseen regions are absent when viewed from behind.Additional views render the programs from the reference camera, rotated and raised orbits, and close-ups without re-optimization.Source: Figure 10 · p. 16

Conclusion

  • Logged solver-call trees for four representative cases show the complete whole–parts–whole cycle, and trees for all ten scenes are released with the code.Source: Figure 11 · p. 17

A REPRODUCIBILITY DETAILS

The appendix specifies solver instructions and execution settings, including recursive-call inputs, parent resumption, and depth limits.

A REPRODUCIBILITY DETAILS

  • Each solver call receives a reference crop, matched camera specification, task brief, and inherited parent snapshot; the runner resumes the parent after child components return.When the agent requests children, the runner starts their solver calls.Source: A REPRODUCIBILITY DETAILS · p. 19
  • The runner allows five levels total, after which the agent finishes the current component through local refinement; the fixed-depth ablation caps calls at two levels.The appendix provides the solver instruction and execution settings used in the reported experiments.Source: A REPRODUCIBILITY DETAILS · p. 19

A.1 THE SOLVER INSTRUCTION

At every node, the solver compares a rendered whole with its target, recursively reconstructs unresolved parts, and revisits the whole to refine their integration. Completion combines pixel-aligned visible detail with style-consistent world knowledge for hidden regions, while checking that the full scene remains coherent.

A.1 THE SOLVER INSTRUCTION

  • Each solver call follows a whole-parts-whole loop: inspect matched and rotated views, repair locally or descend into unresolved subproblems, then integrate child results and refine the whole.A subproblem merits its own loop when it contains unresolved object arrangements or a complex object’s internal structure; simple objects and scattered small differences are repaired at the current level.Source: §A.1 · p. 19
  • Completion matches visible regions to the reference while using a style-calibrated mental model and world knowledge to complete hidden or occluded geometry.The solver checks renders from four compass directions for flatness or a stage-set feel, then re-verifies the visible region against the reference.Source: §A.1 · p. 19
  • The solver makes reversible craft choices independently; visual inspection guides decisions, metrics are recorded, and child nodes have restricted write scopes.Schema checks, resource limits, and evidence recording are runtime responsibilities, not success gates based on generated appearance scores.Source: §A.1 · pp. 19–20

B DESIGN PHILOSOPHY

Complex-world reconstruction requires detailed work on individual components and repeated checks of how they fit together. The design applies a complete reconstruction process at every scale, with visual inspection guiding edits and parent-level review refining composition.

B DESIGN PHILOSOPHY

  • Each subworld receives its own complete reconstruction loop, in which the agent establishes its structure, recursively reconstructs details, and inspects the assembled result.The agent chooses decompositions based on visible structure and remaining work; the procedure applies to facades, buildings, assemblies, and terrain regions.Source: B DESIGN PHILOSOPHY · p. 20
  • Child reference crops share the parent camera pose and use matching crop coordinates and magnification to preserve a consistent frame for local comparisons.Affected child views are refreshed when the root camera or a shared scene dependency changes.Source: B DESIGN PHILOSOPHY · p. 20
  • The agent compares matched-magnification references and renders to guide edits and stopping, while numerical appearance metrics are recorded for evaluation.Visual judgments target discrepancies in shape, placement, and spatial relationships, with focused child views and repeated inspections after edits and returns.Source: B DESIGN PHILOSOPHY · p. 20
  • For hidden surfaces and out-of-frame continuations, the agent uses object and style knowledge to make assumptions, inspects geometry from rotated views, then checks visible changes from the reference view.Visible fidelity is assessed against the reference; other views examine geometry produced by the assumptions.Source: B DESIGN PHILOSOPHY · p. 20
  • After child calls, the parent revisits the assembled scene to refine placement, contact, occlusion, and shared geometry, repairing shared errors at the ancestor that controls their cause.Compatible children may run concurrently, followed by joint refinement; the parent can edit relationships directly or reopen affected child tasks.Source: B DESIGN PHILOSOPHY · p. 20
  • Executable scene code exposes geometry, materials, arrangement, procedures, and parameters, letting parents compose and edit child subprograms while preserving components for reuse and new viewpoints.A parent can reposition a reconstructed building while retaining the code that generates its windows and roof.Source: B DESIGN PHILOSOPHY · pp. 20–21

Related papers

Loading 2609.11499v1…