PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Code the World: Harness Evolution for Compositional 3D Reconstruction

Haoyu Yao, Chenguo Lin, Ruize Du, Yadong Mu

arXiv:2610.37046v1cs.CV

TL;DR

Compact models have limited visual and spatial reasoning, and effective reconstruction also depends on the external harness that governs how a model observes, plans, constructs, and executes a scene program. SCENEEVOLVE combines SCENEACT’s executable compositional scene interface with SCENEHARNESS-RSI, which uses teacher feedback to evolve a reusable harness while keeping model parameters, action semantics, and backend fixed. SCENEEVOLVE improves over its base configuration and outperforms GPT-5.6-Sol with none reasoning effort, reducing mean inference time by 47.0% and equivalent deployment cost per scene by 59.8%.

  • Problem

    Compact models have limited visual and spatial reasoning, and effective reconstruction also depends on the external harness that governs how a model observes, plans, constructs, and executes a scene program.

    Source: §1 · p. 2

  • Method

    SCENEEVOLVE combines SCENEACT’s executable compositional scene interface with SCENEHARNESS-RSI, which uses teacher feedback to evolve a reusable harness while keeping model parameters, action semantics, and backend fixed.

    Source: §1 · p. 2

  • Results

    SCENEEVOLVE improves over its base configuration and outperforms GPT-5.6-Sol with none reasoning effort, reducing mean inference time by 47.0% and equivalent deployment cost per scene by 59.8%.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The evolved harness generalizes to unseen scenes without teacher assistance, and the Qwen3.8-27B harness transfers to Kimi-K3 with results competitive with GPT-6-Astra.

    Source: §1 · p. 2

    Harness evolution incurs substantial computational costs, and optimization does not directly enhance model capabilities or consistently outperform GPT-6-Astra.

    Source: E LIMITATION · p. 18

Abstract

from arXiv · show

Reconstructing editable and compositional 3D scenes from a real image requires jointly reasoning about geometry, object structure, and spatial relationships. We introduce SceneEvolve, a framework that represents 3D scenes as executable code and improves reconstruction by evolving the harness around a small open-source agent. SceneEvolve combines SceneAct, a compositional semantic action interface built on parameterized primitives, hierarchical composition, and semantic spatial relations, with SceneHarness-RSI, which leverages a stronger teacher agent to iteratively evolve the harness using feedback from development-set reconstructions. Throughout harness evolution, the model parameters, action semantics, and execution backend remain fixed. The resulting harness provides a reusable reconstruction procedure that enables the small open-source model to reconstruct unseen scenes without parameter updates or teacher assistance. Across two in-domain datasets and one out-of-domain dataset, SceneEvolve with the comparatively compact Qwen3.8-27B model improves all four evaluated reconstruction metrics while reducing mean inference time by 47.0% relative to the base model. More notably, despite relying on a substantially smaller model, SceneEvolve achieves 32.3% higher reconstruction quality than the large proprietary GPT-5.6-Sol model, while reducing mean equivalent deployment cost by 59.8%. Furthermore, the harness evolved using Qwen3.8-27B also generalizes to Kimi-K3, achieving comparable reconstruction performance to GPT-6-Astra. The project page is https://haoyu-agent.github.io/SceneEvolve.

1 INTRODUCTION

The introduction frames single-image reconstruction as a joint challenge in visual reasoning and harness design. SceneEvolve pairs a compositional scene interface with teacher-guided evolution of a reusable harness, and reports gains over GPT-5.6-Sol with lower deployment cost and inference time.

1 INTRODUCTION

  • The paper targets direct scene-code generation by a substantially smaller open-source model through harness design rather than parameter updates, motivated by frontier-model deployment costs.It also aims to reduce reliance on costly render-and-refine loops.Source: §1 · p. 2
  • Compact models have limited visual and spatial reasoning, while reconstruction also depends on the instructions, tools, and control logic of the external harness.The paper seeks an interface that reduces geometric-reasoning demands and a reusable procedure that guides reconstruction.Source: §1 · p. 2
  • SCENEEVOLVE combines SCENEACT’s compositional scene interface with SCENEHARNESS-RSI, which uses a stronger teacher to revise the reconstruction harness from development-set feedback.Model parameters, action semantics, and the execution backend remain fixed; the resulting harness is frozen for unseen scenes without teacher assistance.Source: §1 · p. 2
  • SCENEACT maps scene intent to executable code through parameterized primitives, hierarchical composition, and semantic spatial relations, giving explicit control over geometry and scene structure.These abstractions reduce the need for low-level geometric programming.Source: §1 · p. 2
  • Across in-domain and out-of-domain datasets, SCENEEVOLVE with Qwen3.8-27B outperforms GPT-5.6-Sol, reducing cost by 59.8% and inference time by 47.0% relative to the baseline.The evolved harness also transfers to Kimi-K3, with results competitive with GPT-6-Astra.Source: §1 · p. 2

2 RELATED WORK

The related work spans 3D generation, multi-stage compositional reconstruction, and programmatic coding agents. SceneEvolve instead represents object-centric reconstruction as an executable scene program with explicit construction and spatial operators.

3D Generation for Compositional 3D Scene Reconstruction

  • SCENEACT unifies object construction and scene composition in an executable program using primitives and spatial operators.Prior work includes methods for recovering object geometry and appearance, modeling layouts and relations, and jointly predicting objects with spatial configurations.Source: §2 · p. 3

Multi-stage Pipelines for Compositional 3D Scene Reconstruction

  • SCENEEVOLVE builds object-centric reconstruction as an executable scene program from parameterized primitives, part hierarchies, and spatial relations.The passage contrasts this with prior work on editable geometry and layout, inverse rendering, multi-stage scene pipelines, multi-view assembly, and joint geometry-and-configuration generation.Source: §2 · p. 3

Programmatic Coding Agents for Compositional 3D Scene Reconstruction

  • SCENEACT exposes fixed semantic construction operators, while SCENEHARNESS-RSI optimizes the reusable procedure.Related work includes procedural generators, executable hierarchical assemblies, structured scene descriptions, and visual coding agents that refine programs using rendered feedback or execution-guided iteration.Source: §2 · p. 3

3 METHODS

SceneEvolve pairs a semantic interface for turning scene intent into executable actions with teacher-guided evolution of the reconstruction harness, while keeping the student and execution semantics fixed.

3.1 OVERVIEW

  • SceneEvolve combines SceneAct’s intent-to-action interface with SceneHarness-RSI’s iterative harness refinement around a frozen student.The student uses the resulting harness to reconstruct scenes without changing its parameters.Source: §3.1 · p. 3, §3.3 · p. 5

3.2 SCENEACT: A COMPACT SEMANTIC INTERFACE FROM INTENT TO ACTIONS

  • SceneAct maps high-level scene reasoning to deterministic Blender operations through parameterized primitives, hierarchical composition, and semantic spatial relations.These abstractions shift construction away from direct low-level geometric operations and numerical parameters.Source: §3.2 · pp. 3–4
  • Scene, Asset, and Part organize construction from complete scenes to object structure and part-level geometry, supporting modular construction and localized editing.Each Part is built from one or more parameterized primitives.Source: §3.2 · p. 4
  • Semantic relations such as alignment and distribution position elements relative to scene geometry, avoiding independent coordinate prediction and allowing dependent elements to update after edits.The interpreter resolves constraints into deterministic transformations using geometry, reference frames, extents, and anchors.Source: §3.2 · pp. 4–5

3.3 SCENEHARNESS-RSI: A TEACHER-GUIDED RECURSIVE HARNESS EVOLUTION

  • SceneHarness-RSI uses a teacher to inspect reconstruction outputs and execution traces, propose targeted harness revisions, and search over an archive of evaluated versions.The search changes harness components while holding SceneAct semantics, student parameters, and the Blender backend fixed.Source: §3.3 · p. 5
  • Candidates are scored on development-set reconstructions using an average of rescaled SSIM, LPIPS, and DINOv2 cosine similarity; only strict score improvements replace the retained harness.The rule keeps retained development scores non-decreasing, but does not guarantee better expected performance or results on unseen scenes.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

SceneEvolve is evaluated on two in-domain datasets and one out-of-domain dataset, against its base model and frontier models, with qualitative comparisons and component ablations. Results report improved reconstruction quality, reduced inference time and cost, and harness transfer across models.

4.1 EXPERIMENTAL SETTINGS

  • The evaluation uses 100 development scenes, 40 in-domain test scenes, and 20 out-of-domain test scenes, with ARKitScenes reserved for out-of-domain evaluation.ScanNet and CA-1M development sets support harness search; held-out test scenes are reserved for evaluation, and the harness transfers to ARKitScenes without its data or feedback being used in development.Source: §4.1 · p. 6

4.2.1 COMPARISON WITH THE BASE MODEL

  • Across three datasets, DINOv2-S increases from 0.198 to 0.392, while SSIM improves by 6.7%, LPIPS decreases by 6.1%, and PSNR increases by 0.118 dB over the base model.Mean inference time falls 47.0%, from 1,510.7 to 800.2 seconds; the evolved harness also improves all four metrics on Kimi-K3 without target-specific search, fine-tuning, or manual adaptation.Source: §4.2.1 · p. 8

4.2.2 COMPARISON WITH FRONTIER MODELS

  • 32.3% higher mean SSIM than GPT-5.6-Sol with none reasoning effort, with improvements in eleven of twelve dataset–metric comparisons.All four metrics improve on CA-1M and ARKitScenes; mean equivalent deployment cost is 59.8% lower than Sol-none, from approximately $0.290 to $0.117 per scene.Source: §4.2.2 · pp. 8–9
  • The Qwen3.8-27B harness transfers to Kimi-K3 without target-specific optimization, producing reconstruction performance competitive with GPT-6-Astra.The transferred harness also consistently outperforms Claude-Opus-5 across almost all metrics on all three datasets.Source: §4.2.2 · p. 9

4.3 QUALITATIVE RESULTS

  • Qualitative comparisons show improved scene coherence and furniture arrangements, while cross-model transfer better preserves colors, furniture, and spatial layout.Remaining discrepancies include an overly bright lounge and simplified seating with inaccurate colors; the transferred Kimi-K3 reconstruction retains the lounge’s dark green wall more distinctly than either frontier model.Source: §4.3 · p. 9

4.4 ABLATION STUDY

  • Across three datasets, SCENEACT raises DINOv2-S from 0.305 to 0.392 and reduces mean inference time by 22.7% against direct Blender operations.With SCENEACT fixed, harness evolution raises DINOv2-S from 0.285 to 0.392, reduces mean inference time by 43.8%, and lowers mean equivalent cost per scene from $0.155 to $0.117.Source: §4.4 · p. 10

5 CONCLUSION

SceneEvolve combines SCENEACT with teacher-guided evolution of a reusable reconstruction harness. It improves over its base configuration, outperforms GPT-5.6-Sol with none reasoning effort, and transfers to Kimi-K3 with results competitive with GPT-6-Astra.

5 CONCLUSION

  • SceneEvolve combines the SCENEACT interface with teacher-guided evolution of a reusable reconstruction harness.The framework targets editable, compositional 3D reconstruction in code-generated scene settings.Source: §5 · p. 10
  • Mean inference time falls by 47.0% and equivalent deployment cost per scene by 59.8% while SceneEvolve outperforms GPT-5.6-Sol with none reasoning effort.The framework also improves over its base configuration.Source: §5 · p. 10
  • A harness evolved using Qwen3.8-27B transfers to Kimi-K3, achieving reconstruction results competitive with GPT-6-Astra.Source: §5 · p. 10

A.1 DATASET DETAILS

The evaluation uses ScanNet and CA-1M for in-domain development and testing, and ARKitScenes exclusively for out-of-domain testing. Scene-level splits keep repeated views and captures together and prevent development data from overlapping with test data.

A.1 DATASET DETAILS

  • The study uses 160 scenes: 100 for development, 40 for in-domain testing, and 20 for out-of-domain testing.All methods use the same held-out scenes under a consistent single-image reconstruction protocol.Source: §A.1 · p. 15
  • ScanNet and CA-1M each contribute 50 development scenes and 20 held-out in-domain test scenes, while ARKitScenes contributes 20 out-of-domain test scenes.ARKitScenes contributes no images or reconstruction feedback to harness development; student parameters, selected harness, and inference configuration transfer without modification.Source: §A.1 · p. 15
  • All views and repeated captures of each in-domain physical scene stay within one split, preventing development–test overlap.Source: §A.1 · p. 15

A.2 COUNTERPART DETAILS

The evaluation compares SceneEvolve with its unchanged Qwen3.8-27B backbone and several proprietary multimodal models under specified reasoning-effort settings. It also tests whether a Qwen-evolved harness transfers to Kimi-K3 without target-specific adaptation.

Kimi-K3

  • The transferred Qwen3.8-27B harness is evaluated on Kimi-K3 without further harness evolution, fine-tuning, or target-model-specific optimization.The comparison uses the same reasoning effort, reconstruction protocol, and execution budget; the interface and execution backend remain unchanged.Source: §A.2 · pp. 15–16

Qwen3.8-27B

  • Qwen3.8-27B at xhigh reasoning effort serves as the primary base-model counterpart, matching the frozen student’s model and effort to isolate framework improvements.Source: §A.2 · p. 15

GPT-5.6-Sol

  • GPT-5.6-Sol is evaluated at none, low, medium, and high reasoning effort, with none as the primary proprietary reference.Source: §A.2 · p. 16

GPT-6-Astra

  • GPT-6-Astra is evaluated at low, medium, and high effort, and its medium configuration serves as the development-time teacher, not a test-time reconstruction agent.Source: §A.2 · p. 16

Claude-Opus-5

  • Claude-Opus-5 is evaluated at none, low, medium, and high effort without task-specific fine-tuning.Source: §A.2 · p. 16

Gemini-Flash-3.8

  • Gemini-Flash-3.8 is evaluated at low, medium, and high effort to broaden comparisons across model families and inference configurations.Source: §A.2 · p. 16

B ADDITIONAL RELATED WORK

Prior self-improving agent systems refine memories, executable scaffolds, textual components, or agent source code using feedback. SceneEvolve extends system-level harness refinement to code-based 3D reconstruction.

B.1 SELF-IMPROVING AGENTS AND HARNESS OPTIMIZATION

  • SceneEvolve extends iterative system-level refinement to code-based 3D reconstruction, where execution logic translates reconstruction objectives into executable code and rendered scenes.Source: §B.1 · p. 16
  • Prior approaches refine different targets: Reflexion records natural-language reflections, STOP recursively improves executable scaffolds, and other methods optimize textual components or agent source code.The passage contrasts Reflexion (Shinn et al., 2023), STOP (Zelikman et al., 2024), TextGrad, ProTeGi (Pryzant et al., 2023), GEPA (Agrawal et al., 2026b), Feedback Descent (Lee et al., 2025), Meta-Harness (Lee et al., 2026), and Self-Harness (Zhang et al., 2026a).Source: §B.1 · p. 16
  • Evaluations of Meta-Harness and Self-Harness span classification, reasoning, coding, and agent benchmarks, while both pursue improvement through system-level refinement rather than model training alone.Source: §B.1 · p. 16

C IMPLEMENTATION DETAILS

The implementation evaluates a minimal single-call baseline and an evolved harness that plans scenes, generates code through SceneAct, and iteratively inspects rendered output. The evolved stages use structured plans and targeted repairs to express scene geometry and relations.

C IMPLEMENTATION DETAILS

  • Experiments ran on a server with four NVIDIA L40S GPUs, an AMD EPYC 9654 CPU, and 755 GiB RAM, using Ubuntu 24.04.3 LTS, Python 3.10.12, and PyTorch 2.5.1+cu124.Source: C IMPLEMENTATION DETAILS · p. 16

D.1 INITIAL HARNESS

  • The initial baseline uses one direct model call with a system and user prompt, returning a complete Blender Python program; a separate local script handles saving and rendering.It excludes external coding-agent frameworks, tools, memory, and orchestration components.Source: §D.1 · p. 17
  • The baseline prompts require actual 3D modeling, preservation of scene layout and viewpoint, and complete executable code without invented off-frame content.Source: §D.1 · p. 17

D.2 THE EVOLVED FINAL HARNESS

  • The evolved harness plans a scene as structured JSON, generating a hierarchy of assets, parts, attributes, and spatial relations grounded conservatively in visible evidence.Source: §D.2 · p. 17
  • SceneAct translates the plan into Blender Python using parameterized primitives, asset–part hierarchies, and semantic spatial operators for relations such as alignment, support, and attachment.The harness encourages consistent reference frames and explicit camera, material, color, and lighting choices, allowing edits to target particular objects or parts.Source: §D.2 · p. 17
  • Rendered scenes are checked for execution errors, then inspected against the reference and plan using visual assessment and spatial validation before targeted code repairs.The loop reruns and renders revised code, modifying relevant objects or attributes rather than regenerating the entire scene.Source: §D.2 · pp. 17–18

E LIMITATION

SceneEvolve’s harness optimization has substantial search costs and does not directly improve the underlying model’s capabilities. Its evolved system also does not consistently outperform GPT-6-Astra.

E LIMITATION

  • Harness evolution incurs substantial computation from repeated teacher-guided analysis, candidate generation, reconstruction, rendering, and evaluation, creating a trade-off between harness quality and search efficiency.Source: E LIMITATION · p. 18
  • Harness optimization does not directly enhance the underlying model, and the evolved system does not consistently outperform GPT-6-Astra.Source: E LIMITATION · p. 18
Loading 2610.37046v1…