PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects

Puyi Wang, Yuhao Wang, Linjie Li, Zhengyuan Yang, Kevin Qinghong Lin, Yangguang Li, Yu Cheng

arXiv:2605.19587v1cs.AI

TL;DR

Indoor scene synthesis often represents content as static meshes and relies on curated datasets for articulation, limiting on-demand control of interactable objects. SceneCode compiles natural-language prompts into executable, part-aware indoor worlds; evaluations report improvements across scene, asset, human-judgment, and robot-interaction measures.

  • Problem

    Many indoor-scene pipelines use static meshes and obtain articulation from curated datasets, constraining object-level control and on-demand generation of new interactable assets.

    Source: §1 · p. 2

  • Method

    SceneCode turns a natural-language prompt into an executable indoor world through room-level planning, part-wise object programs, simulation compilation, and persistent scene-state registration.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across scene, asset, human, and robot evaluations, SceneCode reports prompt-faithful scenes and interaction-ready articulated assets; human-rated prompt fidelity exceeded SceneSmith by ∆PF = +2.8%.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    Executable, part-aware programs provide a unified representation for editable indoor scenes and objects with simulation-ready interaction structure.

    Source: §5 · p. 9

    Program synthesis, execution, repair, and refinement introduce nontrivial runtime.

    Source: K Limitations · p. 34

Abstract

from arXiv · show

Indoor scene synthesis underpins embodied AI, robotic manipulation, and simulation-based policy evaluation, where a useful scene must specify not only what the environment looks like, but also how its objects are structured. Existing pipelines, however, typically represent generated content as static meshes and inherit articulation only from curated asset libraries, which limits object-level controllability and prevents new interactable assets from being produced on demand. We address this gap by formulating physically interactable indoor scene synthesis as programmatic world generation, and present SceneCode, a framework that compiles a natural language prompt into an executable, code-driven indoor world rather than a collection of opaque meshes. A room-level agentic backbone first turns the prompt into a structured house layout and emits per-object AssetRequests through a planner--designer--critic loop. Each request is then routed to one of five code-generation strategies and converted into a synthesized part-wise Blender Python programs that are validated through an execution-guided repair-and-refine loop. The resulting programs are compiled into simulation-ready assets, and exported as SDF for physics simulation. A persistent scene-state registry links object requests, executable programs, rendered geometry, and simulation assets, turning scene assembly into a traceable and locally editable world-building process. We evaluate SceneCode across scene-level synthesis, object-level asset quality, human judgment, and downstream robot interaction. Results show that executable world programs improve prompt-faithful indoor scene generation and produce assets with cleaner mesh structure, and simulator-loadable articulation metadata. Project page: https://scene-code.github.io/.

1 Introduction

SceneCode addresses the limits of static meshes and curated articulated-object libraries by representing indoor scenes as executable programs with explicit object structure and interaction mechanisms. Its pipeline generates editable assets on demand, and evaluations report prompt-faithful scenes and interaction-ready objects.

1 Introduction

  • Existing pipelines often use static meshes and inherit articulation from curated datasets, limiting object-level control and on-demand creation of new interactable assets.This is consequential because embodied scenes must specify object structure, movement, and how agents can physically act on objects.Source: §1 · p. 2
  • SceneCode represents indoor scenes as executable programs that unify geometry, part structure, materials, physical attributes, and motion mechanisms.Interaction is intrinsic to the generated program rather than supplied only by a selected articulated asset.Source: §1 · p. 2
  • A room-level plan becomes per-object Blender programs with part-wise geometry and physical attributes, then editable objects registered in persistent scene state.The resulting objects are locally regenerable and support constraint modification and simulation interaction.Source: §1 · pp. 2–3
  • Across scene, asset, human, and robot evaluations, SceneCode produces prompt-faithful scenes and usable articulated assets that retain executable joints for contact-based interaction.Human raters found scenes more prompt-faithful than each baseline; MuJoCo demonstrations showed independent movable links and executable joints.Source: §1 · p. 3

2 Related Work

Prior work approaches indoor scene synthesis through learned layouts, retrieval, and procedural generation, while code-driven 3D systems often target single objects or visual layout. SceneCode instead synthesizes interactable indoor objects as executable programs and registers their physical structure in scene state.

2 Related Work

  • Indoor scene synthesis includes learned layout distributions and LLM- or retrieval-guided systems that populate rooms from curated 3D libraries.Examples include autoregressive and diffusion models, Holodeck, LayoutVLM, HSM, and SceneSmith, which combines retrieval with image-to-3D generation for simulation-ready scenes.Source: §2 · p. 3
  • Code-driven 3D systems offer editable representations but mostly target single-object modeling or visual layout, with limited downstream physical interaction.SceneCode extends this direction to interactable indoor scenes using routed, verified ObjectPlans, part-wise Blender programs, URDF/SDF assets, and persistent scene state.Source: §2 · p. 3

3 Method

SceneCode couples room-level object planning with code-driven asset construction, simulation compilation, and persistent scene-state registration. The stages carry object requirements through executable, part-aware programs into editable simulation assets.

3 Method

  • The method separates indoor scene synthesis into room-level decisions about required objects and placement, and code-driven construction of each object.The pipeline continues through simulation-ready compilation and scene assembly with state serialization.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Room-Level Agentic Scene Backbone

  • A planner–designer–critic loop builds a structured house layout and ordered AssetRequests through large-furniture, wall-mounted, ceiling-mounted, and manipulable-object stages.Each request carries category, description, target dimensions, style, placement transform, and support relation into object synthesis.Source: §3.1 · p. 4

3.2 Code-Driven Object Generation

  • Object generation routes requests to construction strategies, verifies an ObjectPlan, synthesizes separate part-wise Blender programs, and validates them through execution, repair, and visual refinement.Routes cover wall art, static furniture, simple or structured manipulable objects, and articulated objects; a fixed template handles thin coverings. Verification checks part completeness, dimensions, spatial consistency, and movable-part independence.Source: §3.2 · pp. 4–5

3.3 Simulation-Ready Asset Compilation

  • Compilation packages rigid objects as single-body SDFs and articulated objects as link–joint structures with inferred revolute or prismatic joints.Assets receive approximate mass and inertia attributes plus collision proxies for downstream physics simulation.Source: §3.3 · p. 6

3.4 Scene Assembly and State Serialization

  • A shared scene identifier links each object's request, executable programs, rendered mesh, and simulation artifacts, enabling traceable placement and local edits or partial re-execution.Placement scales geometry to target dimensions, applies the planned transform, and aligns it with the support relation.Source: §3.4 · p. 6

4 Experiments

SceneCode is evaluated as a room- and object-generation system, with results spanning prompt faithfulness, asset usability, editability, and robot interaction.

4.1 Experimental Setup

  • Evaluation compares SceneCode with three scene-level baselines on 30 prompts spanning six room categories, using scene metrics and a nine-participant prompt-faithfulness study.The study reports within-group SceneCode-minus-baseline differences because each baseline was rated by a separate participant group.Source: §4.1 · p. 6

4.2 Room-Level Scene Synthesis

  • ∆PF was +2.8% vs SceneSmith, +13.2% vs HSM, and +24.6% vs LayoutVLM; SceneCode alone led semantic-fidelity and physical-usability metrics.Figure 3 shows room-level prompt matching, while HSM's cross-subtree relations are not reliably realized.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 Object-Level Geometry and UV Quality

  • SceneCode averaged about 22 UV islands per asset versus 96 for SAM 3D Objects, and produced 0 non-manifold edges on average.Its assets also had 6013.64 faces versus 12036.57 and 4945 vertices versus 7400; Figure 4(a) shows more coherent UV charts.Source: §4.3 · pp. 8–9, Table 2 · p. 8
  • Named Blender Python parameters let users locally re-execute an object program to create variants without rerunning the room-level agent or regenerating the reference image.Figure 4(b) varies arr_count and ring_count while retaining the same object identity.Source: Figure 4 · p. 8, §4.3 · p. 9
  • Part-decomposed programs with explicit movable parts can synthesize requested articulated structures or materials unavailable in curated retrieval libraries.Figure 4(c) illustrates requests that specify functional structure or material.Source: Figure 4 · p. 8, §4.3 · p. 9

4.4 Articulation and Robot Interaction

  • SceneCode-generated movable parts remain independent links with compiled joints, allowing a robot to open or slide them through contact in MuJoCo.Figure 5 demonstrates the resulting simulation interaction.Source: §4.4 · p. 9, Figure 5 · p. 9

5 Conclusion

SceneCode concludes that executable, part-aware programs unify scene geometry and interaction structure to produce prompt-faithful scenes with editable, simulation-ready assets. The related-work discussion distinguishes this full-scene integration from retrieval-based assets and methods focused on articulated-object priors or URDF inference.

5 Conclusion

  • SceneCode combines a room-level planner and execution-guided object-program synthesis to generate prompt-faithful scenes with editable, interaction-ready assets.The framework represents geometry, semantic structure, materials, articulation, and simulation metadata in executable, part-aware programs.Source: §5 · p. 9

A More Related Work

  • Related systems often rely on retrieved static meshes, while articulated-object methods focus on part-graph priors or URDF inference rather than full-scene executable integration.Other VLM-driven code-generation systems synthesize Blender Python or domain-specific code for single-object modeling or scene-level visual layout.Source: A More Related Work · p. 13

B Route-Specific Prompt Design for Code-Driven Object Generation

SceneCode routes object requests to specialized construction prompts while preserving one shared code-generation and validation pipeline. Route-specific priors improve object structure and articulation readiness over generic prompting.

B Route-Specific Prompt Design for Code-Driven Object Generation

  • All profiles share an ObjectPlan schema and planner–checker–constructor structure, differing in the construction priors supplied to the VLM.The profiles guide generation toward executable, geometrically reliable Blender Python code.Source: B Route-Specific Prompt Design for… · pp. 13–14

B.1 Routing Strategies and Object Categories

  • The router selects among five strategies using category, description, support relation, size, and affordances, prioritizing requests with functional movable parts.Print-like wall objects, large rigid furniture, and tabletop or handheld objects are routed to family-specific profiles.Source: §B.1 · p. 14

B.2 Prompt Feature Taxonomy and Targeted Failure Modes

  • Specialized construction priors target fragile geometry, including symmetry, radial copies, curves, Boolean-cut faces, repeated components, and hollow structures.Profiles tailor decomposition and construction rules to object families, while shared engineering rules support materials and export conventions.Source: §B.2 · p. 14

B.3 Interaction with Articulated Asset Compilation

  • Articulated requests preserve movable parts as independent components, then compile those parts into links and supported revolute or prismatic joints for URDF-to-SDF export.Other profiles are packaged as rigid assets unless the articulated branch is selected.Source: §B.3 · pp. 14–15

B.4 Shared Structure and Style Preservation Across Profiles

  • Profiles retain a common pipeline while varying construction priors; conservative routing also preserves requested style and avoids unsupported materials, colors, or mechanisms.Separate objects in a multi-object request may be routed independently.Source: §B.4 · p. 15
  • Route-specific prompts improve VLM-based Blender code reliability by matching construction constraints to object-family geometry and function without fragmenting verification, repair, and packaging.The ObjectPlan is verified, programs are executed with up to three repair attempts, and rendered outputs can receive up to two critic-driven refinements.Source: §B.2 · p. 14, §B.4 · p. 15

B.5 Why Route-Specific Construction Prompts Matter

  • Route-specific prompts better preserve semantic parts, internal structure, material grounding, and articulation-ready components than generic prompting.Generic prompts can still produce visually plausible assets, but may omit structures such as nightstand drawers.Source: Figure 6 · p. 15, §B.5 · p. 15

C Evaluation Prompts

The evaluation prompt set contains 30 indoor-scene descriptions across six room categories, spanning varied object combinations and layout detail.

C Evaluation Prompts

  • The benchmark includes 30 prompts across bedroom, living room, dining room, kitchen, bathroom, and basement categories, with 15 easy, 12 medium, and 3 hard prompts.Source: C Evaluation Prompts · pp. 15–16

C.1 Bedroom

  • Bedroom prompts range from a bed-and-wardrobe scene to detailed layouts with multiple furnishings, storage, and wall decorations.Source: §C.1 · p. 16

C.2 Living Room

  • Living-room prompts cover basic furniture groupings, seating and table arrangements, and more detailed scenes with specified furnishings and room atmosphere.Source: §C.2 · p. 16

C.3 Dining Room

  • Dining-room prompts include bar-table and chair arrangements, cabinets, and tables with chairs or decorative items.Source: §C.3 · p. 17

C.4 Kitchen

  • Kitchen prompts vary from a bistro table and refrigerator to counters, cabinets, bar seating, and dining or serving arrangements.Source: §C.4 · p. 17

C.5 Bathroom

  • Bathroom prompts specify combinations of tubs, vanities, mirrors, toilets, hampers, baskets, or shelving.Source: §C.5 · p. 17

C.6 Basement

  • Basement prompts cover recreation, exercise, movie, and gaming layouts with distinct furniture and equipment.Source: §C.6 · p. 17

D.1 Scene-Level Metrics

Scene-level metrics assess prompt satisfaction and physical plausibility, while object-level metrics characterize mesh topology, geometric compactness, material editability, and UV organization.

D.1 Scene-Level Metrics

  • Scene-level metrics assess explicit prompt requirements through object counts, attributes, and object-to-object or object-to-architecture relationships.SceneEval also includes implicit physical-plausibility measures.Source: §D.1 · p. 18
  • Physical-plausibility metrics check support, functional-side accessibility, navigable free floor space, and object collisions.Support classification uses rendered images and ray casting; accessibility uses functional sides and 2D occupancy analysis.Source: §D.1 · p. 18
  • Out-of-bounds placement and opening-clearance metrics detect objects outside the room footprint and objects obstructing door or window clearance volumes.Source: §D.1 · p. 18

D.2 Object-Level Mesh and Material Metrics

  • The asset evaluation reports material slots, PBR channel coverage, non-manifold edges, face and vertex counts, and UV islands, averaged across generated assets.These metrics represent material editability, rendering readiness, topology, compactness, and UV organization.Source: §D.2 · p. 19
  • Material-slot count reflects material-region editability, while PBR coverage measures represented channels and UV-island count characterizes layout organization.Fewer UV islands indicate a more coherent parameterization that is easier to inspect, texture, and edit.Source: §D.2 · p. 19
  • Non-manifold edge counts measure topology cleanliness, while face and vertex counts characterize mesh size and geometric compactness.Lower face counts indicate a lighter representation only under comparable visual quality.Source: §D.2 · p. 19

E SceneCode Assets versus Prior Asset Strategies

SceneCode represents assets as executable, part-aware programs rather than fixed meshes or opaque generated shapes. This representation couples geometry, semantic structure, articulation, editing, and simulation packaging at the object level.

E SceneCode Assets versus Prior Asset Strategies

  • Prior asset strategies include retrieval of static meshes, retrieval of articulated assets, and generation of opaque meshes, each with constraints on interaction or editability.Static retrieval systems include LayoutGPT [6], ATISS [21], InstructScene [14], Holodeck [39], LayoutVLM [28], and HSM [23]; SceneSmith [22] uses articulated retrieval and image-to-3D generation.Source: E SceneCode Assets versus Prior Asset… · p. 20
  • Static retrieval depends on existing libraries, articulated retrieval requires a suitable dataset asset, and opaque meshes lack editable functional parts and reliable simulator links or joints.Source: E SceneCode Assets versus Prior Asset… · p. 20
  • SceneCode couples geometry construction, part hierarchy, and packaging metadata in one executable object program, supporting novel variants and editable substructure.Movable parts can be marked before export for downstream simulation packaging.Source: E SceneCode Assets versus Prior Asset… · p. 20
  • Unlike SceneCode, SceneSmith [22] relies on curated articulated-object coverage for interactable assets and produces image-to-3D meshes without semantic parts or code-level editability.SceneCode can declare prismatic or revolute joints and other simulation properties within the object program.Source: E SceneCode Assets versus Prior Asset… · p. 20
  • SceneCode’s stated advantages include on-demand articulated coverage, first-class part hierarchies and joints, local code editing and regeneration, and unified simulation packaging.Individual dimensions, materials, joint ranges, and drawer counts can be edited and re-executed locally.Source: E SceneCode Assets versus Prior Asset… · p. 20

F More Analysis

SceneCode leads on semantic fidelity and the reported physical-plausibility metrics, while SceneSmith leads on inter-object and architectural relations. Other baseline-leading metrics reflect specific design choices, and the appendix describes corresponding trade-offs.

Semantic fidelity (CNT, ATR)

  • SceneCode leads on CNT (0.7951) and ATR (0.7344), with its largest stated semantic-fidelity gap over LayoutVLM on ATR (0.4236).Its CNT margin over SceneSmith is +0.0068; the authors attribute the attribute gains to code that realizes planner-specified attributes during construction.Source: F More Analysis · p. 21

Physical plausibility (NAV, COL, OOB)

  • SceneCode leads on NAV (0.9999), COL (0.0934), and OOB (0.0042), with collision rates roughly half those of each baseline.The authors attribute these results to clean, axis-aligned, bounding-box-faithful geometry used by both rendering and placement checks.Source: F More Analysis · p. 21

Inter-object and architectural relations (OOR, OAR)

  • SceneSmith leads on OOR (0.3723) and OAR (0.7241); SceneCode is second on OOR (0.3472) and scores 0.6078 on OAR.The authors link the gap partly to skipping object programs that fail ObjectPlan checks beyond the retry budget, which can remove relation-bearing objects.Source: F More Analysis · p. 21

Per-baseline trade-offs (SUP, ACC, OPC) and overall

  • HSM leads on SUP (0.7072), LayoutVLM on ACC (0.9330), and SceneSmith on OPC (0.0792); SceneCode trails on SUP and OPC.The authors tie SceneCode's localized weaknesses to the absence of per-object support-type classification and an opening-clearance check; LayoutVLM's sparse scenes are said to inflate accessibility.Source: F More Analysis · p. 21

G Full User Study Results

The user study compares SceneCode with three baselines using paired ratings and preferences on the same prompts. The appendix also reports token-based monetary cost and wall-clock runtime statistics.

G Full User Study Results

  • Nine participants were split into three groups, each comparing SceneCode with one baseline on the same 30 prompts using pairwise preference and 1–5 ratings.Because SceneCode was rated by all nine participants but each baseline by only its own three-person group, the authors use within-trial paired differences rather than comparing absolute means across baselines.Source: G Full User Study Results · p. 22
  • Baseline-preferred fractions were 3.3% against LayoutVLM, 20.0% against HSM, and 36.7% against SceneSmith.The study used 90 paired trials per comparison; Table 6 reports preference loss rate and paired prompt-faithfulness and realism differences with 95% confidence intervals.Source: G Full User Study Results · p. 22, Table 6 · p. 26
  • SceneCode's realism ratings were higher than HSM (+0.27) and LayoutVLM (+1.93), but lower than SceneSmith (−0.34).Against SceneSmith, SceneCode still had a positive prompt-faithfulness difference (+0.14); the authors relate the realism trade-off to visually rich retrieved meshes versus attribute-faithful programmatic geometry.Source: G Full User Study Results · p. 23

H Computational Cost Statistics

I.1 Scene Demo

The demonstrations span coherent room scenes, editable object assets, and articulated household objects. A nightstand example shows how a generated drawer is constructed in Blender and represented as a movable simulation link.

I.1 Scene Demo

  • Room demonstrations show requested objects arranged in plausible configurations while retaining prompt-specific attributes and free space for navigation and physical use.Examples span kitchens, basements, dining rooms, living rooms, bathrooms, and bedrooms.Source: §I.1 · pp. 28–29, Figure 8 · p. 28

I.2 Object Demo

  • Object demonstrations pair rendered appearances with wireframes that expose editable geometry, parts, dimensions, and material assignments.Examples include text- and image-like wall art, a numbered billiard ball, and a board-game box with layered structure.Source: §I.2 · p. 29, Figure 9 · p. 29

I.3 Articulated Object Demo

  • Articulated storage objects preserve interiors such as shelves and dividers, with independent doors, drawers, and lids supporting opening or sliding behavior.The code-conditioned assets allow dimensions, internal layout, materials, and joint configuration to be customized on demand.Source: Figure 10 · p. 29, §I.3 · p. 30

J Executable Object-Code Demonstration

  • The nightstand example constructs a movable drawer from geometric primitives and procedural materials, then exports it as an independent SDF link with a prismatic joint.The drawer's code exposes semantic subparts and construction properties rather than storing it as an opaque mesh.Source: J Executable Object-Code Demonstration · p. 30, §J.1 · p. 30

J.1 Blender Drawer Construction Program

  • The Blender drawer program builds named front, side, back, and bottom components plus a knob, applies materials and beveling, and sets the origin for movement.The listed code groups the components into a drawer object while retaining construction dimensions and positions.Source: §J.1 · pp. 30–32

J.2 SDF Drawer Link and Articulation

  • The SDF articulation metadata connects the drawer link to the nightstand base with a prismatic joint that slides along negative y over [0.0, 0.4].The drawer link also carries inertial properties, a visual mesh, and collision proxies.Source: §J.2 · pp. 32–33

K Limitations

The authors identify scope, visual-detail, and runtime boundaries for SceneCode. Their proposed future directions address broader outdoor settings, richer asset appearance, and faster generation.

Scope

  • SceneCode focuses on physically interactable indoor scenes organized by functional and architectural priors; extending the code-driven approach to outdoor or large-scale mixed environments remains open.The authors note that such settings may require additional procedural priors and verification strategies.Source: K Limitations · p. 33

Visual richness

  • Primitive-based programs may not capture the fine-grained surface detail or photorealistic texture variation available from retrieval or image-to-3D pipelines.The authors suggest combining programs with neural texture synthesis, material refinement, or detail-preserving geometry augmentation.Source: K Limitations · pp. 33–34

Runtime

  • Program synthesis, execution, repair, and refinement introduce nontrivial runtime in an implementation emphasizing generation quality, traceability, and execution-guided validation.The authors anticipate improved throughput from parallel object generation and specialized code-generation models distilled for 3D asset construction.Source: K Limitations · p. 34

Related papers

Loading 2605.19587v1…