PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes

Nicholas Pfaff, Thomas Cohn, Sergey Zakharov, Rick Cory, Russ Tedrake

arXiv:2602.09153v2cs.ROcs.AIcs.CVcs.GR

TL;DR

Simulation can support scalable home-robot training and evaluation, but existing indoor scenes are often sparse and static compared with cluttered, articulated homes. SceneSmith generates simulation-ready indoor environments from natural-language prompts through hierarchical agentic scene construction. It generates 3-6x more objects than baselines, with <2% inter-object collisions and 95.6% static stability; human evaluators report 92% realism and 91% prompt-faithfulness win rates.

  • Problem

    Existing simulated indoor environments are often sparse, have limited object diversity, and are largely static compared with real homes’ dense clutter and articulated furniture, which challenge robotic manipulation.

    Source: §1 · p. 2

  • Method

    SceneSmith generates simulation-ready indoor environments from natural-language prompts through hierarchical scene-construction stages, each driven by designer, critic, and orchestrator VLM agents.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    SceneSmith generates 3-6x more objects while achieving <2% inter-object collisions and 95.6% static stability; human evaluators give it 92% realism and 91% prompt-faithfulness win rates.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors demonstrate SceneSmith in an end-to-end pipeline for automatic robot policy evaluation, from natural-language task descriptions to automatic success verification.

    Source: §1 · p. 3

    Current text-to-3D approaches do not yet reliably produce articulated structures and kinematics suitable for robotics simulation.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces. Current scene synthesis methods produce sparsely furnished rooms that lack the dense clutter, articulated furniture, and physical properties essential for robotic manipulation. We introduce SceneSmith, a hierarchical agentic framework that generates simulation-ready indoor environments from natural language prompts. SceneSmith constructs scenes through successive stages$\unicode{x2013}$from architectural layout to furniture placement to small object population$\unicode{x2013}$each implemented as an interaction among VLM agents: designer, critic, and orchestrator. The framework tightly integrates asset generation through text-to-3D synthesis for static objects, dataset retrieval for articulated objects, and physical property estimation. SceneSmith generates 3-6x more objects than prior methods, with <2% inter-object collisions and 96% of objects remaining stable under physics simulation. In a user study with 205 participants, it achieves 92% average realism and 91% average prompt faithfulness win rates against baselines. We further demonstrate that these environments can be used in an end-to-end pipeline for automatic robot policy evaluation.

1. Introduction

SceneSmith addresses the mismatch between sparse simulated rooms and physically complex real homes by generating simulation-ready indoor environments from natural-language prompts. It combines hierarchical scene construction, integrated asset generation and routing, and physical-property estimation for scalable robotic training and evaluation.

1. Introduction

  • Sparse simulated rooms lack the dense object arrangements, articulated furniture, and fine-grained clutter of real homes, limiting simulation-based training and evaluation (Maddukuri et al., 2025; Barreiros et al., 2025a).Manual scene design can produce high-quality environments but is costly, time-consuming, and difficult to scale.Source: §1 · p. 2
  • The framework jointly generates simulation-ready assets and house-level scenes, unlike asset-focused or scene-focused prior work and methods centered on furniture-level layout and visual realism (Pfaff et al., 2025a; Yang et al., 2024b; Pfaff et al., 2025b; Deitke et al., 2022b; Tang et al., 2024; Yang et al., 2025).Robotic scenes require dense manipulable objects, hierarchical support relationships, and physically valid configurations.Source: §1 · p. 2
  • SceneSmith builds scenes from architectural layout through furniture placement to small-object population, using designer, critic, and orchestrator VLM agents for iterative refinement.Rooms and supporting surfaces form independent branches; asset routing combines text-to-3D for static objects, retrieval for articulated furniture, and physical-property estimation.Source: §1 · pp. 2–3
  • 92.2% average realism win rate and 91.5% average prompt faithfulness win rate against baselines in a 205-participant user study.The evaluation covered 210 diverse room- and house-level prompts.Source: §1 · p. 3
  • 3-6x more objects than baselines (71.1 vs 11-23 objects per room on average), with <2% inter-object collisions and 95.6% object stability under physics simulation.Baselines had 3-29% collisions and 8-61% stability.Source: §1 · p. 3
  • SceneSmith demonstrates end-to-end robot policy evaluation, converting natural-language tasks into scene prompts, running simulated policies, and verifying task completion with symbolic state and visual observations.The paper also reports teleoperation demonstrations with an RB-Y1 humanoid and zero-shot policy rollouts.Source: §1 · p. 3

2. Related Work

Prior indoor scene synthesis spans procedural, data-driven, and language-guided approaches, but leaves gaps in semantic flexibility, scene density and diversity, spatial and physical feasibility, or robotics-ready properties. SceneSmith builds on hierarchical and agentic precedents while adding coordinated placement and agent self-verification.

Related Work

  • Procedural methods scale but lack semantic expressivity, while data-driven models use restrictive floor-aligned SE(2) layouts and produce sparse scenes with limited object diversity.The cited procedural examples are ProcTHOR (Deitke et al., 2022b) and Infinigen Indoors (Raistrick et al., 2024); data-driven examples include Tang et al. (2024), Yang et al. (2024a), and Hu et al. (2024).Source: §2 · p. 3
  • Language-guided methods add open-vocabulary control but often struggle with fine-grained spatial reasoning and physical feasibility; prior methods also lack robotics-required collision geometry and physical properties.The cited language-guided works are Yang et al. (2024b), C¸ elen et al. (2025), Sun et al. (2024), Zhou et al. (2025), and Sun et al. (2025).Source: §2 · p. 3
  • SceneSmith adopts HSM’s hierarchical furniture-then-small-object generation and support-surface detection, adding globally constrained prompt refinement and joint surface population for coordinated placement.Joint population can coordinate related surfaces, such as placing books on one shelf and plants on another.Source: §2 · p. 3
  • SceneSmith adapts LL3M’s designer-critic refinement to scene generation and extends agentic synthesis with unlimited per-turn tool use, visual observations, and state feedback for self-verification.SceneWeaver uses a single LLM planner that selects one tool per turn; LL3M (Lu et al., 2025) iteratively refines 3D assets between a designer and critic.Source: §2 · p. 3

3. Agentic Scene Construction for Simulation-Ready Environments

SceneSmith builds simulation-ready indoor environments from natural-language prompts through hierarchical stages implemented as interactions among designer, critic, and orchestrator agents. Its pipeline combines scene construction, asset acquisition, physical-feasibility post-processing, and robot-policy evaluation.

3.1. Scene Hierarchy

  • SceneSmith builds scenes from room layouts through furniture, wall- and ceiling-mounted objects, then manipulable objects on selected supporting entities.Room-specific and entity-specific prompts refine the original scene intent; the resulting branches are assembled into a flat scene representation for simulator export.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2. Agentic Trio: Designer, Critic, and Orchestrator

  • The designer proposes scene changes, the critic scores plausibility, feasibility, and stage alignment, and the orchestrator accepts, refines, or rolls back proposals.Agents use role-dependent structured tools for observation, modification, acquisition, and feasibility checks; stage-specific memory retains recent turns and summarizes earlier ones.Source: §3.2 · pp. 4–5

3.3. Asset Generation and Routing

  • The asset router selects generated static assets, retrieved articulated assets, or thin coverings, decomposes composite requests, and validates candidates before returning simulation-ready objects.Static meshes are synthesized from text-derived images, while articulated objects are retrieved because current text-to-3D methods do not reliably produce robotics-suitable articulation and kinematics; coverings add visual detail without rigid-body complexity.Source: §3.3 · p. 5

3.4. Simulation Readiness

  • Post-processing makes scenes directly simulatable by resolving object penetrations and settling unstable objects under gravity into static equilibrium.SceneSmith augments objects with estimated physical properties; architectural elements use volumetric geometry, and feasibility correction follows furniture and manipuland placement.Source: §3.4 · pp. 5–6

3.5. Application: Robot Policy Evaluation

  • SceneSmith enables end-to-end robot-policy evaluation by generating task-relevant environments, running robot policies in simulation, and using an evaluator agent to verify task completion.The evaluator combines symbolic scene state with rendered visual observations rather than fixed success predicates, supporting open-ended tasks at the cost of determinism.Source: §3.5 · p. 6

4. Evaluation

Evaluation shows strong human preference for SceneSmith scenes and favorable automated collision and stability results. Policy evaluation and robot simulation demonstrations further test its use in robotic workflows.

4.1. Experimental Setup

  • The evaluation covers 210 prompts across five categories, compares five baselines, and combines pairwise human preferences with SceneEval and Drake physics metrics.The human study collected 3,051 valid responses from 205 participants; Drake metrics assess collision rate and static equilibrium, and baseline scenes were augmented with collision geometry and physical properties.Source: §4.1 · pp. 6–7

4.2. Scene Generation Results

  • NotGenerated scores 63.8% realism and 67.0% faithfulness, NoAssetValidation scores 63.0% and 62.2%, and NoObserveScene scores 61.5% realism.These ablations show significant effects of generated asset acquisition, asset validation, and visual feedback. NoCritic preferences (51–55% realism) did not reach significance; it is 70% cheaper but produces 24% fewer objects.Source: §4.2 · p. 7
  • SceneSmith achieves 92.2% average realism and 91.5% average prompt faithfulness win rates against room-level baselines (all p < 0.001), and 80.3% and 84.7% against Holodeck for house scenes.The smallest room-level margins are against HSM: 88.5% realism and 85.2% faithfulness.Source: §4.2 · p. 7, Table 1 · p. 8
  • SceneSmith leads on CNT, ATR, OOR, OAR, COL, and STB, with 2.2x OOR improvement; collision rate is 1.2% vs 3-29%, and stability 95.6% vs 8-61%.Room-level scenes contain 71.1 vs 11-23 objects; lower ACC and NAV scores are attributed to higher object density. House-level scenes have 214 vs 81 objects, 0.9% vs 3.8% collisions, and 79.8% vs 17.9% stability. Remaining collisions average 3.8mm penetration depth.Source: §4.2 · p. 7, Table 2 · p. 8
  • Generated hotels connect rooms plausibly, with en suite bathrooms accessible through their associated bedrooms and all rooms linked by a central hallway, unlike implausible Holodeck layouts.Examples of Holodeck failures include a hotel reachable only through a guest room or a second bedroom accessible only through the first bedroom’s bathroom.Source: §4.2 · pp. 7–8

4.3. Robot Policy Evaluation

  • Across 100 generated scenes and four pick-and-place tasks, the standard policy achieves 16% success versus 12% for a degraded policy; evaluator judgments agree with human labels 99.7%.All 300 evaluator judgments were manually verified across initial, standard-policy, and degraded-policy states; the sole disagreement involved fruit landing on a plate edge.Source: §4.3 · p. 8

4.4. Robot Simulation Demonstrations

  • A zero-shot text-conditioned policy from Lin et al. (2026), trained without SceneSmith data, recognizes an apple and cutting board and places the apple as instructed.Teleoperation demonstrations also show an RB-Y1 opening an articulated cabinet and placing a bottle inside.Source: §4.4 · pp. 8–9, Figure 5 · p. 9

5. Conclusion

SceneSmith generates simulation-ready indoor scenes through hierarchical agentic construction and an integrated asset pipeline. The conclusion reports denser, physically valid scenes, strong evaluator preference, and suitability for interactive robot simulation.

5. Conclusion

  • SceneSmith generates 3–6x more objects, with <2% inter-object collisions and 95.6% static stability, versus baseline collisions of 3–29% and stability of 8–61%.Source: §5 · p. 9
  • SceneSmith builds scenes through hierarchical stages driven by designer–critic–orchestrator interactions and combines text-to-3D synthesis with articulated-object retrieval.This integrated pipeline enables open-vocabulary generation while ensuring simulation readiness.Source: §5 · p. 9
  • Human evaluators preferred SceneSmith with a 92% average win rate for realism and a 91% average for prompt faithfulness.Source: §5 · p. 9
  • Qualitative teleoperation and zero-shot policy rollouts indicate that generated scenes support interactive robot simulation beyond static scene-quality metrics.Source: §5 · p. 9

Impact Statement · Appendix

SceneSmith aims to support safer robot evaluation before real-world deployment through improved simulation environments, and the authors foresee no direct negative societal impacts.

Impact Statement

  • The work aims to enable safer robot evaluation before real-world deployment through improved simulation environments, with no direct negative societal impacts foreseen.Source: Impact Statement · p. 10

A. System Architecture … B.6. Collision Geometry

SceneSmith builds indoor scenes through five staged Designer-Critic-Orchestrator cycles, then acquires and processes assets for simulation. Its agents observe and refine scenes, while parallel asset generation and specialized pipelines support scale, interactability, and physical simulation.

A. System Architecture; A.1. Agent Roles; A.2. Model Configuration

  • Five sequential stages—layout, furniture, wall-mounted objects, ceiling-mounted objects, and manipulands—use designer, critic, and orchestrator agents to construct scenes.Designers modify scenes with stage-specific tools, critics score quality from 0–10 and suggest improvements, and orchestrators coordinate iterations, checkpoints, and termination; agents use GPT-5.2 with reasoning effort varying by role and task.Source: A. System Architecture · p. 13, §A.1 · p. 13, §A.2 · p. 15

A.3. Scene Observation; A.4. Checkpoint and Rollback

  • Agents combine structured spatial queries with annotated multi-view renders, while checkpoint rollback lets designers retry failed changes without losing agent context.Checkpoints retain complete scene state and scores; when scores fall beyond thresholds, the orchestrator restores the previous state and prompts an alternative approach.Source: §A.3 · p. 15, §A.4 · p. 15

A.5. Parallel Scene Generation; B. Asset Acquisition

  • Batching asset requests across a GPU worker pool overlaps text-to-3D generation with mesh processing; 25 scenes ran concurrently across 8 NVIDIA L40S GPUs with nearly constant per-scene generation time.Completed assets undergo collision-geometry extraction and physics estimation while GPUs process remaining requests.Source: §A.5 · p. 15

B.1. Asset Router

  • A VLM router filters assets by construction stage, decomposes composite requests into manipulable parts, and routes standard objects to text-to-3D, articulated objects to retrieval, and thin coverings to procedural generation.For example, it separates a desk from its monitor and a fruit bowl from its fruits; articulated assets include objects with moving parts, while coverings include flat textured surfaces.Source: §B.1 · pp. 15–16

B.2. Text-to-3D Generation; B.3. Articulated Object Retrieval

  • Standard assets use a GPT Image 1.5–SAM3–SAM3D pipeline, while articulated objects are retrieved from ArtVIP using type filtering, CLIP similarity, and optional bounding-box dimension re-ranking.The image pipeline generates a reference image, segments its foreground, and reconstructs a textured 3D mesh; articulated retrieval uses precomputed multi-view embeddings, and the authors report that PartNet-Mobility (Xiang et al., 2020) had insufficient visual and joint quality.Source: §B.2 · p. 16, §B.3 · p. 16

B.4. Thin Covering Generation

  • Thin coverings support rectangular or circular shapes and use retrieved PBR textures when available, falling back to generated artwork or seamless tileable textures with edge blending.Tileable textures suit repeating patterns, while single images cover artwork; coverings are decorative and have no collision geometry.Source: §B.4 · p. 16

B.5. Physical Property Estimation; B.5.1. RIGID BODY ESTIMATION; B.5.2. ARTICULATED BODY ESTIMATION

  • A VLM estimates rigid-body material, mass ranges, and canonical orientation from six views; articulated-object estimates assign per-link materials and masses constrained to sum to the estimated total mass.Rigid-body inertia is computed from mesh volume and geometry assuming uniform density, and canonicalization establishes Z-up, Y-forward coordinates for support-surface placement.Source: §B.5.1 · p. 17, §B.5.2 · p. 17

B.6. Collision Geometry

  • Interactable visual meshes are decomposed into convex pieces with V-HACD, capped at 128 for furniture, 64 for wall objects and manipulands, and 16 for ceiling objects.Compared with CoACD, V-HACD avoids inflated collision geometry that can make stacked objects appear to float and is reported to compute one to two orders of magnitude faster.Source: §B.6 · p. 17

B.7. Asset Validation … C.4. Layout Validation

SceneSmith validates generated and retrieved assets with VLM checks and builds room layouts through an iterative placement process. Its layout workflow carries room-specific prompts through downstream agents and validates exterior-to-room connectivity.

B.7. Asset Validation

  • Asset validation checks object type, style, singularity, completeness, proportions, and closed state, then retries failures or returns feedback to the designer.Validation is lenient about text and labels, metallic or transparent materials, and cosmetic or minor structural differences in retrieved assets; Figure 7 illustrates rejected and accepted generations.Source: §B.7 · pp. 17–18, Figure 7 · p. 18

C. Layout Generation; C.1. Room Layout Creation

  • The layout agent turns a global prompt into room geometry and room-specific prompts, while designers can iteratively regenerate layouts, rooms, or architectural details.Room-specific prompts carry room furniture, colors, features, house-wide style, and resolved room assignments to downstream agents.Source: C. Layout Generation · p. 18, §C.1 · p. 18

C.2. Room Placement Algorithm

  • Room placement uses best-first backtracking over candidate positions, returning the best layout found before timeout; it is complete and optimal only within fixed ordering and discretized positions.The first room is placed at the origin, later rooms are attached to placed rooms, and the search backtracks to test alternatives.Source: §C.2 · p. 19

C.2.1. ROOM ORDERING

  • Room ordering places anchors first, then connectors once a required neighbor is placed, sorting by descending area within each category.This lets adjacency constraints guide placement while prioritizing larger, more constrained rooms.Source: §C.2.1 · p. 19

C.2.2. EDGE-BASED ATTACHMENT

  • New rooms attach along existing room edges using 11 sampled positions per edge and both original and 90°-rotated orientations for non-square rooms.Source: §C.2.2 · p. 19

C.2.3. PLACEMENT SCORING

  • Placement scoring ranks candidates by adjacency and compactness cues, rejects candidates violating required adjacencies, and evaluates complete layouts for compactness and stability.Stability helps keep rooms near their previous positions during iterative editing.Source: §C.2.3 · pp. 19–20

C.3. Architectural Elements

  • After room placement, the designer adds walls, doors, windows, open connections, and semantically retrieved floor and wall materials, and can revise these elements or the layout.Door and window positions are sampled within specified wall segments; windows also have a specified sill height.Source: §C.3 · p. 20

C.4. Layout Validation

  • Layout validation confirms rooms were placed and uses breadth-first search from exterior doors to check that every room is reachable through interior doors or open connections.At least one exterior door is required; unreachable rooms are reported as errors.Source: §C.4 · p. 20

C.5. Critic Evaluation … D.4. check physics Tool

These sections describe evaluation and observation tools for layout design, followed by furniture-placement methods that align objects, preserve robot traversability, and detect scene-feasibility problems.

C.5. Critic Evaluation

  • The critic scores layouts from 0–10 across room proportions, spatial flow, natural lighting, material consistency, and prompt following.Source: §C.5 · p. 20

C.6. Observation

  • Layout-agent observations combine rendered scene views with an ASCII floor plan that labels wall segments for door and window placement.The tools also show room labels, materials and texture swatches, room dimensions, connectivity validation, and door/window specifications.Source: §C.6 · p. 20, Figure 8 · p. 21

D. Furniture Placement

  • Furniture agents process each room independently using its geometry and furnishing prompt, enabling parallel generation and smaller agent contexts.Designers specify floor-coordinate poses as (x, y, θ), with z = 0 and roll and pitch fixed to zero.Source: D. Furniture Placement · p. 22

D.1. check facing Algorithm

  • The check-facing tool verifies whether furniture faces toward or away from a target and returns the rotation needed for alignment.It derives the source yaw, tests whether its forward ray intersects the target, and computes an optimal rotation; circular targets are detected by a mesh-to-AABB volume ratio below 0.80.Source: §D.1 · p. 22

D.2. snap to object Algorithm

  • The snap-to-object tool combines collision resolution, optional toward/away orientation alignment, and iterative gap elimination for precise furniture placement.It pushes overlapping objects along the minimum-overlap axis, aligns yaw when requested, then advances until collision and reverts to the last collision-free position.Source: §D.2 · pp. 22–23

D.3. check reachability Algorithm

  • A configurable robot footprint guides reachability checks that identify whether walkable space is connected and which furniture may block it.The method erodes floor boundaries and dilates furniture by the robot radius, then checks the resulting walkable region; feedback includes disconnected regions, reachability ratio, and blocking furniture IDs.Source: §D.3 · p. 23

D.4. check physics Tool

  • The check physics tool detects furniture collisions, floor-covering overlaps or boundary violations, blocked doors and room connections, and window-clearance warnings.It uses signed-distance queries for furniture collisions and 2D oriented bounding-box intersections for floor-covering overlaps; reported issues include penetration depths and object identifiers.Source: §D.4 · pp. 23–24

D.5. Critic Evaluation … E.1. Physics Validation

These sections specify how the critic evaluates furniture layouts, how agents observe scenes, and how wall objects are placed and checked for geometric validity.

D.5. Critic Evaluation

  • The critic scores furniture placement from 0–10 on Realism, Functionality, Layout, Holistic Completeness, Prompt Following, and Reachability.Holistic Completeness measures expected furnishing for a room type, while Prompt Following checks explicit requests; a living room with only a requested chair can score high on the latter but low on the former.Source: §D.5 · p. 24

D.6. Observation

  • Scene observation provides one annotated top-down view and four corner views, hiding intervening walls in corner views and adding coordinate markers for spatial reference.Top-down annotations include a coordinate grid and frame, furniture bounding boxes and labels, facing-direction arrows, and labels for doors, windows, and open connections.Source: §D.6 · p. 24, Figure 9 · p. 24
  • Object identifiers combine a human-readable name with a base-36 sequential suffix (e.g., chair 0, chair a, chair 10), reducing visual clutter while remaining easy for the agent to reference.Figure 9 shows an example observation.Source: §D.6 · p. 24, Figure 9 · p. 24

E. Wall Object Placement

  • The wall agent adds art, mirrors, shelves, and clocks to each furnished room independently, using the room-specific prompt.It places wall-mounted objects across all walls jointly in a single session.Source: E. Wall Object Placement · pp. 24–25
  • Wall objects use local wall-frame SE(2) coordinates (x, z, θ), with x along the wall, z from floor to ceiling, rotation around the wall normal, and implicit y=0.Each wall has a unique identifier; the agent specifies the target wall and local pose, while the system excludes doors, windows, and open connections.Source: E. Wall Object Placement · p. 25

E.1. Physics Validation

  • The physics check detects wall-object interpenetrations, placements beyond wall edges or above ceiling height, and overlaps with doors, windows, or open connections.Source: §E.1 · p. 25

E.2. Critic Evaluation … F.1. Physics Validation

The sections describe how wall-object placement is evaluated and observed, then how ceiling fixtures are placed and checked for collisions.

E.2. Critic Evaluation

  • The wall-placement critic scores realism, functionality, layout, holistic completeness, and prompt following on a 0–10 scale.Source: §E.2 · p. 25

E.3. Observation

  • The scene-observation tool pairs a top-down view of the full room with orthographic views of individual walls for precise placement.Source: §E.3 · p. 25
  • Wall views show wall-local coordinates, wall identifiers, openings, object labels, and nearby furniture as spatial context.Source: §E.3 · p. 25

F. Ceiling Object Placement

F.1. Physics Validation

  • The physics-checking tool detects collisions between ceiling fixtures and other scene objects.Source: §F.1 · p. 27

F.2. Critic Evaluation … L.3. Example Policy

This section describes SceneSmith’s scene population and physical-feasibility tools, prompt-conditioned placement styles, and critic feedback, along with simulator export and policy evaluation. It also covers task-prompt diversification, evaluation of task success from final scene states, and model-based robot execution of pick-and-place tasks.

F.2. Critic Evaluation

  • Placement critics score realism, functionality, layout, and prompt following, while manipuland evaluation also scores holistic completeness.The ceiling-fixture critic uses four categories scored 0–10; the manipuland critic adds holistic completeness, also using 0–10 scores.Source: §F.2 · p. 27, §G.4 · p. 29

G. Manipuland Placement

  • The manipuland agent populates selected furniture, wall, and architectural surfaces, using entity-specific prompts and local surface coordinates to guide placements.A VLM derives required and optional items, prompt constraints, and style notes, while support surfaces are extracted from meshes using the HSM algorithm (Pun et al., 2026).Source: G. Manipuland Placement · p. 28, §G.1 · p. 29, §G.2 · p. 29

G.6. Composite Object Tools

  • Composite tools create stacks, fill containers, place precise arrangements, or form random piles, using physics simulation to settle objects and tool-specific validation.Stacks are rejected if objects fall or exceed displacement thresholds; container filling retries objects that fall out, while arrangements fail if any item falls off and piles require at least two objects to remain.Source: §G.6 · p. 30, §G.6.1 · p. 30, §G.6.2 · p. 32

H. Physical Feasibility Post-Processing

  • Post-processing projects poses to collision-free configurations, settles objects under gravity, and removes objects detected as fallen.Projection keeps orientations fixed; the pipeline runs after furniture placement, after each entity’s manipulands, and again after all manipulands are placed, with furniture and manipulands handled in different scene configurations.Source: §H.1 · p. 34, §H.2 · p. 34, §H.3 · p. 34

I. Stochastic Placement

  • Gaussian placement noise adds variation, with prompt keywords selecting natural or perfect styles and critics calibrating expectations accordingly.Natural mode is the default and uses 1–3 cm position and 0.5–3° rotation noise; natural style accepts subtle imperfections, whereas perfect style expects alignment within tolerances.Source: §I · p. 35, §I.1 · p. 35, §I.2 · p. 35

J. Export and Simulator Compatibility

  • SceneSmith uses Drake for simulation and supports MJCF and USD exports with geometry, physical properties, and articulated joints.Its native scene representation is SDFormat; Blender export is also available for visualization and rendering, with visual geometry only.Source: J. Export and Simulator Compatibility · p. 35

L. Robot Policy Evaluation

  • The policy-evaluation pipeline uses an LLM to generate task-enabling scene prompts, SceneSmith to generate scenes, and a robot policy and evaluation agent to assess task success.The evaluation agent verifies success using simulator state queries and visual observations, enabling evaluation without manual environment or success-predicate design.Source: Figure 19 · p. 36, L. Robot Policy Evaluation · p. 36, §L.1 · p. 36

L.1. Scene Prompt Generation

  • The prompt generator identifies room requirements, required objects, initial-state constraints, and flexible dimensions, then creates varied scenes with required objects in non-goal positions.Prompts vary in style, object variants, placement configurations, and contextual objects before each is passed to SceneSmith for scene generation.Source: §L.1 · pp. 36–37

L.2. Evaluator Agent

  • The evaluator tests candidate object pairs with geometric and multi-view visual tools, using vision for ambiguous cases to assess diverse tasks without hand-crafted success predicates.It receives the task description and final scene state; geometric tools check object pairs, distances, contacts, and surface overlap, while renders provide top-down and side views.Source: §L.2 · p. 37

L.3. Example Policy

  • The example policy uses a KUKA LBR iiwa14 on a mobile base, parsing tasks into goal predicates, object categories, and optional preconditions before ranking candidate object bindings.A VLM performs task parsing and object matching; the paper notes that a generalist policy could instead act directly from natural language.Source: §L.3 · p. 37
  • The policy samples collision-free antipodal grasps, solves collision-aware pick-and-place inverse kinematics, and plans segmented trajectories with gripper clearance and motion limits.The method uses RRT-Connect, randomized shortcutting (Sekhavat et al., 1998, §6.4.1), and time-optimal path parameterization; grasp sampling and inverse kinematics are described in Tedrake (2024, §§5.4, 6.1).Source: §L.3 · pp. 37–38
  • The deliberately inferior policy omits gripper pose velocity and acceleration limits and increased gripper clearance, making object drops and environmental collisions more likely.Source: §L.3 · p. 38

L.4. Experimental Results … M.1.1. BASELINES

The automatic evaluator matched human judgments in 299/300 cases and detected a difference in overall success rates between standard and degraded policies. The evaluation covered four pick-and-place tasks and compared five external LLM/VLM scene-generation baselines.

L.4. Experimental Results

  • The study evaluated four pick-and-place tasks using 25 generated scenes per task, including one house-level task spanning the kitchen and living room.Tasks included moving fruit, a can, a cup, or a water bottle to a specified destination.Source: §L.4 · p. 38
  • The evaluator agreed with human judgment in 299/300 evaluations (99.7%), with the sole disagreement involving a fruit on the edge of a plate.The evaluator judged the edge landing a failure, while the human labeled it a success.Source: §L.4 · p. 38
  • 16% overall success for the standard policy versus 12% for the degraded variant shows the evaluation system detected a meaningful policy-performance difference.Table 12 presents per-task success rates across 100 generated scenes.Source: Table 12 · p. 38, §L.4 · p. 38

M. Evaluation Methodology

M.1.1. BASELINES

  • Five external LLM/VLM scene-generation baselines were evaluated using open-source code; HSM recursively generates furniture and small objects using scene motifs and HSSD retrieval (Pun et al., 2026).HSM resolves collisions through spatial optimization.Source: §M.1.1 · p. 38
  • Holodeck (Yang et al., 2024b) converts LLM-generated object relations into layouts with a constraint solver and is the only baseline supporting both room- and house-level generation.Its approach separates semantic understanding from geometric optimization to avoid collisions and boundary violations.Source: §M.1.1 · p. 38
  • I-Design (C¸ elen et al., 2025) builds scene graphs through specialized prompt roles and backtracking, while SceneWeaver iteratively refines scenes using VLM feedback.SceneWeaver uses evaluation results to select refinement tools and a standardized interface to maintain spatial consistency.Source: §M.1.1 · pp. 38–39
  • LayoutVLM (Sun et al., 2024) combines visual prompting with differentiable optimization; its open-vocabulary extension was evaluated with curated and Objathor asset libraries (Deitke et al., 2022a;b).The extension retrieves objects from a library before estimating poses and spatial relations and optimizing for coherence and collision avoidance.Source: §M.1.1 · p. 39

M.1.2. ABLATIONS … M.2.1. PHYSICS METRICS FOR SIMULATION READINESS

These sections specify SceneSmith’s ablations and evaluation scope, defining the prompt corpus, semantic benchmark metrics, and Drake-based measures of simulation readiness.

M.1.2. ABLATIONS

  • Six ablations test the contributions of critic iterations, generated assets, asset validation, specialized placement tools, visual observations, and agent memory.NotGenerated substitutes HSSD retrieval assets for generated assets; NoSpecializedTools removes snap, facing, stack, fill, arrangement, and pile tools.Source: §M.1.2 · p. 39

M.1.3. PROMPT CORPUS

  • The prompt corpus contains 210 prompts across room-level and house-level scenes, including room-type diversity, high-density object requests, and themed scenes.It includes 94 SceneEval-100 room prompts, 50 Type Diversity prompts, 25 Object Density prompts, 10 Themed Scenes prompts, and 31 House-Level prompts; the SceneEval-100 house subset contributes six prompts to House-Level.Source: §M.1.3 · p. 39
  • Room-level prompts (179) and house-level prompts (31) are evaluated separately because most baselines support only room-level generation.Room-level comparisons appear in Sections N and O; house-level comparisons against Holodeck appear in Tables 1 and 2 of the main paper.Source: §M.1.3 · p. 39

M.2. SceneEval Metrics

  • SceneEval metrics assess object-count satisfaction (CNT), attribute satisfaction (ATR), and object-object relationships (OOR).The measures use VLM-based object matching and attribute evaluation, while OOR checks predefined relationship types (Tam et al., 2025).Source: §M.2 · p. 39
  • SceneEval also measures object-architecture relationships (OAR), accessibility (ACC), navigability (NAV), and out-of-bounds objects (OOB).ACC checks whether functional sides are unobstructed; NAV measures the largest connected free-floor-space share, and OOB identifies objects with less than 99% of sampled rays hitting the floor (Tam et al., 2025).Source: §M.2 · pp. 39–40

M.2.1. PHYSICS METRICS FOR SIMULATION READINESS

  • SceneSmith adds Drake-based collision and stability metrics to evaluate whether generated scenes are ready for robotic simulation.Baselines lack collision geometry and physical properties, so they are postprocessed with V-HACD and uniform-density estimates; SceneSmith scenes need no such postprocessing, and carpets are excluded.Source: §M.2.1 · p. 40
  • COL reports the fraction of objects with collision penetration exceeding 1 mm, using Drake signed-distance queries to assess physical feasibility.Deep penetrations may produce high contact forces that cause objects to explode apart during simulation, while slight penetrations are generally tolerable.Source: §M.2.1 · p. 40
  • STB marks objects stable when, after a 5-second gravity simulation at a 1 ms time step, displacement is below 1 cm and rotation below 0.1 rad.Wall and ceiling objects are welded to the world frame, and support types are classified by SceneEval’s VLM for all methods.Source: §M.2.1 · p. 40

N. Human Evaluation … O. Automated Evaluation

The human evaluation compares scenes on realism and prompt faithfulness and reports room-category win rates; the adjoining automated-evaluation section covers SceneEval benchmark scores and generation costs.

N. Human Evaluation

  • The study used two-sided binomial tests against a 50% win-rate null, Benjamini-Hochberg correction for 28 tests, and was powered to detect win rates of at least 63%.Effect sizes used Cohen’s h; detecting 53–55% win rates would require 6–18 times more comparisons.Source: N. Human Evaluation · p. 40

N.1. Study Setup

  • 205 participants completed 3,051 valid pairwise responses, with each participant judging 15 scene pairs from room- and house-level prompts.210 participants were recruited; each comparison paired SceneSmith with a baseline or ablation.Source: §N.1 · p. 40

N.2. Interface Design

  • The interface offered interactive 3D inspection, alternative camera and image views, and prominent prompts; method names were hidden and scene positions randomized.It also included an interactive tutorial, contextual tooltips, and loading optimizations for varying internet speeds.Source: §N.2 · p. 41

N.3. Questions

  • Participants chose the more realistic scene and the scene that better followed the prompt, with faithfulness guidance covering requested objects, room type, and described details.Faithfulness allowed additional objects that fit the room.Source: §N.3 · p. 41

N.4. Randomization and Counterbalancing

  • Comparison order and question order were randomized, baseline assignments were balanced, and participants saw each prompt at most once.Source: §N.4 · pp. 41–42

N.5. Results by Prompt Category

  • 94.7% realism win rate for Object Density prompts and 94.9% prompt-faithfulness win rate for Themed Scenes were the highest by category; all room-level categories exceeded 89% on both metrics.These room-level win rates average comparisons across external baselines.Source: Table 13 · p. 42, §N.5 · p. 42

O. Automated Evaluation

O.1. SceneEval Benchmark Results … Q. User Study Interface Screenshots

This appendix section documents SceneEval metrics and ablations, baseline controls, generation and evaluation details, and diverse scene-prompt suites. It also presents the user-study interface and question format.

O.1. SceneEval Benchmark Results

  • NotGenerated degrades CNT (75.7% vs 82.9%), OOR (53.7% vs 67.6%), and STB (88.4% vs 95.6%), supporting the contribution of generated assets over a fixed library.NotGenerated keeps SceneSmith’s construction pipeline but replaces generated assets with retrieved HSSD assets; its improvement over HSM and gap to full SceneSmith indicate gains are not solely due to generated asset appearance or vocabulary.Source: §O.1 · pp. 42–43

O.2. Base Model Swap Control

  • SceneSmith remains ahead on most metrics when HSM and Holodeck are rerun with GPT-5.2, with especially large gaps on CNT, ATR, OOR, OAR, COL, and STB.The baseline model swap changes both methods, but not consistently for the better; each rerun retains its original prompts and tools.Source: §O.2 · p. 43, Table 15 · p. 43

O.3. Equilibrium Statistics

  • 12.3 ± 4.5mm mean displacement for SceneSmith contrasts with baseline means of 382–932mm, while NoAgentMemory has 795 ± 1079mm displacement.The displacement figures come from a 5-second physics simulation; NoAgentMemory’s confidence interval exceeds its mean, indicating inconsistent behavior across scenes. Some instability from support-type misclassification is expected to affect all methods equally.Source: §O.3 · p. 43, Table 16 · p. 44

O.4. Collision Depth Statistics

  • 1.2% collision rate and 3.77 ± 1.28mm mean penetration depth among colliding objects characterize SceneSmith’s rare, shallow collisions, versus 10.87–47.05mm for baselines.IDesign has 47.05 ± 25.31mm penetration despite a 3.0% collision rate; NotGenerated has the highest ablation collision rate (7.4%) and deepest penetrations (6.14mm), possibly because retrieved HSSD geometry produces worse V-HACD decompositions.Source: §O.4 · pp. 43, 45, Table 17 · p. 44

O.5. Generation Cost and Time

  • NoCritic cuts cost 70% ($4.24 vs $13.98) and generation time 51% (1:40 vs 3:26), but yields 54.0 vs 71.1 objects on average; NoObserveScene is 31% cheaper and 37% faster.NoAgentMemory is 17% more expensive ($16.41 vs $13.98) and 23% slower (4:14 vs 3:26), attributed to retries of failed approaches without memory. Time was measured with 25 concurrent scenes on 8 L40S GPUs, not as optimized single-scene latency.Source: §O.5 · p. 45, Table 18 · p. 45
  • 75.9% of SceneSmith’s generation cost comes from designer calls, making the refinement loop the main target identified for further cost reduction.The breakdown covers 179 room-level scenes; replacing some designer or critic calls with smaller or open-weight models is proposed as a direction to reduce cost and time.Source: §O.5 · p. 45, Table 19 · p. 45

P.1. SceneEval-100 (Room) (Tam et al., 2025) (94 prompts)

  • The cited prompts depict furnished living rooms, dining rooms, playrooms, bathrooms, basements, and bedrooms, using sofas, tables, chairs, and room-specific accessories in varied arrangements.Examples pair board games with small chairs, bath essentials with a basket, and a bedside clock and decorative candle with nightstands.Source: §P.1 · p. 48

P.2. Type Diversity (50 prompts)

  • The cited prompts depict varied settings—a pet store, private office, gym, robotics lab, and server room—with specified furnishings and equipment, including aquariums, exercise bikes, robotic arms, and server racks.Source: §P.2 · pp. 48–49
  • The 50-prompt type-diversity suite spans settings including a daycare, wine cellar, science lab, Egyptian museum, and escape room.Prompts specify varied furnishings and objects, such as lab equipment, museum artifacts, and wine-cellar seating and bottles.Source: §P.2 · pp. 49–50

P.3. Object Density (25 prompts)

  • The cited passages specify object quantities and arrangements, including a three-level pantry shelf with at least 10 cans, 10 pasta packs, and 10 potatoes, and a table set for 8.Source: §P.3 · p. 50

P.4. Themed Scenes (10 prompts)

  • The 10 themed-scene prompts combine recognizable franchises or artistic styles with coordinated furniture, decor, and room contents.Examples include a Star Wars teen bedroom, an Art Deco living room, a steampunk home office, and a Harry Potter library.Source: §P.4 · p. 50

P.5. SceneEval-100 (House) (Tam et al., 2025) (6 prompts)

  • The six SceneEval-100 house prompts describe furnished bedrooms, bathrooms, and larger connected home spaces.Examples range from a master bedroom with an attached closet and bathroom to a three-room living, dining, and kitchen space.Source: §P.5 · p. 51

P.6. House-Level (25 prompts)

  • The 25 house-level prompts cover varied multiroom settings, from apartments and workplaces to clinics, community centers, and hotels.Layouts specify room functions and furnishings, including exam rooms and labs in clinics, activity rooms in a community center, and guest bedrooms with en-suite bathrooms in a hotel.Source: §P.6 · pp. 51–52

Q. User Study Interface Screenshots

  • The user-study interface offers interactive 3D viewing, static-angle alternatives, tutorial tooltips, and separate realism and faithfulness questions.Participants can orbit, pan, and zoom scenes independently, inspect selected objects, and use click-through or grid views; the faithfulness question includes an “Equal” option with a confirmation dialog.Source: Q. User Study Interface Screenshots · p. 52, Figure 20 · p. 53, Figure 21 · p. 54

R. Additional Qualitative Results … S.1. Representative Failure Modes

SceneSmith’s qualitative results show plausible room connectivity, contextually populated furniture, and themed scenes, while a review identifies asset-level, VLM-reasoning, and hierarchical-construction failures. The reported issues were local imperfections or pipeline scope limits, not catastrophic scene failures.

R.2. Additional SceneSmith Examples

  • Additional examples span diverse room types and complexity levels, including a dental office and boutique hotel suite with multiple rooms.The two house-level examples include reception, exam rooms, an X-ray room, and bathroom in the dental office, and a bedroom, living area, walk-in closet, dressing room, and bathroom in the hotel suite.Source: §R.2 · p. 57, Figure 27 · p. 61

R.3. Manipuland and Themed Generation

  • Furniture receives contextually appropriate, individually manipulable objects, while generated assets and retrieved materials match requested themes in rooms such as an Art Deco living room.Examples include books on a library table and themed rooms such as a Jurassic Park kids bedroom, steampunk home office, and Frozen nursery.Source: §R.3 · p. 57, Figure 32 · p. 66, Figure 33 · p. 67

R.1. Baseline Comparisons

  • SceneSmith connects a hotel’s requested rooms through a central corridor, while Holodeck produces implausible routes through guest rooms or bathrooms.In SceneSmith, the entrance leads through reception, en suite bathrooms are reached through their associated guest rooms, and the kitchen connects to the restaurant.Source: §R.1 · p. 57, Figure 24 · p. 58
  • In room-level comparisons across four prompts, SceneSmith shows appropriate object density, realistic arrangements, and faithful prompt following against six baselines.The prompts include a bunk bed bedroom, Japanese-style living room, grocery store, and nail salon.Source: §R.1 · p. 57, Figure 26 · p. 60

S. Limitations and Failure Analysis

  • Across review of 210 generated evaluation scenes, all outputs were used without discarding or regenerating; reported issues were local imperfections or scope limits, not catastrophic failures.The paper also characterizes SceneSmith as improving over prior baselines in scene density, prompt faithfulness, physical stability, and simulation readiness.Source: S. Limitations and Failure Analysis · p. 68

S.1. Representative Failure Modes

  • Retrieved articulated assets can be semantically close but incorrect, while SAM3D reconstructions may misrepresent transparent, metallic, or highly specular materials.A retrieved storage cabinet can replace a requested mini fridge because the retrieval library limits coverage; the glass may appear opaque and cutlery lack metallic response because SAM3D does not output full PBR channels. The passage suggests TRELLIS.2 (Xiang et al., 2025) as a possible remedy.Source: §S.1 · p. 68
  • VLM reasoning errors can misorient a wall-mounted lamp relative to its surface or place utensils identically on both sides of a table instead of mirroring them.These examples arise during asset canonicalization or placement.Source: §S.1 · p. 68
  • Sequential layout-to-furniture-to-manipuland construction limits later revisions, and furniture may absorb objects intended for the manipuland stage.A TV stand may already contain a TV; support-surface detection can miss a plant stand’s small top, placing the plant on the floor, while a shelf may leave too little space for bottles.Source: §S.1 · p. 68

S.2. System Limitations

SceneSmith’s limitations include runtime and cost trade-offs, bounded articulated-object coverage, estimated rather than measured physical properties, and simulated phenomena outside its current scope; a formal sim-to-real transfer study remains future work.

Runtime and cost

  • The full system prioritizes high-fidelity simulation-ready scenes over single-scene latency, with costs dominated by the Designer and refinement loop.NoCritic offers a lower-cost operating point but produces sparser scenes, exposing a quality-cost trade-off.Source: §S.2 · p. 68

Articulated-object diversity

  • Articulated-object coverage is bounded by the retrieval library; text-to-3D fallback yields static objects, limiting novel or out-of-distribution mechanisms.Integrating maturing image-to-articulated-object reconstruction and related generative methods is identified as a promising way to expand coverage beyond fixed libraries.Source: §S.2 · p. 68

Estimated physical properties

  • Automatically estimated mass, friction, center of mass, and inertia enable broad simulation but cannot replace measured parameters for contact-rich tasks requiring precise dynamics.SceneSmith estimates material categories before mapping them to friction coefficients and represents mass as a range for domain randomization.Source: §S.2 · p. 68

Scope of simulated phenomena

  • Deformable, fluid, breakable, and detailed-material simulations are out of scope, and formal sim-to-real transfer evaluation remains future work.These phenomena require specialized asset representations, while efficient and stable simulation remains challenging.Source: §S.2 · p. 68

Related papers

Loading 2602.09153v2…