PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneBloom: Agentic Indoor Environment Generation with Realistic and Diverse Layouts for Embodied Learning

Yuhan Liu, Lianhui Qin, Shenji Wan

arXiv:2610.32978v1cs.CVcs.AIcs.RO

TL;DR

Realistic, varied, simulation-ready indoor environments are needed for generalizable embodied learning, but translating plausible visual arrangements into valid 3D scenes while preserving their relations is difficult. SceneBloom combines zone-based visual sampling with relation-guided geometric compilation. It reports strong visual quality and physical stability, layouts closer to real rooms, and better navigation success on held-out real scans than equal-count real-scan training.

  • Problem

    Turning visually plausible furniture arrangements into physically valid 3D scenes while preserving functional spatial relationships remains difficult.

    Source: §1 · p. 2

  • Method

    SceneBloom plans functional zones, samples furniture arrangements with diffusion models, and compiles them into 3D using extracted relation contracts to guide fitting and pose correction.

    Source: §1 · p. 2

  • Results

    Across five room types, SceneBloom leads on visual quality and physical stability, has no detected mesh collisions or out-of-bounds objects, and its scenes yield higher held-out real-scan navigation success than equal-count real-scan training.

    Source: §1 · p. 2, §4.1.1 · p. 6

  • Takeaways & Limitations

    SceneBloom’s adjustable scene difficulty can match real homes and further improve navigation transfer, while its realistic layouts support embodied learning.

    Source: §5 · p. 10

Abstract

from arXiv · show

Training generalizable embodied agents requires realistic, varied, and simulation-ready indoor environments. Language-guided methods support semantic planning and object-level reasoning, but precise spatial grounding remains difficult. Visual generative models offer rich priors over plausible arrangements, yet their outputs require geometric reasoning to become physically valid 3D scenes. We introduce SceneBloom, an agentic framework that combines visual layout sampling with contract-guided 3D compilation. A hierarchical sampler uses diffusion priors to generate furniture arrangements within functional zones. A compiler then uses relations extracted from each proposal to guide geometric fitting and local pose correction. Across five room types, SceneBloom attains the highest visual quality and physical stability among the compared methods and is the only method with no detected mesh collisions and out-of-bounds objects. Its layouts are also closer to the distribution of real rooms than those of all compared baselines. Object navigation policies trained in these realistic scenes achieve a higher success rate in held-out real scans than those trained on an equal number of real training scans. Code is available at https://github.com/Tinaliu0123/scenebloom.

1 INTRODUCTION

SceneBloom addresses the challenge of turning plausible visual furniture arrangements into geometrically valid 3D scenes while preserving their functional relationships. It combines hierarchical visual sampling with contract-guided compilation, and reports realistic layouts and improved navigation transfer to held-out real scans.

1 INTRODUCTION

  • The central challenge is translating plausible visual arrangements into geometrically valid 3D scenes without breaking the relationships that make them functional.Errors in depth, scale, or pose can cause overlaps and misalignment, while geometric corrections can turn objects away from their functional partners.Source: §1 · p. 2
  • SceneBloom separates the relationships to preserve from the numerical poses to repair through hierarchical visual sampling and contract-guided 3D compilation.A vision–language planner defines functional zones and furniture inventories; diffusion models propose each zone’s arrangement, and a compiler restores extracted relations while correcting poses.Source: §1 · p. 2
  • Across five room types, SceneBloom scores highest on every visual-quality axis and physical stability, with no detected mesh collisions or out-of-bounds objects.Its layouts are closer to real-room distributions than all compared baselines, and policies trained in its scenes outperform those trained on an equal number of real scans on held-out real scans.Source: §1 · p. 2
  • The contributions include a simulation-ready environment generator, evidence of realistic and physically valid layouts, and navigation-policy transfer to real scans.The paper also reports that controllable scene difficulty strengthens transfer.Source: §1 · p. 2

2 RELATED WORK

Prior scene-generation work combines procedural rules, learned 3D layouts, visual priors, language-guided planning, and agentic refinement. SceneBloom combines functional zoning with visual sampling and relation-preserving compilation to address geometric and functional limits while supporting realistic variation for embodied learning.

3D Indoor Scene Synthesis

  • Procedural, autoregressive, diffusion, and physically guided 3D methods draw on encoded rules or learned 3D scenes, while image-guided approaches add visual priors but may produce layouts incompatible with asset geometry.Correcting those conflicts can disrupt functional relations; SceneBloom uses proposals depicting the actual assets and compiles them under extracted relation contracts.Source: §2 · p. 3

Language-Guided and Agentic Scene Generation

  • Language-guided systems express inventories and constraints, while agentic systems use iterative planning, editing, and review that can slow generation and leave functional inconsistencies.SceneBloom uses agentic planning for zoning, samples each zone’s furniture arrangement in one pass, and uses relation contracts during compilation.Source: §2 · p. 3

Synthetic Environments for Embodied Learning

  • Synthetic environments provide controllable variation for embodied learning, while navigation studies motivate realism alongside dataset size and real-to-sim work retains relevant affordances.These directions motivate environments combining variation with realistic spatial organization; SceneBloom also adjusts scene difficulty for policy learning.Source: §2 · p. 3

3 METHOD

SceneBloom separates visual layout sampling from 3D scene compilation: a zone plan structures the room, visual proposals supply furniture arrangements, and compilation fits them while restoring functional relations. The resulting scene is completed with wall-mounted and surface objects.

3 METHOD

  • SceneBloom first divides a room into functional zones and samples a visual proposal for each zone, then lifts, fits, and refines those proposals before scene completion.The visual prior supplies plausible arrangements while compilation makes them physically valid.Source: §3 · p. 3

3.1 HIERARCHICAL VISUAL SAMPLING

  • The vision–language model plans each functional zone using retrieved exemplar rooms, specifying its region, inventory, anchor, adjoining wall, and prompt.Placement within each zone is left to visual proposals and geometric compilation.Source: §3.1 · p. 4
  • A diffusion model jointly proposes each zone’s furniture arrangement from its prompt and asset-specific adapters, grounding diverse arrangements in the catalog assets used for construction.Each adapter is trained on renders of one catalog asset; varying the diffusion seed produces alternative arrangements of the same assets.Source: §3.1 · p. 4

3.2 CONTRACT-GUIDED 3D COMPILATION

  • Lifting proposals into 3D introduces geometric conflicts and disrupted functional relations, so SceneBloom extracts relation contracts from proposal images before compilation.Contracts record relations among lifted objects, including a relation and confidence, and high-confidence relations guide later steps.Source: §3.2 · pp. 4–5
  • Hierarchical fitting resolves overlaps within zones, then assembles frozen zone envelopes according to the zone plan, enlarging the room only when the envelopes cannot fit.Related objects guide separation within a zone, while room-level assembly moves each fitted zone as a rigid whole.Source: §3.2 · p. 5
  • Relational refinement locally rotates or translates objects to restore violated high-confidence relations, subject to room-boundary and footprint checks.Scene completion then adds wall-mounted and surface objects, checking available space, boundaries, and intersections.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

SceneBloom is evaluated as both a scene generator and a source of training environments for object-goal navigation. Across scene quality, physical validity, real-room alignment, and navigation transfer, the experiments compare it with alternative generators and real-scan training.

4.1 SCENE GENERATION

  • The scene-generation evaluation covers five room types and compares physical plausibility, visual quality, and real-layout alignment against three methods using their official codebases.Metrics include mesh collisions, out-of-bounds objects, stability, GPT-4o scores, and layout-distribution discrepancy against Matterport3D and Scan2CAD.Source: §4.1.1 · p. 6, §4.1 · p. 6

4.1.1 MAIN RESULTS

  • SceneBloom is the only method with no detected mesh collisions or out-of-bounds objects, scores highest on visual-quality axes and stability, and aligns most closely with both real-room references.The scene-generation comparison covers five room types; the visual comparison shows median-scoring scenes for each method and room type.Source: §4.1.1 · pp. 6–7, Table 1 · p. 7, Figure 4 · p. 7
  • Generation takes 2.7 minutes per layout, approximately 27× faster than SAGE on the same hardware; Holodeck is faster, but 78% of its scenes contain collisions.SceneBloom samples each zone’s arrangement once and compiles it without an iterative placement loop.Source: §4.1.1 · p. 7

4.1.2 ABLATION STUDY

  • With visual proposals, contract guidance raises layout coherence from 4.31 to 4.65 and realism from 5.66 to 5.83, and reduces contract violations from 37.45% to 4.25%.Visual proposals outperform textual proposals on every physical and visual-quality metric under both contract settings; for textual proposals, contract guidance improves visual quality but increases collisions and reduces stability.Source: §4.1.2 · pp. 7–8, Figure 5 · p. 8

4.2 OBJECT-GOAL NAVIGATION LEARNING

  • For navigation, the study trains policies from scratch and fine-tunes a pretrained policy using identical recipes across scene sources, then evaluates transfer in held-out real scans.The from-scratch setup uses 40, 80, or 120 houses and 1,000 episodes in 36 real HM3D validation homes; the fine-tuning study uses 120 generated scenes or 142 HM3D scans.Source: §4.2 · p. 8, §4.2.1 · p. 8, §4.2.2 · p. 9

4.2.1 TRAINING FROM SCRATCH

  • 35.3% average success rate from scratch in real HM3D homes is 1.7× the gain from real-scan training; SceneBloom training episodes end at the policy’s STOP in 91.9% of cases versus 75.0% for real scans.The advantage holds at every house count and start distance.Source: Figure 6 · p. 8, §4.2.1 · p. 9

4.2.2 FINE-TUNING A PRETRAINED POLICY

  • On unseen goals, fine-tuning on SceneBloom scenes achieves the highest success rates in held-out generated and real scanned environments; adding occluders raises real-scan success from 48.57% to 49.50%.With occluders, randomized floor objects yield 48.34%, while real-scan continuation yields 47.57% under the same training budget.Source: §4.2.2 · pp. 9–10, Table 2 · p. 9, Table 3 · p. 10
  • The occluder experiment adds an interior zone of tall wardrobes to generated scenes; its randomized control keeps the occluders fixed while resampling other floor objects.Both policies use the same fine-tuning recipe.Source: §4.2.2 · pp. 9–10, Table 3 · p. 10

5 CONCLUSION

SceneBloom separates the relationships a layout should preserve from the poses that need repair. Its realistic layouts and adjustable task difficulty are associated with improved navigation transfer and unseen-goal generalization.

5 CONCLUSION

  • SceneBloom separates layout relationships to preserve from object poses to repair, using visual proposals and contract-guided compilation to produce physically valid, coherent scenes close to real room layouts.Sampling each zone once and resolving each room jointly also makes generation faster than iterative agentic methods.Source: §5 · p. 10
  • Scenes improve real-environment transfer for navigation policies trained from scratch and unseen-goal generalization after fine-tuning.Further gains from matching scene difficulty to real homes support combining realistic layouts with adjustable task difficulty.Source: §5 · p. 10
  • Combining realistic layouts with adjustable task difficulty is valuable for embodied learning.The conclusion reports further gains when scene difficulty matches real homes.Source: §5 · p. 10

A IMPLEMENTATION DETAILS

The implementation compares released baselines using their native assets and procedures, then constructs SceneBloom scenes through exemplar-informed zone proposals, relation-guided compilation, and geometric placement of wall and surface objects.

A.1 BASELINE CONFIGURATIONS

  • Baseline evaluations retain each released system’s native assets, prompts, solver settings, and generation procedures, with runtime measured over executable pipelines.Holodeck and SceneWeaver use Qwen3-VL-8B-Instruct; SAGE retains its specified model pair and tool configuration.Source: §A.1 · p. 15
  • The 192-asset library combines generated and dataset meshes, and is extensible when a category-compatible mesh has a trained LoRA adapter.Baselines retain their own asset pipelines because substituting SceneBloom meshes would evaluate different systems.Source: §A.1 · p. 15

A.2 ZONE-WISE VISUAL PROPOSALS

  • For each zone, the system uses one nearest retrieved exemplar as an anchor and similarity-weighted sampling to select two additional references.Retrieved furniture meshes have asset-specific LoRA adapters; FreeFuse binds up to five category adapters to object tokens and image regions, grounding proposals in the meshes used for construction.Source: §A.2 · p. 15

A.3 CONTRACT-GUIDED SCENE COMPILATION

  • Before geometry fitting, the VLM extracts a zone contract containing only relations shown in the proposal; only relations with confidence γ ≥0.75 guide compilation.The compiler resolves zones around their anchors, places zones rigidly in the room, and applies local rotations or translations to restore violated high-confidence relations.Source: §A.3 · pp. 15–16, Table 4 · p. 16

A.4 WALL AND SURFACE OBJECT PLACEMENT

  • Wall and surface objects are proposed in normalized surface coordinates, converted to room coordinates, and rejected when their geometry intersects existing objects.Wall placements face the wall normal; proposals referring to supporting furniture align horizontally to its center and place the object above its top.Source: §A.4 · p. 16
  • A surface placement is retained only when its oriented footprint lies inside the inset support rectangle and does not overlap an accepted object.Its world pose combines the support surface’s center, extent, yaw, and top height with the VLM’s normalized position and relative yaw, plus a small clearance.Source: §A.4 · p. 16

B EVALUATION PROTOCOLS AND METRIC DEFINITIONS

The evaluation defines a structured measure of layout-distribution alignment against real-room references, alongside geometric stability and judge-rated visual quality protocols.

B.1 LAYOUT DISTRIBUTION DISCREPANCY

  • Layout Distribution Discrepancy (LDD) compares generated layouts with pooled empirical distributions of real rooms using anchor-relative descriptors of object categories and positions.The descriptor encodes category presence and counts, angular and radial organization around functional anchors, and pairwise angular organization; lower LDD is closer, and the unbiased finite-sample estimate may be slightly negative.Source: §B.1 · pp. 16–17
  • LDD uses an unbiased maximum mean discrepancy with an RBF kernel and reference-derived normalization, averaged equally across bedrooms, dining rooms, and living rooms.The descriptor uses categories and positions, not object size, orientation, asset identity, texture, or mesh; multi-scale LDD checks robustness across three bandwidths.Source: §B.1 · pp. 17–18

B.2 PHYSICAL PLAUSIBILITY

  • Physical plausibility is measured with mesh-collision rates, mean out-of-bounds objects per scene, and stable-object rate under a shared dynamic replay protocol.Stability replay uses Isaac Gym; objects count as stable when final translation is at most 0.1 m and rotation at most 0.1 rad.Source: §B.2 · p. 18

B.3 VISUAL QUALITY

  • Visual quality is scored by GPT-4o on realism, functionality, layout coherence, and completeness using top-down renders and object-property lists.The protocol uses temperature zero and averages five judgments per scene, scoring all four methods in one run.Source: §B.3 · p. 18

C ADDITIONAL SCENE GENERATION RESULTS

Additional results report lower motion severity for SceneBloom under dynamic replay and break down generation time and language-model usage.

C.1 PHYSICAL STABILITY

  • SceneBloom has the lowest mean position drift and lowest peak linear and angular velocities under the shared dynamic replay.Table 5 complements the stable-object rate with continuous motion-severity measures.Source: Table 5 · p. 18, §C.1 · p. 18

C.3 COST AND EFFICIENCY

  • Visual proposal generation takes 45.4% of layout time and functional zoning 27.6%, while compilation and wall-surface completion together take under a minute.SceneBloom uses 34,867 language-model tokens and $0.0057 per scene—8 times fewer tokens than SAGE and 4 times fewer than SceneWeaver; only Holodeck uses fewer.Source: §C.3 · p. 19

D.1 RETRIEVAL

Retrieval helps the planner decompose rooms into functional zones, while visual proposals and contract guidance define the compared compilation variants. The section also specifies the controls used to compare training sources.

D.1 RETRIEVAL

  • Without exemplars, plans average 3.14 functional zones versus 3.94 with retrieval; scenes score lower on all four visual-quality axes and lie farther from both real-layout references.The largest reported quality drops are functionality (5.25 versus 6.28) and completeness (5.87 versus 6.76); Scan2CAD LDD rises to .2171 from .1340.Source: §D.1 · pp. 19–20, Table 10 · p. 21

D.2 PROPOSAL SOURCE AND CONTRACT-GUIDED COMPILATION

  • Visual and textual proposals share the same prompt, zoning plan, and asset set; they differ in whether placements come from a diffusion image or VLM text.The contract for textual proposals uses VLM-stated relations and placements; without contract guidance, the compiler separates objects geometrically and skips relational refinement.Source: §D.2 · p. 20

E.1 TRAINING SOURCES

  • The training-source controls include randomized floor-object poses and text-agent layouts, alongside SceneBloom’s visual-proposal, contract-guided pipeline.Randomized retains object content while resampling positions and yaw; Agent-planned replaces placement with a text-only call using the same room, zone, and asset information.Source: §E.1 · pp. 20–21

E.2 NAVIGABLE SPACE IN THE TRAINING SCENES

SceneBloom’s training houses have more connected walkable space than the randomized control, and fragmented scenes contribute a larger share of randomized training episodes.

E.2 NAVIGABLE SPACE IN THE TRAINING SCENES

  • 98.4% of walkable area lies on the largest island in our houses, compared with 86.7% in randomized houses across the 80-house training sets.Across the measured pools, our layouts are the least fragmented and have the highest Walk.Source: §E.2 · p. 21
  • Only 2.0% of our training episodes come from fragmented scenes, versus 41.1% for the randomized control.Episode construction permits fragmented scenes when a finite start-to-goal geodesic exists, keeping each episode on one island.Source: §E.2 · p. 21
  • The median geodesic length is 1.94 m in fragmented randomized episodes, versus 2.27 m in its connected episodes.Source: §E.2 · p. 21

E.3 TRAINING FROM SCRATCH

SceneBloom-trained policies are evaluated on real scans, with results reported for training from scratch and fine-tuning a pretrained policy. Additional analyses examine cleaned evaluation episodes and search difficulty.

E.3 TRAINING FROM SCRATCH

  • STOP rates in our houses are 94.7%, 94.6%, and 88.7% across <2 m, 2–5 m, and 5 m or more, versus 85.4%, 82.1%, and 65.9% in real scans.The rates are grouped by geodesic distance to the nearest goal viewpoint.Source: §E.3 · p. 21

E.4 FINE-TUNING A PRETRAINED POLICY

  • Fine-tuning uses ground-truth target masks for training and YOLOE-predicted masks for evaluation; episode starts are retained when visible goal viewpoints are 1.5–12 m away geodesically.Each fine-tuning run, including real-scan continuation, scales the state encoder’s learning rate by 0.1.Source: §E.4 · p. 22

E.5 FURTHER EVIDENCE ON FINE-TUNING

  • Adding occluders raises the share of episodes hidden after a full turn from 0.8% to 16.4%, and hidden at the start heading from 50.1% to 68.2%.Randomized scenes with occluders are harder to search, yet transfer worse than SceneBloom with occluders on full and cleaned real episodes.Source: §E.5 · p. 23
  • On 631 cleaned HM3D-OVON episodes, SceneBloom’s SR gain over the pretrained policy rises from 0.31 to 0.52 points, and SceneBloom leads in SPL.The occluder zone further raises SR over SceneBloom by 1.48 points and over the pretrained policy by 2.00 points.Source: §E.5 · pp. 22–23, Table 14 · p. 23

F QUALITATIVE RESULTS

The appendix extends qualitative comparisons to offices and bathrooms and documents prompts for zoning, relation extraction, and visual-quality evaluation. The prompts specify both generation outputs and evaluation criteria.

F QUALITATIVE RESULTS

G PROMPT TEMPLATES

  • The prompt appendix covers functional zoning, relation-contract extraction, and the visual-quality judge, which is used only for evaluation.Source: G PROMPT TEMPLATES · p. 24

G.1 FUNCTIONAL ZONING

  • The zoning prompt asks for non-overlapping axis-aligned rectangles that tile the floor, with room dimensions, zone categories, counts, and placement prompts.Retrieved floor plans serve as spatial-layout references; the plan is returned as JSON.Source: §G.1 · p. 24

G.2 RELATION-CONTRACT EXTRACTION

  • Relation extraction records clearly visible qualitative relations from the original zone image, without coordinates or numeric yaw.It uses five specified relations and returns up to ten constraints with confidence and rationale.Source: §G.2 · p. 25

G.3 VISUAL-QUALITY EVALUATION

  • The visual-quality prompt scores realism, functionality, layout coherence, and completeness independently on a calibrated 0–10 scale.The instructions require substantial score reductions for critical issues such as missing key objects, obvious layout errors, unrealistic elements, or clearly wrong size.Source: §G.3 · p. 25

Related papers

Loading 2610.32978v1…