PaperlayerView exact revision on arXiv

Source-linked AI summary

ScenePilot: Grow-and-Repair Policy for Text-Driven 3D Indoor Scene Generation

Jiawei Zhang, Hongsong Wang, Pan Zhou

arXiv:2608.30307v1cs.CVcs.AI

TL;DR

Text-driven indoor scene generation must maintain physically plausible, functionally coherent layouts despite path-dependent intermediate errors and limited reusable priors. ScenePilot combines hierarchical retrieval, sequential functional-group growth, and multimodal move–rotate–scale repair trained on SceneReverse-17k. It reports improved physical validity and visual-semantic quality, while retrieval quality and reward scope remain important boundaries.

  • Problem

    Text-driven indoor scene generation must satisfy physical plausibility, structural coherence, and functional utility while correcting path-dependent intermediate errors.

  • Method

    ScenePilot retrieves hierarchical spatial priors, grows scenes through sequential functional groups, and applies local plus final global repair using learned move–rotate–scale actions.

  • Results

    ScenePilot achieves the best overall full-scene performance across physical validity and visual-semantic quality, reducing PBL from 186.2 to 75.4 versus ReSpace and improving VR from 0.73 to 0.86.

  • Takeaways & Limitations

    The grow-and-repair perspective supports more physically plausible, functionally coherent, and controllable 3D indoor scene synthesis.

  • Takeaways & Limitations

    Performance depends on the coverage and quality of indexed layout priors, while richer aesthetic or style feedback may increase evaluation cost.

Abstract

from arXiv · show

Text-driven 3D indoor scene generation has advanced from dataset-bound layout modeling to open-vocabulary synthesis with large language and vision-language models. Yet existing methods remain limited: one-pass generators often yield geometrically invalid layouts, heavy post-hoc optimization is costly and unstable, and prompt-only planners lack reusable layout priors for functional grouping and object relations. We propose \textbf{ScenePilot}, a retrieval-augmented \textbf{Grow-and-Repair} framework that formulates scene generation as prior-guided incremental growth with learned rectification. Given a prompt, the Hierarchical Retrieval-Augmented Planning (HRAP) module retrieves room-, group-, and anchor-level layout priors to support functional group planning. A text-driven base generator then inserts object groups sequentially, while the Reinforcement Multimodal Repair (RMR) module performs lightweight local correction after each insertion and a final global repair after completion. To train this policy, we construct \textbf{SceneReverse-17k}, a repair-trajectory dataset built by perturbing high-quality 3D scenes in position, rotation, and scale, then using inverse operations as executable rectification targets. The policy predicts structured \emph{move--rotate--scale} actions from rendered views, scene state, retrieved priors, and edit history. By combining HRAP with RMR, ScenePilot offers an efficient alternative to one-shot generation and heavy full-scene optimization, improving physical plausibility, functional coherence, and controllability while preserving diversity.

1 Introduction

ScenePilot frames indoor scene generation as process-aware growth: it retrieves spatial priors, inserts functional groups sequentially, and repairs intermediate layouts before final global coordination.

  • Motivation: Indoor scene generation must jointly satisfy physical plausibility, structural coherence, and functional utility.These requirements include grounded, non-overlapping objects within room boundaries, coherent organization, and accessible furniture.
  • Motivation: Existing methods largely treat generation as final-layout prediction or post-hoc optimization, overlooking path-dependent intermediate states.Early collisions, misplaced anchors, or incomplete functional zones can bias later placements and produce failures that final refinement may not recover.
  • Method: ScenePilot retrieves room-, group-, and anchor-level priors to enrich prompts and plan coherent functional groups rather than isolated furniture.The enriched prompt is decomposed into groups that a base text-driven generator inserts sequentially.
  • Method: Its Reinforcement Multimodal Repair policy observes each intermediate scene and predicts executable move–rotate–scale edits instead of regenerating the whole layout.The policy uses rendered views, structured scene state, retrieved context, and action history for targeted correction.
  • Training: SceneReverse-17k converts perturbed high-quality scenes into repair trajectories with inverse perturbations as executable rectification targets.Perturbations alter positions, rotations, and scales, providing supervision for moving from invalid intermediate states toward improved scenes.

2 Related Work

Prior work advances language-guided planning, retrieval, optimization, and sequential decision-making, while ScenePilot distinguishes itself by learning executable repair over intermediate states during group-wise growth.

  • 3D Indoor Scene Generation: Indoor scene generation has progressed from dataset-driven layout modeling to language-guided open-vocabulary synthesis.Earlier approaches use convolutional, autoregressive, transformer, or diffusion-based layout models, while language-based methods improve semantic flexibility.
  • Retrieval-Augmented Layout Priors: Retrieval is commonly used to obtain assets, examples, relation rules, or layout templates before scene planning.ScenePilot instead uses hierarchical spatial priors to guide functional grouping and anchor selection.
  • Scene Optimization and Repair: Optimization and physically grounded guidance improve plausibility but can remain initialization-sensitive or rely on explicit optimization.Representative methods enforce relational, collision, room-layout, or reachability constraints.
  • Process Supervision and Policy Learning: Recent methods treat layout generation as sequential decision-making through structured rewards, reflective planning, tool use, or preference optimization.ScenePilot differs by centering supervision on repair trajectories and deploying rectification within group-wise generation.
  • Process Supervision and Policy Learning: ScenePilot centers training and inference on intermediate scene states rather than only improving the final generated layout.It learns executable repair actions that operate during group-wise scene growth.

3 ScenePilot Method

ScenePilot formulates indoor scene generation as sequential growth through functional groups, guided by retrieved spatial priors and stabilized by executable repair actions. Its process-oriented training uses perturbed 3D scenes and reverse trajectories to supervise correction of invalid intermediate states.

  • Process-aware scene growth: ScenePilot constructs scenes through sequential functional-group insertion, making intermediate states explicit targets for evaluation and correction.Each group represents a functional zone organized around a dominant anchor object.
  • Reinforcement Multimodal Repair: RMR restricts repair to executable move, rotate, and scale actions, leaving object addition and deletion to planning and generation.This repair-only design supports targeted, checkable edits focused on geometric and functional errors.
  • Hierarchical Retrieval-Augmented Planning: HRAP retrieves room-, group-, and anchor-level priors to provide layout tendencies, functional compositions, and local object-relation hints.The priors help the planner reason about coherent groups rather than isolated object instances.
  • Grow-and-Repair inference: The repair procedure combines policy proposals with lightweight local search over translations, rotations, wall alignment, and anchor-aware attachment moves.Candidate edits are evaluated under a quality score covering physical violations, relation consistency, and functional usability.
  • Process-oriented training: SceneReverse-17k trains repair from corrupted intermediate layouts by converting perturbation trajectories into supervised and policy-optimization samples.The dataset is derived from high-quality 3D-FRONT scenes and includes multimodal observations with target repair sequences.
  • Process-oriented training: Process-oriented supervision teaches action on invalid intermediate states, executable object-level correction, and history-aware decision making.This matches the policy’s role during incremental scene construction rather than static final-scene prediction.

4 Experiments

ScenePilot is evaluated against text-driven 3D scene-generation baselines through quantitative, human, qualitative, and ablation studies. Results show improved physical validity, semantic-functional quality, perceived layout quality, and component-level effectiveness.

  • Experimental Setup: The evaluation covers 100 scenes across five room types, using both 75 detailed prompts and 25 short room-type prompts.Scenes are assessed with physical-validity metrics and VLM-judge scores, while human evaluation uses 50 valid responses.
  • Experimental Setup: ScenePilot is compared with ReSpace, Reason-3D, and a post-hoc fine-tuned Qwen3-VL repair baseline under the same asset library and evaluation protocol.ReSpace is the primary structured-generation baseline, while Reason-3D provides a reasoning-based comparison.
  • Main Results: ScenePilot reduces PBL from 186.2 to 75.4 versus ReSpace, improves VR from 0.73 to 0.86, and raises the average VLM-judge score from 6.8 to 8.1 versus Reason-3D.It also reduces PBL by 38.4% relative to the post-hoc fine-tuned Qwen3-VL repair baseline.
  • Main Results: ScenePilot receives the highest human scores across LC, SPA, and FC, with an average score of 7.90 and improvements of 0.80 and 1.00 over Reason-3D and ReSpace.The human ranking is broadly consistent with the VLM-judge ranking.
  • Main Results: Qualitative comparisons show more organized layouts, clearer functional grouping, better object arrangement, and more usable free space than the baselines.The reported examples include a more coherent bedroom sleeping zone and less scattered or congested dining-room and living-room arrangements.
  • Ablation Studies: Removing HRAP increases PBL from 75.4 to 135.5, while the HRAP-only variant reaches PBL 147.7 versus 75.4 for the full model, showing that retrieval and repair-in-the-loop generation both contribute.Removing group-wise insertion also degrades physical and semantic-functional metrics; the expert GPT-5.2 reference reaches PBL 53.5 and Avg 8.2.

5 Conclusion

The framework integrates retrieved layout priors with learned rectification during scene growth, offering an alternative to one-pass generation and heavy post-hoc optimization. Its remaining limitations concern aesthetic feedback and the coverage and quality of the retrieval memory.

  • ScenePilot combines retrieved room- and anchor-level priors, reverse repair trajectories, multimodal move–rotate–scale repair, and group-wise local plus global repair.
  • The growth-with-repair perspective supports more physically plausible, functionally coherent, and controllable 3D indoor scene synthesis.
  • Limitations and Future Work: Richer aesthetic or style-level feedback could improve visual quality but may increase evaluation cost.
  • Limitations and Future Work: Weak retrieval may introduce irrelevant relations or bias scenes toward common arrangements because the RAG memory depends on indexed-prior coverage and quality.

A Details of the Experiment Setup

The repair policy is trained from multi-view renders and serialized scene JSON to predict structured repair operations, using supervised fine-tuning followed by GRPO optimization. Experiments use distributed training with specified batch, optimization, and hardware settings.

  • Each training sample pairs multi-view rendered images and serialized scene JSON with a structured SceneRepairPlan specifying scene repair operations.
  • Supervised fine-tuning: Supervised fine-tuning runs for 2 epochs with a global batch size of 32, learning rate 1 × 10−4, weight decay 0.01, cosine decay, and 0.03 warmup.
  • GRPO fine-tuning: GRPO uses 2 sampled generations per prompt, learning rate 5×10−6, weight decay 0.1, cosine scheduling, and a trainable vision tower with a frozen language backbone.
  • Implementation details: Experiments run on Linux servers with NVIDIA RTX 4090 D and RTX A6000 GPUs under CUDA 12.4.

B Details of ScenePilot

The framework uses a JSON-based scene representation that stores room geometry and object-level attributes, including textual descriptions, dimensions, positions, rotations, and asset identifiers.

  • The JSON scene file stores room geometry through top and bottom boundary vertices and maintains an object list with textual descriptions, sizes, positions, rotations, and asset identifiers.

B.1 Scene Representation

ScenePilot represents rooms and objects explicitly, mines anchor-centered functional groups and spatial relations from clean scenes, and retrieves these priors to guide sequential group insertion and repair.

  • Scene Representation: Room geometry is represented by top and bottom boundary polygons, while room_type records the functional category of the space.
  • Scene Representation: Each object stores a description, size, position, rotation, and asset identifier, enabling layout generation, validation, and object-level editing.
  • Scene Representation: The explicit room-bound and object-attribute decomposition supports repeated updating, evaluation, and refinement in the grow-and-repair pipeline.
  • Retrieval-Augmented Spatial Prior Construction: The spatial prior memory stores reusable anchor-centered knowledge about functional-group membership and typical arrangements around dominant anchors.
  • Scene-level group mining: Scenes are parsed by room type, normalized object categories, and valid anchor categories before functional groups are mined.
  • Scene-level group mining: Nonanchor objects are assigned to compatible nearby anchors using category, distance, surface-gap, support, and companion-prior cues.
  • Scene-level group mining: Example groups include bedroom anchors with nightstands, lamps, rugs, and dressers, and dining-table anchors with multiple chairs.
  • Anchor-member relation statistics: Anchor-member statistics encode local offsets, distances, angles, yaw differences, surface gaps, support ratios, and common member counts.

B.3 Grow-and-Repair Inference Algorithm

ScenePilot grows scenes by sequentially inserting functional groups, repairing intermediate states, and applying a final global repair. Its visual judge evaluates layout correctness, semantic plausibility, and functional completeness from top and diagonal renders.

  • Grow-and-Repair Inference Algorithm: The inference procedure retrieves spatial priors, decomposes prompts into functional groups, inserts groups sequentially, and repairs each intermediate state before proceeding.A final global repair is applied after all groups are committed.
  • Evaluation Criteria: The visual evaluation scores layout correctness, semantic plausibility, and functional completeness as three separate criteria.The criteria cover geometric arrangement, believable object relations, and whether the room contains furniture needed for its intended function.
  • Visual Observation: The repair policy uses rendered top-down and diagonal views as complementary evidence for judging spatial arrangement, realism, scale, and accessibility.Top views emphasize global arrangement, while diagonal views support judgments about perspective and object realism.
  • Layout Correctness: Layout correctness penalizes collisions, boundary violations, blocked circulation, awkward spacing, misalignment, implausible scale, and unusable placement.Scores range from 0 to 10, with serious collisions or blocked walkways receiving low scores.
  • Semantic Plausibility: Semantic plausibility measures whether furniture pairings, relations, scales, orientations, and usage patterns form a believable room rather than a random object collection.Examples include seating around tables, bedside furniture near beds, desk-chair pairings, and TVs facing seating.
  • Functional Completeness: Functional completeness measures whether essential furniture, main activities, and functional zones are present and sufficiently specified.A scene can be collision-free yet receive low functional completeness when key functions or furniture are missing.

D Human–VLM Agreement Analysis

Human evaluation and the VLM judge agree strongly on layout correctness and semantic plausibility, while functional-completeness agreement is weaker. Human judgments anonymized the methods to reduce name-induced bias.

  • Evaluation Protocol: Human evaluation anonymized the three methods as A, B, and C so participants judged rendered results without method-name expectations.Scores were averaged across evaluated samples.
  • Agreement Results: Human and VLM-judge scores are highly consistent for layout correctness and semantic plausibility.The agreement is reported at the method level across evaluated methods.
  • Agreement Results: Functional-completeness agreement is weaker because human scores for Reason-3D and ReSpace are close while the VLM judge assigns a larger gap.The passage characterizes functional completeness as a more subjective criterion.

E.2 Prior Memory Statistics

The RAG prior memory organizes mined anchor-centered statistics into reusable prompt-ready documents and retrieves them as soft planning guidance. Its coverage is strongest for common room types and weaker for rare ones such as laundry rooms.

  • Document Organization: The memory contains overview, signature, and member-prior document types describing companion patterns, group compositions, and member placement around anchors.Documents include global and room-specific priors.
  • Room Coverage: Bedroom and living-room scenes contribute most anchor-centered groups, whereas laundry scenes are much rarer.Laundry prompts therefore rely more on global anchor priors and the base generator than on dense room-specific priors.
  • Retrieval Setup: HRAP retrieves the top-5 relevant prior documents using the user request, inferred room type, and candidate anchors, with optional filtering.Retrieved priors provide soft planning guidance rather than hard coordinate constraints or direct scene copies.

E.5 Data Leakage and Limitations

The prior memory aggregates anchor-centered statistics and natural-language summaries rather than raw scenes, but evaluation records must still be excluded before mining and indexing. Data imbalance and noisy retrieval further limit benefits for rare or ambiguous cases.

  • Data Leakage: Evaluation scene IDs should be excluded before prior mining, document construction, and FAISS indexing to protect the benchmark protocol.The memory stores aggregated statistics and summaries rather than raw scene JSONs, rendered images, or complete layouts.
  • Coverage Limitations: Sparse room-specific priors for rare room types can reduce retrieval benefits, while weakly related results may introduce noisy planning hints.The limitation is especially relevant to ambiguous prompts and rare object combinations.

F More results

Figures 7 and 8 provide additional qualitative bedroom-generation results, including a modern bedroom prompt with specified furniture and lighting elements.

  • Additional qualitative bedroom-generation results are presented in Figures 7 and 8.
  • Figure 7 shows a modern bedroom prompt specifying a single bed, two nightstands, a drawer chest, a TV stand, a floor lamp, and a pendant lamp.
  • Figure 8 provides additional qualitative generation results.
Loading 2608.30307v1…