PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent

Yandan Yang, Baoxiong Jia, Shujie Zhang, Siyuan Huang

arXiv:2509.20414v2cs.GRcs.CVcs.LGcs.RO

TL;DR

Existing 3D scene synthesis methods often struggle to provide realistic, physically plausible, detailed, and instruction-aligned environments for Embodied AI. SceneWeaver combines extensible synthesis tools with a self-reflective reason-act-reflect loop, and experiments report stronger performance across common and open-vocabulary scenes.

  • Problem

    Existing methods struggle to jointly achieve visual realism, physical plausibility, scene diversity, fine-grained detail, and precise instruction following.

  • Method

    SceneWeaver uses standardized modular tool interfaces, a self-reflective planner, and physics-aware execution for iterative feedback-driven scene refinement.

  • Results

    SCENEWEAVER outperforms existing methods across open-vocabulary room types, achieving an average object count of 36.5 and zero collisions and out-of-boundary violations.

  • Takeaways & Limitations

    SCENEWEAVER supports scene synthesis across common and open-vocabulary settings while combining visual realism, physical plausibility, and instruction following.

Abstract

from arXiv · show

Indoor scene synthesis has become increasingly important with the rise of Embodied AI, which requires 3D environments that are not only visually realistic but also physically plausible and functionally diverse. While recent approaches have advanced visual fidelity, they often remain constrained to fixed scene categories, lack sufficient object-level detail and physical consistency, and struggle to align with complex user instructions. In this work, we present SceneWeaver, a reflective agentic framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement. At its core, SceneWeaver employs a language model-based planner to select from a suite of extensible scene generation tools, ranging from data-driven generative models to visual- and LLM-based methods, guided by self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. This closed-loop reason-act-reflect design enables the agent to identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations. Extensive experiments on both common and open-vocabulary room types demonstrate that SceneWeaver not only outperforms prior methods on physical, visual, and semantic metrics, but also generalizes effectively to complex scenes with diverse instructions, marking a step toward general-purpose 3D environment generation. Project website: https://scene-weaver.github.io/.

1 Introduction

Embodied AI expands 3D scene synthesis beyond visual realism toward physical interaction and precise instruction control, exposing limitations in existing methods. SCENEWEAVER addresses these gaps with extensible tools and reflective, iterative planning.

  • Embodied AI requires scenes that are visually realistic, physically interactable, and controllable through task-specific instructions.
  • Rule-based, data-driven, and LLM-based methods each address only subsets of realism, physical plausibility, scene diversity, or controllability.
  • SCENEWEAVER uses modular tools and feedback-driven planning to iteratively refine scenes across diverse synthesis paradigms.
  • SCENEWEAVER reports state-of-the-art performance across common and open-vocabulary scenes, including visual realism, physical plausibility, and instruction following.
  • The framework combines a reason-act-reflect paradigm with standardized interfaces for synergizing scene synthesis methods according to their strengths.

2 Related work

Related work spans data-driven indoor layout synthesis, MLLM-based spatial reasoning, and general-purpose LLM agents. These approaches motivate combining structured constraints, modular tools, and agentic planning for 3D scene synthesis.

  • 3D Indoor Scene Synthesis: Data-driven models learn realistic but coarse layouts from datasets such as 3D-FRONT, whose limited variety constrains scene diversity and detail.
  • 3D Indoor Scene Synthesis: Language and 2D foundation models supplement missing scene-type priors and fine-grained details in indoor scene synthesis.
  • Spatial Reasoning of MLLMs: MLLM-based methods incorporate structured constraints, post-processing, or role-based decomposition to improve spatial reasoning and physical plausibility.
  • LLM-based Agentic Framework: General-purpose LLM agents coordinate multiple specialized tools collaboratively, with extensibility and planning linked to performance gains in complex reasoning tasks.
  • LLM-based Agentic Framework: These agent-development insights remain largely underexplored for 3D tasks.

3 The SCENEWEAVER Framework

SCENEWEAVER organizes diverse scene synthesis methods as modular tools and uses a self-reflective planner to select and refine them iteratively. A physics-aware executor turns layout changes into concrete, evaluable 3D scenes.

  • Framework Overview: The framework comprises a standardized tool interface and a self-reflective planner that performs feedback-guided refinement and physics-based optimization.
  • Problem Setup and Reflection: Each scene state includes 3D layout information and 2D renderings, while reflections provide quantitative scores and explanations for the previous state.
  • Tool Catalog: Tool interfaces abstract scene synthesis methods into modular tools categorized by synthesis granularity.
  • Standardized Tool Interface: Standardized tool cards specify descriptions, scenarios, constraints, inputs, examples, strengths, and supported room types to guide tool selection.
  • Self-reflective Planning: The planner summarizes memory, identifies the most critical problem, ranks candidate tools, and selects a refinement action using context-aware function calling.
  • Physics-aware Execution: The physics-aware executor loads prior scenes, applies layout modifications, retrieves concrete assets, and supports physical post-optimization and evaluation.

4 Experiment

The experiments evaluate SCENEWEAVER on common and open-vocabulary rooms using physical, visual, and semantic metrics, alongside ablations, qualitative comparisons, iterative-refinement examples, and human judgments. Across these evaluations, SCENEWEAVER generally outperforms existing methods while maintaining physical validity and benefiting from reflective planning and diverse tools.

  • Evaluation Setup: The evaluation covers common room types and open-vocabulary generation, using physical, visual, and semantic metrics against data-driven and open-vocabulary baselines.Common-room results average over living-room and bedroom scenes, while open-vocabulary results average across eight room types.
  • Common Room Types: SCENEWEAVER achieves state-of-the-art performance across most common-room metrics and zero physical errors, while Holodeck reaches a higher object count of #Obj=32.2.The reported physical errors include collisions and out-of-boundary violations.
  • Open-Vocabulary Generation: Across all eight tested open-vocabulary room types, SCENEWEAVER obtains an average object count of 36.5, better visual and semantic scores, and zero collisions and out-of-boundary violations.The passage attributes these improvements to the reflective planner’s refinement of semantic coherence, diversity, and physical plausibility.
  • Qualitative Results: Qualitative comparisons show improved visual realism and finer-grained detail for SCENEWEAVER across common and open-vocabulary room types.Figure 4 compares SCENEWEAVER with existing methods.
  • Agent Design: Ablations show that removing reflection reduces semantic quality, while single-shot multi-step planning can produce globally inconsistent or locally infeasible layouts.The agent-design study also evaluates removing physical optimization and replacing iterative reflection with a single-shot plan.
  • Tool Analysis: The full combination of initializer, implementer, and modifier tools yields the highest performance, with implementers adding detail and modifiers improving functional alignment and layout coherence.Modifier tools can reduce object count and completeness by removing redundant items; one comparison reports 16.3 vs. 23.0 objects and 5.0 vs. 5.7 completeness.
  • Iterative Refinement: With complex instructions, SCENEWEAVER progressively refines object counts, small-object placement, and overall layout to better follow user requirements.Figure 5 provides qualitative examples of this iterative refinement process.
  • Human Evaluation: In a study with twenty participants, SCENEWEAVER consistently outperforms baselines across evaluation dimensions and is preferred in nearly 85% of pairwise comparisons.The pairwise study evaluates three scenes for each room type and also checks diversity.

5 Conclusion

SCENEWEAVER is presented as a reflective, extensible framework that combines standardized tool interfaces with iterative feedback-driven refinement. Its scene representation joins rendered visual information with structured layout data to support spatial reasoning and evaluation.

  • Conclusion: SCENEWEAVER integrates diverse scene-synthesis paradigms through standardized tool interfaces and iterative feedback-driven refinement.The framework adopts a reason–act–reflect paradigm in which a planner selects tools using multimodal self-evaluation.
  • Conclusion: The framework targets high-quality, functionally coherent, and semantically faithful scenes across common and open-vocabulary settings.The conclusion reports validation through extensive experiments and human evaluations.
  • Scene Representation: Each scene state combines a top-down rendered image with coordinate annotations, object bounding boxes, labels, directions, and structured object-layout information.The representation is designed to convey both visual and logical information for the agent.

A.3 Implementation of the executor

The executor adapts Infinigen into an interactive backend for SCENEWEAVER’s modular tools. It supports real-time Blender interaction, physical optimization, and scene annotations, while the planner selects and runs one tool per step.

  • Executor Modifications: The executor is based on Infinigen and modified to support SCENEWEAVER’s agentic framework.The listed modifications include real-time Blender interaction and fitting the executor to individual tools rather than procedural scene generation.
  • Executor Modifications: Physical optimization is applied to avoid collisions and enhance object relations after scene generation.The executor also adds 3D marks and a 2D top-down rendering.
  • Tool Execution: During planning, the agent uses function calling to choose a single tool per step, runs it through the executor, and updates the scene.The tools are described to the agent before selection.
  • Simulation: The executor is demonstrated in robot-interaction simulations across kitchen, meeting-room, and restaurant scenes while preserving distinct material properties.The meeting-room example includes a transparent and reflective table.
  • Spatial Relations: Scene relations encode object-to-room constraints such as against-wall and on-floor, plus object-to-object relations such as front-against.These relation types specify spatial orientation and proximity between entities.

A.6 How to choose assets dataset

SCENEWEAVER selects asset sources according to tool needs, combining dataset assets, model-associated assets, procedural generation, and open-vocabulary retrieval. The iterative procedure can terminate early when reflection judges the scene satisfactory.

  • Asset Selection: Asset selection depends on the usage and requirements of each scene-generation tool.The framework permits choosing among available asset sources.
  • Asset Sources: MetaScenes assets are used directly for dataset-based tools because scenes contain detailed meshes and layout information.The assets are selected with their associated scene structure.
  • Asset Sources: 3D FUTURE assets support models such as PhyScene, DiffuScene, and ATISS because those models are trained on that dataset.This aligns model-based tools with their training asset source.
  • Asset Sources: Infinigen generates standard assets procedurally for common categories, while Objaverse supplies unsupported open-vocabulary categories such as clocks, laptops, and washing machines.The asset source therefore varies with category coverage.
  • Stopping Criterion: The planner allows up to 10 refinement steps but can stop earlier when intermediate results satisfy user requirements with a high score.The reflection module decides whether optimization should continue or stop.
  • Asset Retrieval: For ambiguous asset poses, the system estimates side candidates from minimum bounding rectangles and prompts GPT to identify the front direction.The authors note that GPT performs well on common objects but may fail on more complex or ambiguous cases.

B.2 Additional Scene Generation Results

Additional results show SCENEWEAVER generating detailed, organized scenes across diverse room types and supporting physically interactive simulation. The scenes preserve visual and geometric detail while satisfying collision and boundary constraints.

  • Additional visualizations: SCENEWEAVER organizes many objects coherently in restaurants, garages, and gyms, including detailed arrangements for complex user queries.Examples include bathroom cabinets containing paper rolls and shelves holding baskets or towels.
  • Simulation and interaction: Generated scenes can be exported as USD files and loaded into Isaac Sim for physical simulation and interactive tasks.A remotely controlled Unitree G1 humanoid robot performs object interactions in the virtual environments.
  • Simulation and interaction: The pipeline preserves textures and geometric details, constrains object placement physically, and adapts layouts to diverse embodied-AI requirements.These properties are presented as advantages for embodied-AI applications.
  • Simulation and interaction: SCENEWEAVER is presented as supporting sim-to-real transfer for robotic manipulation while maintaining visual and functional realism.This is the authors’ stated implication of combining high-fidelity simulation, physical interaction, and task-aligned layouts.

B.4 User Study

The user study evaluates visual and semantic qualities through individual ratings and pairwise comparisons, while simulation tests assess object stability. Participants preferred SCENEWEAVER and judged it more diverse than baseline methods, with strong agreement between human and LLM rankings.

  • Human evaluation: The pairwise study compared three SCENEWEAVER scenes with three baseline scenes under identical prompts and asked about preference and diversity.Five scene sets were selected for each baseline method.
  • Human evaluation: SCENEWEAVER received higher average preference and diversity votes than the baseline methods.The reported results state that it surpassed the alternatives by a large degree.
  • Human-LLM alignment: Human and LLM evaluation scores can differ slightly, but user-user and user-LLM rankings show strong agreement across metrics.The reported user-LLM alignment is higher than user-user alignment, indicating more stable LLM evaluation in this analysis.
  • Simulation stability: SCENEWEAVER’s generated objects remain the most stable in simulation despite having the largest object count.Stability is assessed using shift thresholds and average displacement in Isaac Sim.

B.7 Metric improvement during iteration.

SCENEWEAVER improves scenes through self-adaptive, feedback-driven iterations that target the lowest current metric. The framework combines extensible tools with physical optimization, while supporting baseline refinement and broader scene configurations.

  • Metric improvement during iteration: At each iteration, the agent identifies the lowest-scoring metric and selects a tool to address that specific problem.The process is self-adaptive rather than a fixed coarse-to-fine placement sequence.
  • Metric improvement during iteration: Completion improved from 4 to 6 after adding shelf objects, while layout improved from 5 to 6 after removing an illogical bathroom sink.Later refinement rearranged dining tables to create movement space and clear walkways, raising layout to 8.
  • Metric improvement during iteration: The agent can repair errors introduced by earlier steps and reuse the same tool repeatedly until the scene satisfies the demand.Other metrics, including realism and functionality, may also improve while a different metric is optimized.
  • Tool effects: Adding Initializer, Refiner, and Implementor tools greatly improves results, while specialized tools target rotation validity and densely populated supports.The Update Rotation tool improves sensitivity to invalid rotations, and Add Crowd improves realism by filling shelves with child objects.
  • Room structure: The main paper simplifies generation by removing windows and doors, although SCENEWEAVER can handle and update scenes containing them.Examples with these room structures are shown separately.
  • Room scale: The experiments focus on single rooms to improve scene quality, while the process can be repeated for multiple rooms or extended using coordinate transformation.The stated room-scale focus is a scope choice rather than a claimed inability to generate multi-room scenes.
  • Baseline refinement: SCENEWEAVER can refine scenes produced by PhyScene and LayoutGPT by treating each baseline as an initializer tool.The same extensible tool-card design can incorporate these baseline methods.
  • Computation and scope: Physical optimization increases runtime because it enforces geometric-level plausibility in addition to multi-step refinement and tool-specific processing.The authors contrast this with prior work operating only at the bounding-box level.

C.5 API Cost

SCENEWEAVER averages about seven iterations and costs approximately $0.5 per generated scene. Generation time and cost can be reduced by simplifying scenes, using fewer costly tools, or reducing iterations, with limited reported quality loss after early iterations.

  • API cost: The average number of iterations is about 7, and the average API cost is about 0.5 dollar per scene.The authors attribute the cost to the agentic framework used to achieve physical, visual, and semantic results.
  • Cost reduction: Runtime and cost can be reduced by simplifying requirements, reducing physical optimization, removing or replacing time-consuming tools, and using fewer iterations.The authors note that these choices may influence results, but expect limited quality loss from some changes.
  • Cost-quality trade-off: Reducing iteration count has limited effect on final quality because scenes achieve high scores in the first few iterations.Later iterations mainly address corner cases and make smaller improvements.
  • Cost-quality trade-off: With only three steps, bedroom GPT scores were 8.6, 9.3, 7.5, and 7, remaining better than baseline methods.The values illustrate the reported quality retained under an abbreviated process.

D Limitation

The paper identifies limitations in both open-vocabulary asset handling and LLM-based evaluation. Open-vocabulary assets can produce unrealistic scale, placement, or orientation, while LLMs remain imperfect proxies for human judgment.

  • D.1 Unrealistic Scaling & Placement for Functional Items: Open-vocabulary assets may lack standard sizes and front directions, causing unrealistic object scale, placement, and orientation.The paper cites examples such as toys with arbitrary dimensions and elliptical bikes whose front direction is ambiguous.
  • D.1 Unrealistic Scaling & Placement for Functional Items: Balancing asset standardization with generality remains an unsolved problem for open-vocabulary scene synthesis.The paper nevertheless describes open-vocabulary assets as a feasible and promising direction for general scene generation.
  • D.2 Validity of LLM’s Judgement: Using LLM judgments for both generation guidance and validation is treated as acceptable, but remains constrained by evaluator reliability.The authors contrast this practical choice with the expense of scaling human studies and the lack of a clearly superior accessible alternative.
  • D.2 Validity of LLM’s Judgement: LLMs are imperfect proxies for human judgment when used to evaluate generated scenes.The paper presents LLM feedback as a practical interim solution because more reliable, scalable, and human-aligned evaluators are not yet available.
Loading 2509.20414v2…