Source-linked AI summary
Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
Lu Ling, Chen-Hsuan Lin, Tsung-Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming-Yu Liu, Aniket Bera, Zhaoshuo Li
TL;DR
Existing scene-generation methods are constrained by limited indoor datasets, while LLM-based planning lacks visual perception for realistic spatial relations. Scenethesis combines LLM scene planning with vision-guided refinement, physics-aware optimization, and scene judging, and experiments show strong layout coherence, spatial realism, and physical plausibility.
Problem
Existing learning-based methods rely on small, indoor object-annotated datasets, while LLMs lack visual perception and produce unrealistic placements that violate spatial and physical constraints.
Method
Scenethesis is a training-free framework that combines LLM-based coarse planning, vision-guided layout refinement, physics-aware pose optimization, and scene-level spatial-coherence judging.
Results
Experiments show that Scenethesis significantly outperforms state-of-the-art baselines in layout coherence, spatial realism, and plausibility, while reducing collisions and improving stability, reachability, and walkability.
Takeaways & Limitations
Scenethesis generates diverse, realistic, physically plausible, and interactive 3D scenes for virtual content creation, simulation environments, and embodied AI research.
Takeaways & Limitations
The framework is limited by retrieval databases because current generative 3D methods cannot yet handle articulated objects, and visual pose initialization can be biased by occlusion and segmentation errors.
Abstract
from arXiv · showhide
Synthesizing interactive 3D scenes from text is essential for gaming, virtual reality, and embodied AI. However, existing methods face several challenges. Learning-based approaches depend on small-scale indoor datasets, limiting the scene diversity and layout complexity. While large language models (LLMs) can leverage diverse text-domain knowledge, they struggle with spatial realism, often producing unnatural object placements that fail to respect common sense. Our key insight is that vision perception can bridge this gap by providing realistic spatial guidance that LLMs lack. To this end, we introduce Scenethesis, a training-free agentic framework that integrates LLM-based scene planning with vision-guided layout refinement. Given a text prompt, Scenethesis first employs an LLM to draft a coarse layout. A vision module then refines it by generating an image guidance and extracting scene structure to capture inter-object relations. Next, an optimization module iteratively enforces accurate pose alignment and physical plausibility, preventing artifacts like object penetration and instability. Finally, a judge module verifies spatial coherence. Comprehensive experiments show that Scenethesis generates diverse, realistic, and physically plausible 3D interactive scenes, making it valuable for virtual content creation, simulation environments, and embodied AI research.
1. Introduction
Interactive 3D scene synthesis requires realistic object arrangements and physical interactions, but existing datasets and LLM-based methods leave important spatial limitations. Scenethesis addresses this gap by combining language planning with vision-guided refinement.
- Limitations of Existing Methods: Learning-based methods rely on small, indoor object-annotated datasets that often contain collisions and neglect small-object interactions.These datasets primarily model large furniture layouts, limiting scene diversity and functional relationships.
- Limitations of Existing Methods: LLMs expand scene diversity through textual common-sense knowledge but lack visual perception for accurate real-world spatial relations.This can produce misplaced or misoriented objects and restrict small items to predefined locations.
- Scenethesis: Scenethesis is a training-free framework integrating LLM-based coarse planning with vision-guided spatial refinement and physical-aware optimization.Its pipeline also uses a judge module to verify spatial coherence.
- Scenethesis: Scenethesis reportedly outperforms state-of-the-art methods in scene diversity, layout realism, and physical plausibility.The framework generates both indoor and outdoor scenes and supports downstream virtual content creation, editing, and simulation-related uses.
2. Related Work
Prior interactive scene-generation methods include manual, procedural, learning-based, image-based, and physics-aware approaches. Their limitations include scalability, simplified spatial relations, restricted datasets, and insufficient 3D or physical reasoning.
- Existing Scene Synthesis: Manual design is labor intensive, while procedural methods produce overly simplified scenes that miss diverse real-world spatial relations.
- Image-based Methods: 2D inpainting approaches improve occlusion handling but can produce misaligned, floating, or intersecting objects without physical constraints and 3D reasoning.Such issues make functional object relationships difficult to maintain for embodied AI tasks.
3. Method
Scenethesis generates interactive 3D layouts through coarse language planning, visual refinement, asset retrieval, and physics-aware optimization. Its optimization aligns retrieved assets with visual guidance while enforcing collision avoidance and stability.
- Coarse Scene Planning: The LLM interprets a prompt, selects associated objects, identifies an anchor, and organizes objects into a coarse spatial hierarchy.The anchor provides the central reference for positioning other objects.
- Layout Visual Refinement: The vision module generates image guidance, constructs a scene graph from segmentation, depth, and 3D bounding-box estimation, and retrieves assets.The scene graph encodes inter-object relationships and initializes the layout.
- Layout Visual Refinement: Occlusion, limited viewpoints, segmentation errors, and asset mismatches can bias estimated object scale, orientation, and position.These errors motivate later pose adjustments during optimization.
- Pose Alignment: Semantic correspondence matching minimizes 2D and 3D correspondence error to refine each asset’s scale, translation, and upright rotation.The method is designed to remain robust to occlusions and partial views.
- Physics-aware Optimization: SDF-based physical optimization replaces bounding-box collision approximations with surface-based geometry for more precise collision and stability constraints.It processes objects hierarchically, beginning with the anchor, and constrains retrieved assets to azimuthal rotation.
- Physics-aware Optimization: Collision handling adjusts translation away from collided surface points and can reduce object scale, while stability enforces contact between bottom points and parent surfaces.The collision direction points from each collision location toward the model centroid.
4. Experiment
Scenethesis is evaluated against generative and LLM-based baselines on text alignment, layout realism, physical plausibility, interactivity, and qualitative scene diversity. Results indicate strong performance across these dimensions, with vision guidance and physics-aware optimization supporting realistic, stable, and accessible scenes.
- Evaluation Setup: The evaluation compares Scenethesis with DiffuScene, PhyScene, SceneTeller, and Holodeck using indoor scenes across diverse categories.The setup includes 22 indoor scenes spanning six primary and twelve secondary categories; outdoor scenes are excluded for fair comparison.
- Controllability: Scenethesis achieves the highest CLIP, BLIP, and VQA scores among the evaluated baselines.These metrics assess alignment between generated scenes and their text descriptions.
- Layout Realism: Scenethesis attains state-of-the-art layout realism on most metrics and outperforms Holodeck in broader indoor settings.It remains comparable or superior to trained indoor methods in residential areas while performing strongly in shopping, tourism, and sports facilities.
- Physical Plausibility and Interactivity: Physics-aware layout adjustment reduces collisions and improves stability, while Scenethesis also achieves superior reachability and walkability.The evaluation measures object- and scene-level collision and instability together with object reachability and walkable-area coverage.
- Qualitative Evaluation: Scenethesis generates diverse indoor and outdoor scenes and supports both simple prompts for flexible generation and detailed prompts for controllable layouts.Qualitative examples also show fine-grained placement of small objects at different levels within shelves and carts.
- Ablation Study: The ablation study evaluates pose alignment, collision, and stability constraints as components of the physics-aware optimization.The reported progression distinguishes raw layout, pose alignment, collision prevention, and grounded stable placement.
5. Conclusion and Limitation
Scenethesis combines LLM-based coarse planning, vision-guided refinement, physics-aware optimization, and scene judging for interactive 3D scene generation. The conclusion reports improved layout coherence, spatial realism, and plausibility, while identifying retrieval databases and the lack of articulated-object synthesis as limitations.
- Conclusion: Scenethesis is a training-free framework combining coarse LLM planning, vision-guided refinement, physics-aware position adjustment, and scene judging.Its stated goal is high-fidelity interactive 3D scene generation.
- Conclusion: Experiments show significant improvements over state-of-the-art baselines in layout coherence, spatial realism, and plausibility.
- Limitation: The method is limited by retrieval databases because current generative 3D methods cannot yet handle articulated objects.Future generative 3D advances are identified as a way to enable articulated-object synthesis and increase scene diversity.
6. Implementation Details of Scenethesis
Scenethesis implements text-to-3D generation through coarse LLM planning, vision-guided refinement, physics-aware optimization, and spatial-coherence judgment. Its optimization aligns object poses with image guidance while enforcing collision avoidance and stability.
- Coarse Scene Planning: The pipeline takes user text and uses an LLM to produce an object list, an upsampled prompt, and a coarse scene plan.The LLM reviews available assets, selects relevant objects, checks availability, plans the scene, and generates output files.
- Layout Visual Refinement: Vision-guided refinement generates an image reference, segments objects, estimates depth, extracts initial 5DoF poses, constructs a scene graph, and retrieves assets and environment maps.Grounded-SAM provides masks and crops, Depth Pro provides depth, a VLM generates the scene graph, and CLIP retrieves 3D assets.
- Physics-aware Optimization: Pose alignment minimizes 2D correspondence and 3D point-cloud distances between rendered objects and their counterparts in the guidance image.RoMa supplies dense semantic correspondences, and confident matched points are used during optimization.
- Physics-aware Optimization: The iterative optimization transforms meshes, renders RGB and depth images, matches correspondences, computes pose, collision, and stability losses, and updates object poses.The process traverses objects in the scene graph and updates the scene SDF after optimization.
- Scene Spatial Coherent Judgment: A VLM judges multi-view renders for spatial coherence and returns the optimized interactive scene, regenerating it when the result is not qualified.The implementation also notes that missing unique assets can create discrepancies between generated scenes and image guidance.
- Physics-aware Optimization: Physical plausibility optimization uses surface-point signed distance fields to detect collisions rather than approximating objects with 3D bounding boxes.The optimization addresses collision avoidance and stability, including collision-driven translation and scale adjustments.
7. Prompts Examples
The prompts operationalize Scenethesis through structured object selection, anchor-based spatial planning, and explicit evaluation criteria. Examples then apply these instructions to gym-scene planning, generated-scene comparison, and pose-alignment assessment.
- Coarse Scene Planning Instruction Prompts: The planning task reviews available categories, interprets the prompt, selects relevant objects, identifies an anchor, cross-checks availability, and saves objects and scene plans.The instructions require at least 15 matching object categories and an anchor that influences placement and scene orientation.
- Coarse Scene Planning Instruction Prompts: The anchor is a large ground-based object that influences other placements and defines the scene’s layout orientation.Examples include a dining-room table and a front-facing sofa.
- Coarse Scene Planning Instruction Prompts: Planning emphasizes functionality, spatial hierarchy, depth, supportive items, and balanced mixtures of large and small objects.The instructions place objects across background, central, and foreground positions while avoiding overcrowding or under-population.
- Coarse Scene Planning Output Example: For a gym, the treadmill is selected as the anchor alongside equipment, storage, accessories, and close category alternatives when exact labels are unavailable.The example cross-checks each object against the provided list, accepting alternatives such as “towel rack” for towel and “ball” for exercise ball.
- Coarse Scene Planning Output Example: The gym layout places shelves and cabinets in the background, the treadmill and training equipment in the middle ground, and smaller accessible items in the foreground.This arrangement is intended to create balance, practical access, and visual depth.
- 7.3. Instruction for Evaluating Generated Scenes: Generated scenes are evaluated for object diversity, layout coherence, spatial realism and complexity, physical plausibility, and alignment with the text prompt.Layout coherence considers realistic positions and orientations, while spatial complexity considers varied object relations and hierarchy.
- 7.4. Evaluation Example of Generated Scenes: The evaluation example reports that Scenethesis’s warehouse scene contains 8 object types and 13 total objects, with dispersed placement and varied spatial relations.The comparison describes the scene as more diverse and realistic than the cited Holodeck example, which has 9 total objects and fewer types.
- 7.5. Instruction Prompts for Ablation Study: The ablation prompt scores pose alignment using location and size similarity, orientation similarity, and overall layout similarity against an image guidance.Style, appearance, shape, and texture are excluded from the assessment.