Source-linked AI summary
SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
Xingjian Ran, Xiaoye Mo, Sihao Liu, Jianyu Zhang, Li Luo, Bo Dai
TL;DR
Existing indoor-scene generators trade efficiency against fidelity and usually produce only one layout per input, limiting scalable, diverse simulation environments. SceneMosaic combines image-based initialization with VLM-driven local layout evolution and Cartesian assembly; on SceneEval-100, it matches the strongest agentic baseline in semantic quality with a 24× speedup, reduces physical violations, and receives the highest human ratings.
Problem
Existing agentic pipelines require costly iterative refinement, image-to-3D models often produce physically invalid poses, and both paradigms struggle to generate diverse layouts from one input.
Method
SceneMosaic initializes scenes from image-based layout priors, refines them with VLM agents, and evolves independent local units before Cartesian composition.
Results
SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24× speedup, substantially reduces physical violations, and receives the highest human ratings.
Takeaways & Limitations
SceneMosaic produces diverse, physically plausible scene variants while preserving base-scene coherence for scalable simulation-ready generation.
Takeaways & Limitations
A single input image typically depicts only one room, so SceneMosaic is limited to room-scale scenes and cannot directly generate multi-room houses or buildings.
Abstract
from arXiv · showhide
Diverse and simulation-ready indoor scenes are essential for interactive entertainment and embodied AI, yet their scalable generation remains challenging. Recent agentic text-to-3D scene pipelines that rely on vision-language models (VLMs) can generate scenes of high fidelity but require costly iterative object placement and refinement. Another mainstream paradigm, parametric image-to-3D scene models, produces scenes efficiently from strong priors learned from 2D images but often leads to imprecise and physically invalid scenes. More importantly, both paradigms struggle to output diverse scenes for a single input, making it hard for them to reflect the dynamically changing nature of real scenes. In this paper we propose \textbf{SceneMosaic}, a framework that combines the merits of both paradigms. It obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM agents, ensuring both efficiency and physical validity. Within the evolution process, SceneMosaic exploits the locality of natural scenes and decomposes a scene into independent local units, allowing separate evolution within each unit before composing the global scene via Cartesian product. On SceneEval-100, SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Our code is publicly available at https://github.com/rxjfighting/SceneMosaic.
1. Introduction
SceneMosaic addresses the efficiency–fidelity trade-off and limited diversity of existing indoor-scene generators by combining image-based initialization with agentic refinement and local-unit evolution. It produces diverse, physically and semantically valid scenes while substantially reducing generation cost.
- Motivation: Existing simulation environments are often sparsely furnished and lack diversity, limiting scalable training and evaluation for embodied AI.Simulation is valuable because real-world data collection cannot support the required scale for robots deployed in arbitrary human houses.
- Limitations of Existing Paradigms: Agentic text-to-3D pipelines can produce high-quality, physically valid scenes but require costly iterative object-level operations and multihour generation trajectories.These pipelines rely on VLMs to propose, place, and adjust objects through iterative feedback loops.
- Limitations of Existing Paradigms: Parametric image-to-3D models generate scenes efficiently from learned layout priors, but their object poses are often imprecise and physically invalid.Reported failure modes include floating objects, collisions, and boundary violations.
- SceneMosaic: SceneMosaic combines rapid image-based pose initialization with agentic evolution, decomposing scenes into local units that can be separately refined and recombined.The framework targets both efficiency–fidelity and diversity by evolving local layouts rather than producing only one deterministic layout.
- Results: SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24× speedup, while reducing physical violations and receiving the highest human ratings.Its local-unit variants are composed through the Cartesian product and distilled with novelty-aware selection.
3. Method
SceneMosaic reconstructs a reference input into structured local units, stabilizes their layouts physically, evolves diverse variants with Critic–Actor agents, and assembles globally diverse scenes through Cartesian composition and novelty-aware selection.
- Problem Formulation: Given a reference input, SceneMosaic aims to generate a simulation-ready base scene and m diverse variants.The scene is represented as objects with meshes, positions, rotations, scales, and simulation-related properties.
- Scene Reconstruction and Structuring: Object-centric reconstruction registers objects, segments them, and independently reconstructs meshes with initialized 3D layout parameters.This creates correspondence among semantic identities, image evidence, reconstructed meshes, and spatial layouts.
- Scene Reconstruction and Structuring: Relation-guided structuring infers room boundaries, support and containment relations, and a hierarchical scene tree of locally independent units.A manager agent schedules specialized reasoning agents while DAG dependencies maintain information flow.
- Physics-Based Layout Stabilization: The framework stabilizes each local layout through containment correction and gravity-based simulation before agentic evolution.The overview combines reconstruction, physics-based stabilization, Critic–Actor evolution, and final scene selection.
- Critic-Actor Layout Evolution: Critic–Actor evolution generates and refines local variants using visual evidence and simulation feedback, with scoped context and orthographic 2D views simplifying spatial reasoning.The Critic diagnoses validity and suggests updates, while the Actor changes local spatial parameters.
- Diversity-Driven Scene Composition: Local variants are combined through a Cartesian product into globally complete candidates, then filtered using a novelty-aware distance and dynamic greedy search.The distance combines normalized relative position, absolute distance, and rotation differences, while the selected scene serves as the reference in the weighted formulation.
4. Experiments
Experiments compare SceneMosaic with agentic text-to-3D and image-to-3D baselines on semantic quality, physical validity, efficiency, human judgments, and ablated design choices. Results show that hybrid initialization and local agentic evolution improve the trade-offs among these objectives.
- Evaluation Setup: SceneEval-100 compares agentic and image-to-3D baselines using semantic metrics POS and ROT, physical metrics NAV, COL, and OOB, and average generation time.The evaluation uses text prompts for text-to-3D methods and corresponding generated images for image-to-3D methods and SceneMosaic.
- Baseline Comparison: Agentic methods provide complex scene construction but suffer from instruction-following limitations and high latency, whereas image-to-3D methods initialize rapidly but incur severe physical violations.SceneSmith can achieve high performance but may be slower than manual 3D editing; HoloDeck is faster but lacks global semantic guidance.
- Main Results: SceneMosaic achieves competitive semantic quality, significantly reduces physical violations, and lowers inference overhead relative to agentic baselines while producing faster variants than image-to-3D models.Its qualitative comparisons combine image-derived global structure with agentic evolution to improve object relations and physical plausibility.
- User Study: The user study finds SceneMosaic highest on semantic and physical plausibility, with the lowest standard deviations among methods evaluated by 48 participants.Participants rated 15 scenes on a 5-point Likert scale, and Kendall’s concordance indicates strong agreement on both dimensions (W ≥0.633, p < 0.001).
- Ablation Study: Removing image initialization increases generation latency and slightly degrades semantic alignment, while removing physics tools increases collision and out-of-bounds rates.Replacing orthographic 2D abstraction with perspective 3D operations also increases inference time and degrades semantic orientation and position quality.
- Ablation Study: Global evolution without local unit decomposition produces the lowest overall semantic scores, elevated collision rates, slow convergence, and no acceleration for variant generation.Local decomposition instead enables combinatorial Cartesian assembly of diverse variants at a fraction of the cost.
5. Conclusion
SceneMosaic combines image-based initialization with agentic layout evolution to generate efficient, diverse, simulation-ready scenes. On SceneEval-100 and in a user study, it achieves strong semantic quality, removes reported physical violations, and substantially reduces generation time.
- SceneMosaic initializes scenes from image-based layout priors and refines them through agentic layout evolution over orthographic 2D projections.
- Local unit decomposition enables Cartesian composition of diverse layout variants at negligible extra cost.
- SceneMosaic matches or surpasses state-of-the-art baselines in semantic quality and eliminates collisions and out-of-bounds placements.
- SceneMosaic reduces generation time by an order of magnitude compared to state-of-the-art feedback-driven agentic pipelines.
6. Implementation Details
The implementation uses role-specific VLM agents, physics-based stabilization, and orthographic projections with coordinate mappings to support scene perception and layout evolution.
- Agent Configuration: Perception, relation-reasoning, and Critic-Actor evolution agents use role-specific prompts over the same underlying VLM.
- Agent Configuration: Reference scene images are synthesized with GPT-Image-2 from SceneEval-100 prompts.
- Physics Simulation: Physics-based stabilization and validation use rigid-body simulation with reusable watertight collision hulls from convex mesh decompositions.
- Orthographic Projection and Coordinate Mapping: Each local unit is rendered as a 1024 × 1024 orthographic view along its anchor-dependent projection direction.
- Orthographic Projection and Coordinate Mapping: Grid correspondences map normalized 2D coordinates to 3D anchor-surface positions, allowing predicted 2D poses to be lifted deterministically into 3D layout parameters.
7. Details of Critic-Actor Layout Evolution
SceneMosaic evolves each local layout through Critic-Actor feedback using structured 2D states, collision information, symbolic pose updates, and round-based stopping rules.
- Critic: At each evolution round, the Critic evaluates orthographic renderings, structured 2D layouts, collision pairs, cross-unit constraints, and layout memory.
- Actor: The Actor converts Critic suggestions into symbolic pose expressions resolved against the current layout for exact relational adjustments.
- Figure 4 shows randomly sampled variant scenes that remain physically plausible while preserving the input scene’s semantic structure.
- Layout Memory: Layout memory records poses, suggestions, resolved updates, and collisions across rounds to detect repeated failures and avoid cyclic adjustments.
- Convergence and Round Selection: Evolution stops when the Critic requests no changes, the round limit is reached, or translation, rotation, collision, and boundary criteria indicate convergence.
8. Cross-Unit Constraint Extraction
Because local units evolve independently, SceneMosaic extracts sparse cross-unit constraints to preserve global coherence through orientation and placement alignments.
- Independent local evolution requires sparse cross-unit constraints to preserve global coherence.
- A dedicated agent derives orientation and placement alignments from the scene image and anchor-grouped object relations.
9. Combinatorial Assembly and Validation
SceneMosaic validates local layouts before assembly, expresses them in anchor-relative coordinates, and composes globally complete candidates through Cartesian products and final collision checking.
- Non-original local variants with collisions or out-of-bounds objects are discarded before Cartesian combination.
- Anchor-relative coordinates allow child-unit variants to be transplanted under alternative parent-unit variants.
- Candidate scenes are generated by recursively expanding the scene tree and taking Cartesian products across sibling subtrees.
- Every assembled candidate undergoes a final scene-level collision check before selection.
- Representative selection uses novelty-aware pairwise distances with weights (wp, wu, wr) = (0.4, 0.4, 0.2).
10. Diversity Evaluation
The diversity evaluation controls for asset variation, compares selection strategies and perturbation baselines, and finds that novelty-aware selection produces less redundant layouts without sacrificing validity.
- Diversity comparisons use degraded variants sharing reconstructed assets and producing m = 5 layouts per input.
- Dynamic Max-Min Greedy Search selects representative scenes using iterative max-min selection with dynamic pruning.
- Div-IoU averages pairwise top-down occupancy-map IoU, with lower values indicating greater diversity, while Div-N uses novelty-aware distance, with higher values indicating greater diversity.
- Uniform random sampling preserves quality but yields markedly more redundant layouts, showing that diversity depends on selection strategy rather than the candidate pool alone.
- Repeated sampling is both the least diverse and the most expensive setting because independent resampling tends to rediscover the same locally optimal arrangement.
- At matched diversity, Gaussian perturbations retain substantially lower semantic scores and relation preservation than SceneMosaic.
11. Limitations
SceneMosaic is constrained to room-scale generation because its image-based initialization typically uses a single image depicting one room.
- SceneMosaic cannot directly generate multi-room houses or entire buildings from its single-image initialization.The paper identifies composing room-level generations into coherent house-scale environments as future work.
12. Prompt Design for Layout Evolution
The prompt design uses specialized agents for proposing, instantiating, critiquing, and refining alternative layouts while enforcing geometric, support, collision, and cross-anchor constraints.
- Prompt roles: Prompt 1 proposes alternative layout archetypes, while Prompt 2 instantiates each archetype into concrete 2D poses.
- Prompt roles: Prompt 4 serves as the Actor that updates local spatial layouts based on Critic suggestions.
- Layout constraints: Alternative layouts must be structurally distinct and remain supported by anchors without overlap or boundary violations.
- Layout constraints: Cross-anchor constraints guide objects associated with different anchors so combined layouts align or orient correctly.
- Output format: Prompt outputs limit alternative counts according to object count and permit an empty array when the existing layout is the only representative option.
- Pose representation: Pose outputs specify object identifiers, 2D centers, and absolute rotation angles using the stated coordinate convention.
- Prompt roles: Prompt 3 evaluates physical collisions and semantic placement errors, then returns correction guidance and a quality score from 0 to 100.