PaperlayerView exact revision on arXiv

Source-linked AI summary

Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting

Yian Wang, Xiaowen Qiu, Jiageng Liu, Zhehuan Chen, Jiting Cai, Yufei Wang, Tsun-Hsuan Wang, Zhou Xian, Chuang Gan

arXiv:2411.09823v1cs.CV

TL;DR

Creating complex interactive 3D scenes remains challenging because existing approaches rely on substantial human effort, predefined rules or datasets, or have limited spatial reasoning. ARCHITECT uses controlled, hierarchical 2D inpainting to generate and lift scenes into 3D, and the authors report more complex and realistic scenes than prior approaches. Its furniture diversity remains bounded by the asset dataset.

  • Problem

    Creating realistic, complex interactive 3D environments remains challenging because existing methods require substantial effort, rely on predefined rules or datasets, or have limited 3D spatial reasoning.

    Source: §1 · p. 2

  • Method

    ARCHITECT iteratively inpaints rendered scene images, uses visual perception and depth estimation to reconstruct objects in 3D, and hierarchically generates large furniture and smaller objects.

    Source: §1 · p. 3, §3 · p. 4

  • Results

    The authors report that ARCHITECT outperforms prior approaches in generating interactive 3D scenes that are more complex and realistic.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The framework produces open-vocabulary scenes across home and non-home settings, with detailed, interactable object placements at multiple scales.

    Source: §1 · p. 3

    Furniture diversity is limited by the datasets from which ARCHITECT retrieves furniture and large objects.

    Source: §5 · p. 10

Abstract

from arXiv · show

Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing Architect, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments.

1 Introduction

ARCHITECT addresses the difficulty of creating complex, realistic interactive 3D environments by using 2D diffusion inpainting to generate and reconstruct scenes. Its iterative, hierarchical process supports multiple scene scales and starting points, and the paper reports stronger scene-generation results than prior approaches.

1 Introduction

  • Existing approaches face scaling, vocabulary, spatial-reasoning, and configuration challenges that hinder realistic, complex interactive 3D scene generation.Manual datasets require substantial human effort; procedural methods rely on predefined rules, while LLM-guided methods struggle with nuanced placements and cluttered arrangements.Source: §1 · p. 2
  • ARCHITECT uses diffusion-based 2D inpainting and geometric cues from rendered backgrounds to control camera parameters and depth scale for lifting generated images into 3D.Visual recognition models segment generated objects, which are instantiated in simulation using retrieved assets or image-to-3D generation.Source: §1 · p. 3
  • Iterative, hierarchical inpainting generates large furniture and smaller objects across multiple views, and can start from text descriptions, floor plans, or pre-arranged scenes.The process refines scene complexity by rendering new patches at different scene locations.Source: §1 · p. 3
  • ARCHITECT generates open-vocabulary scenes, including home and non-home environments, and the authors report that it outperforms prior approaches in interactive-scene complexity and realism.The paper describes its results as both quantitative and qualitative.Source: §1 · p. 3

2 Related Works

Prior scene-generation approaches include static mesh creation, rule-based procedures, dataset-trained models, and LLM-guided design. ARCHITECT combines image-generation priors with simulation rendering and controlled inpainting, and the section points to a five-aspect comparison with prior work.

Indoor Scene Generation

  • Prior indoor-scene work includes static mesh generation, rule-based interactive scenes, dataset-trained generative models, and LLM-guided floor-plan, layout, and object placement.The cited comparison table covers five aspects of large-scale 3D scene generation; “No Train” means no indoor-layout training data is needed.Source: §2 · pp. 3–4, Table 1 · p. 4
  • Rule-based methods depend on manually defined rules, dataset-trained models are bounded by their training data, and LLM-based methods have limited 3D spatial reasoning.The authors say their method uses neither manually defined rules nor pre-collected indoor-scene datasets.Source: §2 · p. 4

Text-to-Image Diffusion models

  • The authors combine 2D image-generation models, simulation rendering, and controlled image inpainting to pursue more coherent 3D layouts and higher scene diversity.They motivate the approach with image diffusion models’ visual and 3D priors and their observed priors for furniture and object layout.Source: §2 · p. 4

3 Method

ARCHITECT iteratively renders, inpaints, perceives, and places objects to build interactive scenes at multiple scales. Rendered geometry and depth guide image-to-3D reconstruction, while hierarchical inpainting supports both large furniture and smaller objects.

3 Method

  • The pipeline iterates through initialization, inpainting, visual perception, and placement to generate large furniture first and then smaller objects.After placement, it returns to initialization to continue generating objects.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Initializing Module

  • Initialization renders a selected scene view and provides an inpainting mask and ground-truth depth for later stages.For large furniture, the method selects a view spanning opposite room corners and excludes existing objects from the mask; for small objects, it selects a view and mask based on the target furniture.Source: §3.1 · p. 5

3.2 Hierarchical Inpainting Module

  • Hierarchical inpainting uses LLM-generated prompts with rendered images and masks to generate furniture and small objects, filtering images with too few recognized objects.Mask erosion and Gaussian blur prepare the inpainting region; prompts and negative prompts help encourage suitable objects and avoid duplicates.Source: §3.2 · p. 5

3.3 Visual Perception Module

  • Visual perception detects and segments generated objects, rescales estimated depth using reference pixels, and back-projects object point clouds into 3D bounding boxes.For small objects, reference pixels come from non-masked furniture areas because depth outside the inpainted objects can be discontinuous; DBSCAN removes point-cloud outliers.Source: §3.3 · p. 6

3.4 Placing Module

  • Placement retrieves large furniture by text description and uses constraints derived from images and 3D bounding boxes to avoid collisions and align placements with generated images.Small objects are generated from semantic content and adjusted to match bounding-box position, orientation, and size, using dimensions perpendicular to the viewing direction for scale.Source: §3.4 · pp. 6–7

4 Experiments

ARCHITECT is evaluated against indoor-scene generation methods using qualitative comparisons, image-based metrics, and user studies. The experiments also demonstrate robotic tasks and generated objects at multiple scales.

4 Experiments

  • The experiments compare household and non-household scenes; DiffuScene is compared only for household scenes, while Text2Room comparisons appear in Figure 8.The evaluation uses CLIPScore, BLIPScore, VQAScore, GPT-4o ranking, and user ratings for visual quality, semantic correctness, layout correctness, and overall preference.Source: Figure 3 · p. 7, §4 · p. 8

4.1 Scene Generation

  • Compared with Holodeck, ARCHITECT produces more realistic scenes and more complex small-object placements, including paired shoes, shelf toys, and organized table items.The comparison also reports Holodeck's hair-salon scenes as semantically incorrect.Source: §4.1 · p. 8
  • ARCHITECT outperforms other methods on CLIP, BLIP, and GPT-4o ranking, scores slightly below Text2Room on VQA, and leads all four user-study aspects.The study rates visual quality, semantic correctness, layout coherence, and overall preference.Source: §4.1 · p. 8, Table 2 · p. 9

4.2 Embodied/Robotic task

  • Detailed small objects slow the embodied environment, so LLMs select relevant objects for each task substep.Other objects are designated as background and are not physically simulated during that substep.Source: §4.2 · pp. 8–9
  • The robotic-task pipeline uses floor plans, furniture, and small objects to generate tasks, decompose them, filter irrelevant objects, and produce action trajectories.The paper presents generated tasks and collected trajectories, and compares task-description diversity using self-BLEU.Source: §4.2 · pp. 8–9

4.3 Object Generation

  • For large furniture, ARCHITECT supplements dataset retrieval with text-guided mesh generation that optimizes a differentiable tetrahedron mesh using SDS loss and RichDreamer diffusion models.The paper also shows qualitative small-object generation examples.Source: §4.3 · p. 10, Figure 6 · p. 10

5 Conclusion and Future Work

ARCHITECT uses pre-trained 2D inpainting diffusion models and geometric cues to generate complex 3D environments. Its furniture diversity remains limited by the retrieved dataset.

5 Conclusion and Future Work

  • ARCHITECT iteratively extracts layouts from image inpainting, controls camera parameters and depth scale with geometric cues, and projects results into 3D point clouds.The conclusion reports that qualitative and quantitative comparisons outperform baselines in realism and diversity.Source: §5 · p. 10
  • Furniture diversity is inherently limited by the dataset from which ARCHITECT retrieves furniture and other large objects.The authors identify generating higher-quality, articulated objects as future work.Source: §5 · p. 10

A Implementation Details

ARCHITECT combines constraints and search for object placement with separate asset-selection strategies for large furniture and small objects.

A.1 Constraint and Search

  • Floor-object constraints encode room position, proximity, relative layout, alignment, location, and rotation; DFS filters hard violations and ranks placements by soft scores.Constraints are derived from floor-object bounding boxes except rotation, which uses an LLM; greedy pruning explores only the three highest-scoring nodes at each step.Source: §A.1 · p. 14
  • Wall objects default to facing away from the wall, and soft location constraints suffice because orientation is unnecessary and wall conflicts are less likely.Their search process is nearly the same as for floor objects.Source: §A.1 · p. 15

A.2 Large Furniture Retrieving

  • Large furniture assets are retrieved by text description, then selected using scale similarity and, in the latest pipeline, CLIP image similarity.Similarity is used for candidate selection rather than retrieval because occlusions can impair retrieval accuracy.Source: §A.2 · p. 15

A.3 Small Objects Generation and Selection

  • Small-object 3D assets come from text-to-image-to-3D generation, with DINO feature similarity selecting among candidates; segmented-image-to-3D produced suboptimal shapes and textures.Source: §A.3 · p. 15

A.4 View Selection

View selection uses heuristic camera views for large-furniture inpainting and object-relative views for small objects.

A.4 View Selection

  • For large furniture, up to three views are selected to cover the room, stopping when occupancy exceeds 0.7 or all three views have been used.The views span opposite corners, the room’s middle, and the other diagonal; rendering uses an 84-degree FOV, yielding approximately 95 percent visibility from a single corner-to-corner view in a square room.Source: §A.4 · p. 15
  • Small-object inpainting uses top-down views for tables or desks and front views for shelves or cabinets, with camera distance adjusted to show the full object.LLMs first identify which objects can accommodate small objects on or in them.Source: §A.4 · p. 15

B.1 More Comparison Cases

The comparisons report greater photorealism than Text2Room and more reasonable, detailed scenes than Diffuscene, while noting differences in interactivity and small-object placement.

B.1 More Comparison Cases

  • ARCHITECT is reported to produce more photorealistic scenes than Text2Room and more reasonable, detailed scenes than Diffuscene.Text2Room's projected RGB pixels produce unavoidable artifacts and distorted, non-interactive meshes; Diffuscene's dataset lacks small-object placement information, limiting placement diversity.Source: §B.1 · p. 16, Figure 7 · p. 16

B.2 Comparison with Baselines

The baseline comparisons span static-mesh generation, dataset-trained methods, and foundation-model approaches; the authors caution that Text2Room serves a different purpose.

B.2 Comparison with Baselines

  • The cited PhyScene comparison reports Table 4 results only for generated living rooms, because only that generation weight is released.Source: §B.2 · p. 16
  • The experiments compare ARCHITECT with static-mesh generators, methods trained on existing datasets, and foundation-model approaches.Source: §B.2 · p. 16
  • A completely fair comparison with Text2Room is difficult because it generates meshes without retrieving objects and may achieve higher photorealism through direct 2D-image-to-mesh generation.Text2Room's assets are not interactive.Source: §B.2 · p. 16

B.3 Controllability and Editing

ARCHITECT combines diffusion-based scene generation with LLM-enabled control and editing, and evaluates small-object placement, image fidelity, mask consistency, and text-directed edits. The section also describes limitations of LLM placement and reports initial editing results.

B.3 Controllability and Editing

  • ARCHITECT combines diffusion-based generation with LLM-enabled scene control and editing, using object names, positions, scales, bounding boxes, orientations, and asset identifiers as text.The text representation can be supplied to LLMs to control or edit scene layouts.Source: §B.3 · pp. 16–17

B.4 Comparison of Architect and LLMs in Small Object Placement

  • LLMs struggle with small-object placement because shelf support surfaces may be unknown and table placements can have poor orientations or severe collisions.The paper points to the middle of Figure 9 and Table 3 for comparisons with its method.Source: §B.4 · p. 17

B.5 Similarity of Generated Scenes and Inpainted Images

  • Generated scenes are not identical to inpainted images, but image-similarity evaluation indicates they are somewhat faithful to the inpainting results.The evaluation compares inpainted images and generated scenes against empty scenes.Source: §B.5 · p. 17

B.6 Consistency of Inpainting

  • Softened inpainting-mask boundaries are used to improve consistency; Figure 9 marks an inconsistency when masks are not blurred.The reported consistency is both stylistic and geometric.Source: §B.6 · p. 17, Figure 9 · p. 18

C Scene Editing

  • APIs for adding, removing, and rescaling objects let LLMs carry out text-directed scene edits, and the initial commands produced the requested results.Examples include replacing shelf books with vases, replacing a bookshelf with a cabinet, and reducing its size.Source: C Scene Editing · p. 18

D User Study Details

The appendix details a volunteer study for rating generated scenes, prompts used for object recognition and inpainting, and potential societal benefits and harms. The study aggregates participant ratings across four scene-quality criteria.

D User Study Details

  • 115 volunteers each rated 23 scenes on visual quality, semantic correctness, layout coherence, and overall preference; responses under 230 seconds were excluded.Average ratings were calculated for comparison with Holodeck, Text2Room, and DiffuScene.Source: D User Study Details · pp. 18–19

E.1 Object Recognition Prompt

  • GPT-4V identifies and describes objects in an inpainting image, classifying each as a floor object or wall object for later retrieval or text-to-3D generation.Generated object names are used to prompt Grounded-SAM.Source: §E.1 · p. 20

E.2 Inpainting Prompt and Generation Prompt

  • GPT-4V lists objects that have reached their limits and those still lacking, which are then used as negative and positive prompts for inpainting.The prompt asks for precise missing objects, ideally ones not already in the scene.Source: §E.2 · p. 20

F.1 Positive Impacts

  • The paper identifies robotics assistance, immersive learning, and accessibility technologies as potential positive impacts of generated environments.The appendix describes these as potential uses and benefits.Source: §F.1 · pp. 20–21

F.2 Negative Impacts

  • Potential harms include job displacement, biased outcomes, and misuse of inferred 3D object geometry for unauthorized reproduction or surveillance.The paper calls for consideration of economic and societal impacts and representative training data and algorithms.Source: §F.2 · p. 21

NeurIPS Paper Checklist

The checklist affirms that the paper’s claims reflect its contributions and scope and reports several reproducibility, resource, asset, and human-subject practices. It also records that experiments omit error bars because of computational cost.

NeurIPS Paper Checklist

  • The checklist states that the abstract and introduction accurately reflect the contribution and scope, and that limitations are discussed in Section 5.The contribution is described as a zero-shot pipeline for generating diverse, complex, realistic interactive 3D scenes.Source: NeurIPS Paper Checklist · p. 22
  • The authors report that pipeline details support reproducibility, code will be made public, and ARCHITECT needs no indoor-layout training data.The checklist points to the main text for the training-data statement.Source: NeurIPS Paper Checklist · pp. 23–24
  • Error bars are not reported because the authors say producing them would be too computationally expensive.Source: NeurIPS Paper Checklist · p. 24
  • The checklist says computer-resource details for the experiments are provided in Section 4.Source: NeurIPS Paper Checklist · p. 25
  • The checklist reports that existing assets are credited and their licenses respected, and that the new method is documented alongside its assets.Source: NeurIPS Paper Checklist · pp. 26–27
  • The appendix includes human-study instructions and screenshots, states that volunteers were uncompensated, and reports no participant risks and IRB approval.Source: NeurIPS Paper Checklist · p. 27

Related papers

Loading 2411.09823v1…