Source-linked AI summary
ControlRoom3D: Room Generation using Semantic Proxy Rooms
Jonas Schult, Sam Tsai, Lukas Höllein, Bichen Wu, Jialiang Wang, Chih-Yao Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, Peizhao Zhang, Bastian Leibe, Peter Vajda, Ji Hou
TL;DR
Text-conditioned room generation can yield implausible layouts, while manual 3D scene creation requires modeling expertise. ControlRoom3D guides 2D generative models with a user-defined 3D semantic proxy room. The authors report diverse, globally plausible room meshes and substantial quality gains from mesh post-processing and completion.
Problem
Text-conditioned room meshes may violate typical layouts, such as by placing several beds in one bedroom, while manual 3D scene creation requires modeling expertise.
Source: §1 · pp. 1–2Method
Results
ControlRoom3D generates diverse, globally plausible room meshes, and mesh post-processing and completion substantially improve final generation quality.
Source: §5 · p. 9Takeaways & Limitations
The method produces room meshes aligned with a user-defined layout and style, with the authors reporting greater plausibility than competing methods.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Manually creating 3D environments for AR/VR applications is a complex process requiring expert knowledge in 3D modeling software. Pioneering works facilitate this process by generating room meshes conditioned on textual style descriptions. Yet, many of these automatically generated 3D meshes do not adhere to typical room layouts, compromising their plausibility, e.g., by placing several beds in one bedroom. To address these challenges, we present ControlRoom3D, a novel method to generate high-quality room meshes. Central to our approach is a user-defined 3D semantic proxy room that outlines a rough room layout based on semantic bounding boxes and a textual description of the overall room style. Our key insight is that when rendered to 2D, this 3D representation provides valuable geometric and semantic information to control powerful 2D models to generate 3D consistent textures and geometry that aligns well with the proxy room. Backed up by an extensive study including quantitative metrics and qualitative user evaluations, our method generates diverse and globally plausible 3D room meshes, thus empowering users to design 3D rooms effortlessly without specialized knowledge.
1. Introduction
Text-conditioned room generation can ease the burden of manual 3D modeling, but locally guided methods may produce implausible layouts and duplicate prominent objects. ControlRoom3D uses a user-defined semantic proxy room to guide generation toward globally plausible meshes, with its components evaluated quantitatively and qualitatively.
1. Introduction
- Text2Room can produce unrealistic room layouts and duplicate prominent objects, such as placing several beds in one bedroom.Its iterative mesh inpainting strategy relies solely on local context.Source: §1 · pp. 1–2
- ControlRoom3D uses semantic bounding boxes and room-style text as a 3D proxy that guides 2D models to generate aligned, 3D-consistent textures and geometry.Its approach includes guided panorama generation, geometry alignment, and mesh completion.Source: §1 · p. 2
- The authors report that ControlRoom3D generates diverse, semantically plausible room meshes aligned with the user-defined layout and style, and is more plausible than competing methods.They evaluate the introduced components using quantitative and qualitative metrics.Source: §1 · p. 2
2. Related Work
Prior text-to-3D room methods offer limited control over global composition, while semantic scene-generation approaches face texture, room-setting, or dataset constraints. ControlRoom3D derives 2D guidance from a 3D proxy room to control scene composition and generate consistent room images.
Text-to-3D
- Text-conditioned inpainting and monodepth methods, including SceneScape [15] and Text2Room, offer limited control and typically produce globally implausible room layouts.ControlRoom3D instead uses globally consistent 2D maps derived from 3D semantic proxy rooms to guide scene composition.Source: §2 · p. 2
Semantically Guided 3D Scene Generation
- Semantic scene-generation methods can be constrained to low-resolution textures, partial-room settings, or room styles represented in synthetic 3D datasets.ControlRoom3D uses 2D controls derived from a 3D representation to produce high-resolution images without inward-facing views.Source: §2 · p. 2
Mesh Texturization
- Mesh re-texturization methods stylize existing geometry, whereas ControlRoom3D also creates novel geometry aligned with the texture style.The overview depicts proxy-derived controls guiding panorama generation, depth alignment, and iterative mesh completion.Source: §2 · p. 2, Figure 2 · p. 3
3. Method
ControlRoom3D uses a user-defined semantic proxy room and style description to guide generation of a globally plausible 3D room mesh. Its pipeline renders proxy controls, generates a coherent panorama, aligns depth to the proxy, and completes the mesh.
3D Semantic Proxy Room
- The proxy room encodes a rough layout with semantic bounding boxes and style text, from which semantic, instance, and near/far depth maps are rendered for control.These maps provide geometric and semantic guidance for generating room meshes aligned with the proxy.Source: §3 · p. 3, Figure 3 · p. 3
Guided Panorama Generation
- The method creates a coherent 360° room panorama by dividing the semantic panorama into eight 90°-FOV views, each rotated horizontally by 45°, using proxy-room semantic boxes and homography-linked correspondence-aware attention.The latent diffusion model also uses semantic and depth adapters fine-tuned on semantic bounding boxes.Source: §3 · p. 4
Geometry Alignment
- A pre-trained metric depth estimator is optimized to fit proxy near/far bounds, while SAM instance masks and normal preservation help retain object shapes.For walls, ceiling, and floor, the near and far depth values are equal; the normal loss compares current normals with the estimator's original normals.Source: §3 · pp. 4–5, Figure 5 · p. 4
- Aligned depth frames are converted to rotation-invariant distances and combined through spherical projection and pyramid blending into an initial 3D representation.This avoids visible seams that can arise when individually optimized frame depths are directly unprojected.Source: §3 · p. 5
Mesh Cleaning
- Poisson reconstruction creates a watertight mesh, then low-density vertices and small disconnected components are removed before completion.The mesh is filtered using vertex-density and component-size thresholds.Source: §3 · p. 5, Figure 6 · p. 5
Mesh Completion
- Completion renders the mesh from favorable viewpoints, uses Telea inpainting for small gaps and semantic/depth-guided diffusion for larger holes, then integrates the new content with geometry alignment.The process repeats across selected viewpoints to fill missing regions and align novel content with existing 3D structures.Source: §3 · p. 5, Figure 7 · p. 5
4. Experiments
The experiments compare ControlRoom3D with adapted room-generation baselines and evaluate its components. Results highlight improvements in room plausibility and proxy alignment, alongside contributions from adapter fine-tuning, panorama generation, and mesh completion.
4. Experiments
- The evaluation covers 12 room styles and 175 scenes, with 12 participants rating 1,120 data points; baselines are adapted Text2Room and MVDiffusion.Participants rate 3DS, proxy alignment (PA), layout plausibility (LP), and perceptual quality (PQ); the study also calculates CLIP and Inception scores.Source: §4 · p. 6, §4.1 · p. 6
4.1. Comparison with State-of-the-Art Methods
- ControlRoom3D reports +1.5 LP, +1.7 3DS, +1.2 PQ, and +2.8 CS in its quantitative comparison with related methods.Table 1 reports CLIP Score (CS), Inception Score (IS), 3D Structure Completeness (3DS), Layout Plausibility (LP), and Perceptual Quality (PQ).Source: Table 1 · p. 6, §4.1 · p. 6
4.2. Analysis Experiments
- Table 2 ablates the proxy room, geometry alignment, panorama generation, and mesh completion; completion adds +0.54 3DS and reaches 4.23 PQ.Mesh completion uses proxy-derived controls to inpaint regions unobserved after panorama generation.Source: Table 2 · p. 6, §4.2 · pp. 6, 9
4.1. Comparison with State-of-the-Art Methods
- Inception Score can favor scene diversity even when room layouts are implausible.The authors note that text-only conditioning can produce repeated prominent objects, such as four beds in one bedroom.Source: §4.1 · p. 6
4.2. Analysis Experiments
- Fine-tuning adapters on HyperSim-derived 3D bounding-box data improves their ability to generate objects that fit their designated boxes.Adapters trained only on pixel-precise masks tend to generate box-shaped objects that fill the boxes.Source: §4.2 · p. 8
- Panorama generation supports consistent scene style, while 2D semantic and depth guidance yields more plausible layouts than text-only panorama generation.Participants preferred the panorama module over incremental inpainting across all three evaluated metrics.Source: §4.2 · p. 8, Figure 10 · p. 8
- Iterative geometry alignment with proxy-room boxes improves proxy alignment by +2.32 PA over unaligned estimated depth.The authors describe single-image scale estimation as ill-posed and report implausible geometry without iterative alignment.Source: §4.2 · p. 8
- SAM masks sharpen object boundaries, while normal preservation counters surface distortion from fitting depth within proxy boxes.The depth alignment loss can distort object surfaces as it fits them to their bounding boxes.Source: §4.2 · pp. 8–9
5. Conclusion
ControlRoom3D generates room meshes from a style description and a user-defined layout represented by semantic 3D boxes. Its guidance and completion components target style consistency and plausible geometry.
5. Conclusion
- ControlRoom3D generates high-quality room meshes from a room-style description and a user-defined layout outlined by 3D semantic bounding boxes.The rendered proxy room provides geometric and semantic information for guiding texture and geometry generation.Source: §5 · p. 9
- Guided panorama generation and geometry alignment address style consistency and plausible geometry, while mesh post-processing and completion improve final generation quality.Source: §5 · p. 9
- The authors aim to simplify personalized 3D scene creation for end-users without expert 3D modeling knowledge.Source: §5 · p. 9
6. Adapter Fine-tuning on HyperSim
The HyperSim-derived training data and filtering target the mismatch between pixel-precise training masks and rough rendered 3D-box controls. Fine-tuning on this data improves object fit within the designated boxes.
6. Adapter Fine-tuning on HyperSim
- HyperSim images provide camera poses and aligned 3D semantic boxes, which are projected into 2D semantic and depth maps for adapter fine-tuning.Source: §6 · p. 12
- Filtering excludes images with camera roll above ±8.6°, a semantic class covering over half the image, or maximum depth above 15 meters.The filters target the ground-parallel camera setup, overly close views, and the focus on bounded indoor environments.Source: §6 · p. 12
- Semantic and depth adapters are fine-tuned independently for 5,000 iterations on 8 A100 GPUs, using batch size 8 and learning rate 1e-7.Source: §6 · p. 12
- Adapters trained only on pixel-precise masks tend to fill boxes with box-shaped objects, whereas HyperSim fine-tuning improves object fit within the boxes.Source: §6 · p. 12
7. Details on User Study
The paper reports two user studies: one compares ControlRoom3D with related methods, and another evaluates the influence of its technical components. Participants rate rendered room videos on specified quality dimensions.
7. Details on User Study
- Twelve participants rate scenes on a 1–5 scale in two studies: a comparison with Text2Room and adapted MVDiffusion, and a component ablation.Source: §7 · p. 12
- The comparison study rates 3D structure completeness, layout plausibility, and overall perceptual quality; the ablation study substitutes proxy alignment for layout plausibility.The ablation videos also superimpose proxy-room boxes to show the guidance.Source: §7 · p. 12
- For component evaluation, proxy overlays accompany the videos, and the study examines visual effects including layout guidance, geometry alignment, and completion of unobserved regions.Source: §7 · p. 12, Figure 15 · p. 13
- Each generated-room video lasts 25 seconds and shows objects from multiple challenging angles; participants rate each scene individually.Source: Figure 14 · p. 13
8. Further Qualitative Results
The section presents further qualitative examples of ControlRoom3D outputs and visualizations of its ablation study. The examples show generated rooms conditioned on a user-defined layout and textual style description.
8. Further Qualitative Results
- Additional examples and videos cover room meshes of various room types and styles.The supplementary video also explains a main technical component of ControlRoom3D.Source: §8 · p. 13
- The ablation study includes qualitative visualizations in Figure 15 and videos of an exemplary room for each ablation experiment.The visual labels include rendered view, geometry, and scene layout.Source: §8 · pp. 13–14
- Figure 16 shows colored geometry renderings of rooms generated from a user-defined layout and textual style description.The caption describes the resulting geometries and textures as convincing.Source: Figure 16 · p. 14