PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models

Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, Matthias Nießner

arXiv:2303.11989v2cs.CV

TL;DR

Room-scale text-to-3D generation is difficult because existing methods do not readily produce coherent scenes with walls, floors, furniture, and explicit mesh geometry. Text2Room lifts text-to-image outputs into 3D through iterative viewpoint selection, inpainting, depth alignment, and mesh fusion, producing complete textured indoor meshes that are reported to outperform baselines qualitatively and quantitatively.

  • Problem

    Existing text-to-3D methods do not easily generate dense, coherent room-scale scenes with structural elements and mesh geometry.

  • Method

    Text2Room iteratively fuses frames from a fixed 2D text-to-image model into a mesh using two-stage viewpoint selection, inpainting, monocular depth estimation, and depth alignment.

  • Results

    Text2Room produces high-detail textured geometry fused into complete, hole-free scene meshes containing floors, walls, ceilings, and distributed 3D objects, with the highest scores across reported image-based metrics.

  • Takeaways & Limitations

    The method demonstrates that text alone can generate arbitrary indoor scene meshes renderable with classical rasterization pipelines.

  • Takeaways & Limitations

    The method can leave distortions or over-smoothed regions, may incompletely fill holes, and bakes diffusion-generated lighting into the scene representation.

Abstract

from arXiv · show

We present Text2Room, a method for generating room-scale textured 3D meshes from a given text prompt as input. To this end, we leverage pre-trained 2D text-to-image models to synthesize a sequence of images from different poses. In order to lift these outputs into a consistent 3D scene representation, we combine monocular depth estimation with a text-conditioned inpainting model. The core idea of our approach is a tailored viewpoint selection such that the content of each image can be fused into a seamless, textured 3D mesh. More specifically, we propose a continuous alignment strategy that iteratively fuses scene frames with the existing geometry to create a seamless mesh. Unlike existing works that focus on generating single objects or zoom-out trajectories from text, our method generates complete 3D scenes with multiple objects and explicit 3D geometry. We evaluate our approach using qualitative and quantitative metrics, demonstrating it as the first method to generate room-scale 3D geometry with compelling textures from only text as input.

1. Introduction

Text2Room addresses the difficulty of extending text-to-3D generation to coherent, room-scale meshes by lifting 2D text-to-image outputs into 3D. It iteratively generates and fuses scene content using inpainting, monocular depth estimation, viewpoint selection, and depth alignment.

  • Research gap: Existing text-to-3D methods lack the generality of 2D text-to-image models and often rely on limited 3D training data.Direct 3D supervision is frequently restricted to simple-shape datasets, while 2D-guided approaches optimize in the image domain.
  • Research gap: Prior approaches can generate arbitrary neural shapes but do not easily produce dense, coherent room-scale structure and texture.Large scenes must cover outward-facing viewpoints while representing walls, floors, and furniture.
  • Approach: Text2Room begins with a text-generated image, backprojects it using monocular depth, and iteratively renders novel views, inpaints holes, and fuses new content into the mesh.The method uses off-the-shelf 2D text-to-image models rather than requiring a dedicated 3D generator.
  • Approach: A two-stage viewpoint strategy first covers scene layout and furniture, then adaptively closes remaining holes while depth alignment and mesh filtering support consistent fusion.The design explicitly addresses both camera-pose selection and merging generated content with existing geometry.

2. Related Work

Related work spans text-based image generation, text-to-3D modeling, and 3D-consistent view synthesis. Text2Room uses fixed text-to-image models to construct complete scene meshes rather than single objects or forward-facing view sequences.

  • Text-based generation: Large-scale image-text datasets and scalable generative architectures have enabled text-conditioned synthesis of novel images.Diffusion models improved image synthesis through latent-space denoising, faster sampling, and stronger guidance.
  • Text-based generation: Text-to-image systems such as Stable Diffusion, Imagen, GLIDE, and DALL·E 2 produce diverse, high-fidelity, controllable outputs.Text-based generation has also extended to audio, video, and 4D fields.
  • Text-to-3D: Supervised text-to-3D methods remain constrained by the lack of large-scale aligned text-and-3D datasets.Alternative methods use CLIP-guided image-domain optimization, object alignment, or text-guided refinement of existing 3D inputs.
  • Text-to-3D: Recent approaches combine large text-to-image diffusion models with neural radiance fields to generate 3D objects without training.Text2Room instead uses a fixed text-to-image model to extract meshes containing entire scenes, multiple objects, and structural elements.
  • 3D-consistent view synthesis: Single-image novel-view and perpetual-view methods synthesize scenes along trajectories, while related text-based work focuses on this forward-facing setting.Text2Room targets complete room-scale scenes rather than only trajectory-based view generation.

3. Method

Text2Room continuously builds a textured scene mesh by generating and fusing frames from selected camera poses. Its pipeline combines rendering, text-conditioned RGB and depth completion, depth alignment, triangulation, filtering, and two-stage viewpoint selection.

  • Method overview: The method first generates scene layout and objects, then closes remaining holes through a two-stage viewpoint selection scheme.Each pose updates the mesh through iterative scene generation and frame-to-geometry alignment.
  • Iterative scene generation: For every new pose, the method renders partial RGB and depth, completes both with inpainting models, aligns depth, filters geometry, and fuses the next mesh patch.This follows a render-refine-repeat process over successive viewpoints.
  • Iterative scene generation: The rendered image, depth, and unobserved-content mask are produced from the current mesh and camera pose before text-conditioned inpainting fills missing RGB content.The fixed text-to-image model uses the rendered image, mask, and prompt as inputs.
  • Depth alignment: Monocular depth predictions are aligned to rendered depth because inconsistent scale across viewpoints otherwise creates hard cuts and geometric discontinuities.The method optimizes scale and shift parameters in masked least squares before extracting aligned depth.
  • Mesh fusion: The method backprojects aligned pixels into a world-space point cloud, triangulates neighboring pixels, and removes faces with overly long edges or grazing normals.These filters reduce stretched geometry caused by noisy depth estimates.
  • Mesh fusion: Remaining faces are stitched to existing geometry using boundary vertices, while a second viewpoint stage samples poses that observe unobserved pixels and Poisson reconstruction closes residual holes.The final reconstruction produces a watertight mesh renderable with classical rasterization.

4. Results

Text2Room produces complete, detailed room-scale meshes by combining iterative viewpoint selection, depth alignment, mesh fusion, and completion. Results show complete geometry, high-detail textures, strong quantitative performance, and controllable multi-prompt scene composition, while retaining several artifact and layout limitations.

  • Evaluation setup: The evaluation reports CLIP Score, Inception Score, Perceptual Quality, and 3D Structure Completeness for generated scenes.The user study includes n=61 users rating Perceptual Quality and 3D Structure Completeness on a 1–5 scale.
  • Qualitative results: Text2Room creates high-detail textures fused into complete meshes without holes, including floors, walls, ceilings, and distributed 3D object geometry.Baselines lack complete structural geometry, suffer from occlusion-induced holes, or fail to produce immersive scenes with floors and walls.
  • Quantitative results: The method achieves the highest scores because its renderings remain complete from arbitrary novel poses, satisfy text prompts, and contain high-resolution image features.Users prefer the method, highlighting the quality of its complete geometry and RGB texture.
  • Ablations: Depth alignment enables seamless fusion, while edge and surface-normal thresholds remove stretched faces before later viewpoints complete the geometry.Without depth alignment, scene parts disconnect; without the filters, faces stretch unnaturally.
  • Ablations: Two-stage viewpoint selection first generates layout and furniture, then samples additional poses to complete previously unobserved regions and produce a watertight mesh.Predefined trajectories build the main scene, while adaptive completion addresses holes that cannot be covered a priori.
  • Spatially varying scene generation: Separate prompts for different camera poses allow rooms to combine multiple spatial descriptions, although camera poses only partially control layout extent.Generated chunks can have larger or smaller extents, limiting precise layout control.

5. Conclusion

Text2Room generates complete, textured indoor 3D meshes from text by fusing images from a 2D text-to-image model into explicit geometry. The method targets large-scale 3D asset creation with reduced modeling expertise.

  • Text2Room generates textured 3D meshes of complete indoor scenes from only text input.
  • The method sequences 2D text-to-image outputs from different poses and iteratively fuses them into a joint mesh.
  • Tailored viewpoint selection first creates scene layout and objects, then closes remaining holes in the 3D geometry.
  • The generated meshes represent arbitrary indoor scenes and can be rendered with classical rasterization pipelines.
  • Text-only mesh generation reduces the expertise required to model and design large-scale 3D assets.

C. Limitations

The method can produce detailed, geometrically consistent room scenes, but completion and fusion remain vulnerable to holes, smoothing, and distorted geometry. A user study evaluates perceptual quality and structure completeness across multiple scene views.

  • The method can generate highly detailed room geometry with consistent 3D structure, but it can still fail under certain conditions.
  • Completion may leave holes near walls when suitable camera views are unavailable, with Poisson reconstruction potentially producing overly smoothed geometry.
  • Imperfect depth estimation and alignment can leave stretched-out faces that accumulate into unusual room shapes.
  • Filtering uses fixed thresholds δsn=0.1 and δedge=0.1, but lowering them could remove valid geometry and require more hole inpainting.
  • A user study with n=61 users scores Perceptual Quality and 3D Structure Completeness from 1−5 using multiple images per scene.

E. Additional Implementation Details

The supplied passage only states that additional implementation details are provided in the following subsections.

  • Additional implementation details are provided in the following subsections.

E.1. Importance of Predefined Trajectories

Predefined trajectories are designed to build scene layout and furniture by expanding from mostly unobserved content into each generated scene chunk. Deviating from this construction principle can produce locally plausible but globally inconsistent geometry.

  • E.1. Importance of Predefined Trajectories: The first trajectory stage creates scene layout and furniture by iteratively generating the scene from multiple predefined trajectories.
  • E.1. Importance of Predefined Trajectories: Each trajectory starts by viewing mostly unobserved content, then translates and rotates into the resulting chunk to refine its structure.
  • E.1. Importance of Predefined Trajectories: This construction principle creates mesh patches with convincing 3D structure.
  • E.1. Importance of Predefined Trajectories: Trajectories that violate the principle can generate locally reasonable content that accumulates into inconsistent global structure.
  • E.1. Importance of Predefined Trajectories: The completion stage may fail to inpaint holes when no suitable camera pose can be sampled, while Poisson reconstruction closes them with potentially smoothed geometry.
  • E.1. Importance of Predefined Trajectories: The user study presents multiple scene images from different angles and asks users to rate structure completeness and perceptual quality on a 1−5 scale.

E.2. Effect of Depth Smoothing in Alignment

The method aligns inpainted depth with rendered depth before fusion and smooths the aligned result at image boundaries. This reduces discontinuities and blocky transitions between existing and newly generated mesh content.

  • E.2. Effect of Depth Smoothing in Alignment: Depth alignment combines monocularly inpainted depth with rendered depth using least-squares alignment before mesh fusion.Directly using predicted depth produces unaligned mesh patches.
  • E.2. Effect of Depth Smoothing in Alignment: A 5 × 5 Gaussian blur at rendered–predicted depth boundaries smooths remaining discontinuity artifacts between old and new content.The smoothing is applied at image edges between rendered and predicted depth.
  • E.2. Effect of Depth Smoothing in Alignment: Depth smoothing further reduces sharp borders between objects, improving overall alignment.

E.3. Importance of Mask Dilation in Completion

Completion fills remaining mesh holes by selecting camera poses that view unobserved regions and applying the iterative scene-generation procedure. The first-stage layout determines whether these holes are small or large, while predefined trajectories help produce consistent scene patches.

  • E.3. Importance of Mask Dilation in Completion: The second completion stage selects camera poses that view remaining holes and generates content to fill them.The same iterative scene-generation scheme is used for hole completion.
  • E.3. Importance of Mask Dilation in Completion: Unobserved regions may have arbitrarily small or large sizes depending on how the scene layout is generated in the first stage.
  • E.3. Importance of Mask Dilation in Completion: Predefined trajectories outline the next scene chunk before additional poses translate and rotate into it, producing a consistent mesh patch for fusion.Sampling suboptimal trajectories causes the results to degenerate.

F. Additional Discussion on Related Methods and Baselines

The discussion contrasts Text2Room with object-centric, outpainting, panorama-based, and large-scene baselines. Text2Room uses tailored trajectories and adaptive hole completion to create complete, arbitrarily viewable meshes without supervised 3D training.

  • F. Additional Discussion on Related Methods and Baselines: No direct baseline generates textured 3D room geometry from text, so the method is compared with four related approaches lacking 3D-dataset supervision.
  • F. Additional Discussion on Related Methods and Baselines: PureClipNeRF represents object-centric text-to-3D methods that sample inward-facing poses on a hemisphere for iterative object optimization.
  • F. Additional Discussion on Related Methods and Baselines: Outpainting enlarges a single generated image, then uses depth estimation and triangulation to lift the result into a 3D mesh.
  • F. Additional Discussion on Related Methods and Baselines: Text2Light and Blockade generate RGB panoramas, but a panorama captures one viewpoint and cannot directly simulate translation or rotation around objects.Blockade outputs expressive RGB panoramas that are subsequently used for mesh extraction.
  • F. Additional Discussion on Related Methods and Baselines: Depth alignment smooths transitions between rendered and predicted content, while mask-based completion fills unobserved regions from selected camera poses.
  • F. Additional Discussion on Related Methods and Baselines: GAUDI generates text-conditioned large-scale scenes in a NeRF representation but requires ground-truth camera trajectories from 3D datasets.Its dataset domain is synthetic, low-resolution, and limited in size and diversity.
  • F. Additional Discussion on Related Methods and Baselines: Text2Room creates layout and furniture from predefined trajectories, completes holes with additional poses, and renders the resulting mesh from arbitrary viewpoints.The method directly lifts 2D text-to-image outputs without supervised training from 3D datasets.

G. Additional Qualitative Results

Additional examples show Text2Room generating realistic textured meshes for varied indoor prompts. The results include living rooms, offices, libraries, and bathrooms, with both color and shaded-geometry renderings.

  • G. Additional Qualitative Results: The qualitative examples include a rustic farmhouse living room with a stone fireplace, wood, leather, and wool.
  • G. Additional Qualitative Results: The examples also include a small office with a chair, desk, and monitors and a library with bookshelves, tables, chairs, and reading lamps.
  • G. Additional Qualitative Results: The method produces color and shaded-geometry renderings for realistic indoor scenes satisfying text descriptions.
  • G. Additional Qualitative Results: A large bathroom example contains a shower, bathtub, and cozy wellness area.
Loading 2303.11989v2…