PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields

Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, Jing Liao

arXiv:2305.11588v2cs.CVcs.GR

TL;DR

Text-driven 3D generation remains challenging because paired text–3D data is impractical to obtain and existing methods are limited. Text2NeRF combines NeRF with diffusion and depth priors plus progressive view updating, surpassing baselines in rendered-scene quality and text relevance.

  • Problem

    Text-to-3D generation remains challenging because large-scale paired text–3D data is impractical to acquire, while prior methods have limited scene-generation realism and complexity.

  • Method

    Text2NeRF optimizes a NeRF from diffusion-derived content and monocular-depth geometric priors, using progressive inpainting, support sets, and depth-aware losses for view-consistent scenes.

  • Results

    Text2NeRF surpasses baseline methods, producing higher-quality 3D scenes with lower BRISQUE and NIQE values and higher CLIP scores.

  • Takeaways & Limitations

    The framework supports zero-shot generation of diverse indoor and outdoor 3D scenes with complicated structures, high-fidelity textures, and view-consistent novel content from natural-language prompts.

Abstract

from arXiv · show

Text-driven 3D scene generation is widely applicable to video gaming, film industry, and metaverse applications that have a large demand for 3D scenes. However, existing text-to-3D generation methods are limited to producing 3D objects with simple geometries and dreamlike styles that lack realism. In this work, we present Text2NeRF, which is able to generate a wide range of 3D scenes with complicated geometric structures and high-fidelity textures purely from a text prompt. To this end, we adopt NeRF as the 3D representation and leverage a pre-trained text-to-image diffusion model to constrain the 3D reconstruction of the NeRF to reflect the scene description. Specifically, we employ the diffusion model to infer the text-related image as the content prior and use a monocular depth estimation method to offer the geometric prior. Both content and geometric priors are utilized to update the NeRF model. To guarantee textured and geometric consistency between different views, we introduce a progressive scene inpainting and updating strategy for novel view synthesis of the scene. Our method requires no additional training data but only a natural language description of the scene as the input. Extensive experiments demonstrate that our Text2NeRF outperforms existing methods in producing photo-realistic, multi-view consistent, and diverse 3D scenes from a variety of natural language prompts. Our code is available at https://github.com/eckertzhang/Text2NeRF.

I. INTRODUCTION

Text2NeRF addresses the difficulty of generating realistic, geometrically complex 3D scenes from text without paired 3D data. It combines NeRF, diffusion-derived image priors, depth priors, and progressive view updating to produce diverse, view-consistent indoor and outdoor scenes.

  • I. INTRODUCTION: Existing text-to-3D methods are limited by scarce paired text–3D data and often produce simple geometries with dreamlike, low-detail styles.Image-model priors can constrain high-level semantics while neglecting low-level details, limiting realism.
  • I. INTRODUCTION: Text2NeRF uses diffusion-generated images as content priors and monocular depth estimates as geometric priors to reconstruct detailed, realistic scenes with NeRF.NeRF is selected for fine-grained, photorealistic scene representation and to reduce artifacts associated with triangular meshes at depth discontinuities.
  • I. INTRODUCTION: The progressive inpainting and updating strategy expands scenes view by view, carrying newly generated regions into subsequent views to maintain continuity and view consistency.The method updates the rendered NeRF after each view so the same area is not repeatedly expanded.
  • I. INTRODUCTION: Support sets, depth loss, and two-stage depth alignment provide multi-view constraints and stabilize NeRF optimization during single-view, view-by-view updating.These components address overfitting, geometric ambiguity, independently estimated depth maps, and misalignment across overlapping views.
  • I. INTRODUCTION: Text2NeRF combines NeRF with a pre-trained text-to-image diffusion model and supports zero-shot generation of diverse indoor and outdoor scenes from natural-language prompts.The framework uses image-level content priors rather than requiring additional 3D supervision or multi-view training data.

A. Text-Driven 3D Generation

Text2NeRF initializes a NeRF scene from a diffusion-generated image and monocular depth, then creates support views and progressively expands the scene through novel-view synthesis. Its implicit representation is designed for complex geometry and supports view-consistent reconstruction.

  • A. Text-Driven 3D Generation: Compared with category-specific or mesh-based approaches, the framework targets diverse text-driven scenes and avoids explicit-mesh limitations associated with outdoor representation and fusion artifacts.The related methods cited include category-specific shape generation, text-guided NeRF or mesh methods, and single-image novel-view systems with limited view ranges.
  • A. Text-Driven 3D Generation: Text2NeRF generates an initial image and depth map, warps them with DIBR into support views, and uses the resulting support set to initialize NeRF.The support set supplies multi-view constraints that reduce overfitting and geometric ambiguity from single-view training.
  • A. Text-Driven 3D Generation: The scene is expanded view by view by rendering a novel view, completing missing regions with diffusion inpainting, and adding the completed view's support set to NeRF training.This progressive procedure is the basis of the PIU strategy.
  • A. Text-Driven 3D Generation: NeRF represents the scene through volume-rendered color and depth, enabling reconstruction of arbitrarily complex geometry with implicit 3D structure.The method omits view direction when predicting radiance to avoid view-dependent specularity and adds depth constraints for depth-aware optimization.

B. Text-Driven Inpainting

Text2NeRF completes missing regions in rendered novel views with a text-conditioned diffusion model and updates the NeRF using the resulting views. Because inpainting is stochastic, it samples multiple candidates and selects the one most semantically similar to the initial view.

  • B. Text-Driven Inpainting: For each rendered novel view, Text2NeRF computes a missing-region mask and feeds the rendered image, mask, and text prompt to a diffusion model for inpainting.Known views are warped into the target view to determine which regions are missing.
  • B. Text-Driven Inpainting: Because diffusion inpainting is stochastic, multiple completions are generated and the candidate most similar to the initial view in CLIP semantic space is selected.Candidate selection is used to improve consistency with the established scene content.

ECLIP (I0) , ECLIP

For each novel view, the method generates multiple diffusion-based inpainting candidates and estimates depth, then uses overlapping regions to align the estimated geometry with NeRF-rendered depth.

  • The method generates 30 inpainting results as candidates for each view to be completed.
  • Estimated depth for an inpainted novel view cannot directly supervise NeRF because it may conflict with depth rendered from previously known views.
  • The overlap between rendered and estimated depth is used to align the novel view's geometry with known NeRF depth constraints.

C. Depth Alignment

The two-stage depth alignment strategy addresses scale and value differences between estimated novel-view depth and NeRF-rendered depth before updating the radiance field.

  • Estimated depth can be inconsistent with NeRF-rendered depth because independently predicted maps lack geometric constraints in overlapping regions.
  • The progressive strategy uses NeRF-rendered depth constrained by previous views as the reference for alignment.
  • Two-stage depth alignment first compensates for mean scale and value differences, then fine-tunes a pretrained network for local alignment.
  • Global alignment computes average scale and depth offset from randomly selected pixel pairs in overlapping regions.
  • Because scale and value differences are not fully linear, a pixel-to-pixel network is optimized on overlapping regions to produce locally aligned depth.

D. Progressive Inpainting and Updating

Progressive inpainting updates the radiance field after each view so previously generated content constrains subsequent views and prevents repeated, inconsistent inpainting.

  • The progressive strategy updates the radiance field after every inpainting step, unlike independent per-view inpainting without 3D constraints.
  • The algorithm processes views sequentially, renders masks and depth, constructs support sets, and updates the NeRF model after each view.
  • Each newly inpainted view is added to the updated-view set and used to construct support views for the next NeRF update.
  • Previously inpainted content is reflected in subsequent renderings and treated as known, so it is not inpainted again.

E. Training and Implementation Details

Text2NeRF optimizes its radiance field with RGB, depth, and transmittance losses, using diffusion-generated content and monocular depth estimates in implementation.

  • The total training objective combines RGB, depth, and transmittance losses: Ltotal = LRGB + λdLDepth + λtLT.
  • The RGB loss matches rendered pixel colors to colors generated by the diffusion model, while the depth loss matches rendered depth to aligned estimated depth.
  • The transmittance loss encourages empty density before rays reach the expected aligned depth.
  • The loss weights λd and λt balance the depth and transmittance terms against the RGB objective.
  • The implementation uses TensoRF, Stable Diffusion 2.0 for scene content, and boosting monocular depth estimation with a pretrained LeReS model.

IV. EXPERIMENTS

The experiments evaluate Text2NeRF on diverse text prompts, comparing it with baseline methods and analyzing its major components through ablation studies.

  • The evaluation compares Text2NeRF with state-of-the-art baselines across varied prompts and includes ablations of its major components.The experiments target photo-realistic indoor and outdoor 3D scene generation.

A. Setup

The setup compares Text2NeRF with seven baselines spanning semantic- and image-guided generation, using qualitative, quantitative, and 360-degree scene evaluations.

  • Seven baselines cover high-level semantic-prior and low-level image-prior methods, including NeRF, polygon-mesh, and point-cloud representations.DreamFusion is replaced with a PyTorch implementation using Stable Diffusion because Imagen is unavailable.
  • Qualitative comparisons show two rendered views for each generated scene and method across different text prompts.Additional video results are provided in the supplementary material.
  • Table I compares guidance type, 3D representation, optimization time, and BRISQUE, NIQE, and CLIP similarity scores.Lower BRISQUE and NIQE and higher CLIP similarity are treated as better outcomes.
  • The evaluation also includes 360-degree scenes generated by Text2NeRF.Video results are provided in the supplementary material.

B. Comparisons

Text2NeRF produces higher-quality, text-relevant scenes than the baselines, while supporting complex indoor and outdoor geometry, diverse outputs, and artistic styles.

  • Text2NeRF achieves lower BRISQUE and NIQE values and a higher CLIP score than the baselines, indicating higher-quality and more text-relevant scenes.The paper reports this conclusion from quantitative and qualitative comparisons across varied prompts.
  • Semantic-prior methods struggle with complex scenes, while DreamFusion-Scene still lacks detailed structures and photorealistic textures.The paper attributes this limitation to insufficient low-level constraints from deep semantic priors.
  • Image-prior methods produce photorealistic scenes, but limited or prompt-independent scene extrapolation can cause missing, stretched, incoherent, or blurry content.3DP is sensitive to depth discontinuities, while PixelSynth uses prompt-independent inpainting.
  • Text2NeRF generates indoor and outdoor scenes with complex structures and more photorealistic details in depth-discontinuous regions than Text2Room.Text2Room often produces stretched geometry in those regions and struggles with certain outdoor scenes.
  • Text2NeRF generates diverse scenes from the same prompt and supports scenes rendered in artistic styles.The examples include multiple prompt pairs with diverse outputs and artistic-style results.

C. Ablation Studies

The ablations show that PIU, support sets, depth supervision, and two-stage depth alignment each address distinct sources of view inconsistency, overfitting, geometric ambiguity, or depth discontinuity.

  • Ablation on PIU Strategy: PIU produces more view-consistent scenes with distinct textures than independently inpainting each view, which introduces artifacts in overlapping regions.The strategy updates the scene view by view, allowing newly generated content to constrain subsequent views.
  • Support Set: Support sets prevent single-view NeRF overfitting from producing blurry novel-view renderings and improve the quality of subsequent inpainting.Without support-set constraints, the NeRF fits the training view but fails to render a clear inpainting view.
  • Support Set: Using 8 warping views and shift distance ζ = 0.2 balances support-set computation and training benefit while achieving better performance than other tested shift distances.The benefit of additional warping views tends to saturate as their number increases.
  • Ablation on Depth Loss: L2 depth loss constrains near and far depth values more strictly than no depth loss, GNLL, or SSI losses.Without depth guidance, novel views can have implausible geometry and ambiguous depth values.
  • Two-Stage Depth Alignment: Two-stage depth alignment yields smoother transitions at merged-depth demarcation lines than using no alignment or only one alignment stage.Global alignment reduces overall depth differences, while local alignment further improves unaligned regions; both are needed to eliminate the differences more effectively.

V. CONCLUSION

Text2NeRF generates diverse, photorealistic 3D scenes with complex geometric structures and fine-fidelity textures from text, using diffusion and depth priors with progressive NeRF updating.

  • V. CONCLUSION: Text2NeRF combines diffusion-based content priors, monocular depth priors, NeRF optimization, support sets, depth-aware losses, and two-stage alignment for text-driven scene generation.The PIU strategy updates the scene progressively while support sets provide multi-view constraints during single-view NeRF training.
  • V. CONCLUSION: The method generates photo-realistic, diverse 3D scenes with complex geometric structures and fine-fidelity textures.
  • V. CONCLUSION: The conclusion identifies Text2NeRF as a text-driven framework for generating broad scene content rather than only simple objects.
Loading 2305.11588v2…