PaperlayerView exact revision on arXiv

Source-linked AI summary

RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D

Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, Xiaoguang Han

arXiv:2311.16918v2cs.CVcs.AI

TL;DR

Text-to-3D generation lacks robust geometric priors and struggles to disentangle materials from lighting when lifting 2D diffusion. RichDreamer learns Normal-Depth diffusion from large-scale real-world data and adds albedo diffusion for appearance modeling. Integrated into existing pipelines, the models achieve state-of-the-art geometry and appearance results.

  • Problem

    Text-to-3D methods lack robust geometric priors, while 2D RGB diffusion introduces a normal-map distribution discrepancy and material-lighting entanglement.

    Source: §1 · p. 2, §3.3 · p. 5

  • Method

    RichDreamer trains a generalizable Normal-Depth diffusion model on LAION-derived priors and adds depth-conditioned albedo diffusion for decoupled geometry and appearance modeling.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    The integrated models achieve state-of-the-art text-to-3D results in both geometry and appearance modeling.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Normal-depth priors and albedo constraints provide a generalizable route to detail-rich, geometry-first text-to-3D generation within NeRF and DMTet pipelines.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Lifting 2D diffusion for 3D generation is a challenging problem due to the lack of geometric prior and the complex entanglement of materials and lighting in natural images. Existing methods have shown promise by first creating the geometry through score-distillation sampling (SDS) applied to rendered surface normals, followed by appearance modeling. However, relying on a 2D RGB diffusion model to optimize surface normals is suboptimal due to the distribution discrepancy between natural images and normals maps, leading to instability in optimization. In this paper, recognizing that the normal and depth information effectively describe scene geometry and be automatically estimated from images, we propose to learn a generalizable Normal-Depth diffusion model for 3D generation. We achieve this by training on the large-scale LAION dataset together with the generalizable image-to-depth and normal prior models. In an attempt to alleviate the mixed illumination effects in the generated materials, we introduce an albedo diffusion model to impose data-driven constraints on the albedo component. Our experiments show that when integrated into existing text-to-3D pipelines, our models significantly enhance the detail richness, achieving state-of-the-art results. Our project page is https://aigc3d.github.io/richdreamer/.

1. Introduction

Text-to-3D methods face limited 3D-data generalization, missing multi-view constraints, and entangled geometry, texture, and lighting. RichDreamer addresses these challenges with generalizable Normal-Depth and albedo diffusion models, achieving state-of-the-art geometry and appearance results when integrated into existing pipelines.

Motivation

  • Lifting 2D diffusion into 3D lacks multi-view constraints and entangles surface geometry, texture, and lighting.These issues motivate decoupling geometry generation from subsequent texture generation.Source: §1 · p. 2
  • Optimizing normal maps with 2D RGB diffusion creates a distribution discrepancy that compromises geometric quality and destabilizes optimization.This limitation is identified in the decoupled approach used by Fantasia3D.Source: §1 · p. 2

Proposed Approach

  • RichDreamer learns a generalizable Normal-Depth diffusion model from LAION using automatically estimated normal and depth priors, then fine-tunes it on synthetic Objaverse data.The real-world pretraining is intended to preserve diverse normal-depth distributions after synthetic fine-tuning.Source: §1 · p. 2
  • An albedo diffusion model provides data-driven constraints that improve separation of reflectance from illumination effects such as shadows and specular highlights.The model targets ambiguity in appearance decomposition rather than directly modeling lighting artifacts.Source: §1 · p. 2

Results

  • Integrating the proposed models into existing text-to-3D pipelines yields state-of-the-art geometry and appearance modeling results.The paper presents this as a combined contribution alongside strong generalization and improved reflectance-illumination separation.Source: §1 · p. 2

2. Related Work

3D generative models remain constrained by limited 3D-data scale, while diffusion-based geometric priors increasingly target multi-view consistency and explicit geometry. RichDreamer uses normal-depth diffusion for decoupled geometry-first generation and integrates it with existing 3D representations.

3D Generative Models

  • Existing 3D generative models use voxels, point clouds, meshes, and implicit fields but mainly demonstrate generation within limited object categories.The restricted scale of available 3D data limits broader category coverage.Source: §2 · p. 2

RichDreamer

  • RichDreamer introduces Normal-Depth and albedo diffusion models for a pipeline that generates geometry first and appearance second.The overview describes compatibility with DMTet and NeRF representations.Source: Figure 2 · p. 3

3D Generative Models

  • Prior text-to-3D work improves quality through coarse-to-fine optimization, decoupled generation, new score distillation, optimization strategies, and parametric shape models.Other work accelerates optimization with efficient representations such as hashgrids and 3D Gaussian splatting.Source: §2 · p. 3

Geometry Priors

  • RGB-space diffusion methods address multi-view inconsistency but are less suitable when geometry must be generated before appearance.This mismatch motivates diffusion directly over geometric information for decoupled generation.Source: §2 · p. 3
  • LDM3D introduces RGB-D diffusion on LAION-400M, but the cited work does not validate it for text-to-3D generation.SweetDreamer instead uses a canonical coordinate map that implicitly requires aligned synthetic training objects.Source: §2 · p. 3

3. Method

The method decouples geometry generation from appearance modeling by introducing a Normal-Depth diffusion model for geometry and an albedo diffusion model for materials.

Overview

  • The pipeline first generates geometry and then models appearance, addressing the entanglement of geometry, texture, and lighting in RGB diffusion.Normal and depth are used as geometry representations, while albedo regularization targets material appearance.Source: §3 · pp. 3–4, §3.1 · p. 4

Normal-Depth Diffusion Model

  • The Normal-Depth diffusion model captures the joint distribution of normal and depth maps, combining macrostructure from depth with local surface details from normals.Its architecture adapts Stable Diffusion by extending the VAE from three to four channels for three normal channels and one depth channel.Source: §3.1 · p. 4

Training

  • The model is pretrained on LAION image-text pairs using estimated normal and depth maps, then fine-tuned on rendered Objaverse data with four-view camera conditioning.The LAION stage supports real-world diversity, while Objaverse provides ground-truth 3D object supervision.Source: §3.1 · p. 4

Geometry Generation

  • During geometry generation, the Normal-Depth model applies SDS to rendered normal-depth composites within Fantasia3D’s coarse-to-fine DMTet pipeline.The approach also supports NeRF representations because normal and depth maps can be obtained through volume rendering.Source: §3.2 · p. 5

Appearance Modeling

  • The albedo diffusion model regularizes appearance modeling by constraining albedo separately from lighting effects such as shadows and specular highlights.This addresses inaccuracies that arise when material components are optimized only from final rendered RGB colors.Source: §3.3 · p. 5

4. Experiments

The experiments evaluate geometry and textured-model generation against existing methods, then analyze the contributions of the Normal-Depth and albedo diffusion models. Results show strong geometry quality, competitive appearance quality, and improved structure, detail, generalization, and texture appearance under the proposed components.

Evaluation on Text-to-3D

  • The two method variants achieve the top two average geometry CLIP scores under uniform-albedo rendering, outperforming existing methods.Geometry is evaluated across 16 views for each of 113 objects using rendered shading images with uniform albedo.Source: §4.1 · p. 6
  • The variants achieve the second- and third-highest appearance CLIP scores, with ProlificDreamer slightly higher at 31.8022 versus 31.7099.Appearance is evaluated from textured-model renderings across 16 views; the difference may relate to ProlificDreamer’s additional LoRA fine-tuning.Source: §4.1 · p. 6, Table 1 · p. 7
  • In the user study, the NeRF variant received 75% and 70% of votes for best textured model and best geometry, while Sphere initialization exceeded 59% and 58%.Participants separately voted for the best textured model and best geometry after viewing rendered models and normal maps.Source: §4.1 · p. 7

Method Analysis

  • Using the Normal-Depth model alone robustly generates coherent geometry, while adding the SD model produces finer details and improved shape.The authors characterize Normal-Depth guidance as supporting overall structure and SD guidance as supporting surface details.Source: §4.2 · pp. 7–8
  • Fine-tuning the Normal-Depth model only on synthetic Objaverse data significantly deteriorates prompt-aligned generalization, supporting pre-training on LAION.The ablation compares the ND (w/o LAION)+SD baseline with the proposed pre-training strategy.Source: §4.2 · p. 8
  • The albedo diffusion model improves texture and appearance, while depth conditioning helps the generated texture align with the underlying geometry and reduces shadows and specular highlights.The relighting comparison reports more realistic results with albedo diffusion.Source: Figure 7 · p. 8, §4.2 · p. 8

5. Conclusion

The paper presents a generalizable Normal-Depth diffusion approach for 3D generation and a depth-conditioned albedo model for separating material attributes from lighting effects.

  • The Normal-Depth diffusion model is trained on real-world data and fine-tuned with synthetic datasets for generalizable 3D generation.The approach integrates into current text-to-3D pipelines and supports NeRF and DMTet representations.Source: §5 · p. 8
  • The depth-conditioned albedo diffusion model facilitates separation of material attributes and lighting effects.Source: §5 · p. 8
  • Extensive experiments report state-of-the-art text-to-3D results in geometry and appearance modeling.Source: §5 · p. 8
  • The current method predominantly focuses on object-level 3D generation, with future work targeting text-to-scene generation and specular appearance regularization.Source: §5 · p. 8

Training of VAE

The VAE is initialized from Stable Diffusion 2.1 and trained on high-aesthetic LAION images paired with monocular normal and depth estimates.

  • The VAE input channels are expanded from 3 to 4, while the latent space remains 4-channel and the added input-channel weights use averaged original weights.Source: Training of VAE · p. 13
  • The VAE training uses 8 million LAION-Aesthetics V1 samples with aesthetics scores above 8.0, resized to 384 × 384 after center cropping.Source: Training of VAE · p. 13
  • Augmented images are processed by NormalBae and Midas-3.1 to obtain corresponding normal and depth estimates.Source: Training of VAE · p. 13
  • The VAE is optimized with Adam at a learning rate of 5e-5, with KL regularization and an auxiliary discriminator improving latent-space and output quality.Source: Training of VAE · p. 13
  • The Normal-Depth diffusion model is first trained on Laion-2B-en, then fine-tuned on a higher-aesthetic subset at 512 × 512 resolution.Source: Training of VAE · p. 13

Multi-view Normal-Depth Diffusion Fine-tune

The multi-view fine-tuning pipeline adapts Normal-Depth diffusion to Objaverse views and normalizes synthetic depth to improve scale consistency during 3D optimization.

Depth Normalization

Albedo Diffusion

3D Integration

More Details for Ours (Sphere)

The Sphere and NeRF variants use staged geometry optimization, diffusion-guided refinement, and user studies comparing geometry and textured-model quality.

Geometry Optimization

  • Sphere optimization uses coarse and refinement stages, resizing rendered normal and depth maps first and then using VAE latent features for geometric detail.Source: More Details for Ours (Sphere) · p. 15

Ours (Sphere)

Ours (NeRF)

Geometry Results

User Study

F.1. Comparison with SweetDreamer

The comparison evaluates RichDreamer against SweetDreamer across NeRF- and DMTet-based settings, with preference and geometry-quality results favoring RichDreamer. The two RichDreamer initializations exhibit complementary strengths: NeRF handles multiple objects more easily, while Sphere preserves finer geometric detail.

  • 68% of users preferred Ours (NeRF) over SweetDreamer’s NeRF-based approach.Source: §F.1 · p. 17
  • 64% of users preferred Ours (Sphere) over SweetDreamer’s DMTet-based approach.Source: §F.1 · p. 17
  • RichDreamer generates better geometry than SweetDreamer in comparisons with both NeRF-based and DMTet-based methods.Source: §F.1 · p. 17

Initialization comparison

  • NeRF initialization more easily generates scenes containing multiple objects, such as a group of dogs playing poker.Source: §F.2 · p. 17
  • Sphere initialization better supports highly detailed surfaces, whereas NeRF initialization tends to produce smoother geometry.Source: §F.2 · p. 17
  • A future hybrid representation is proposed to combine the strengths of NeRF and Sphere initialization.Source: §F.2 · p. 17

F.3. More Visual Results

The supplementary visual results present additional RichDreamer outputs using Sphere and NeRF representations across Figures 18–25.

Related papers

Loading 2311.16918v2…