Source-linked AI summary
Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Hyeonsu Kim, Jaehoon Ko, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, Seungryong Kim
TL;DR
SDS-based text-to-3D generation can produce inconsistent geometry because its pretrained 2D diffusion models lack explicit 3D and camera awareness. 3DFuse conditions those models on projected depth maps from coarse 3D structures, and reports improved generation quality and geometric consistency across multiple baselines.
Problem
SDS-based text-to-3D methods suffer inconsistent and distorted geometry because 2D diffusion models lack explicit awareness of 3D space and camera viewpoints.
Method
3DFuse generates coarse 3D structures, projects viewpoint-specific depth maps, and injects them into a pretrained 2D diffusion model alongside semantic coding.
Results
3DFuse improves generation quality and geometric consistency across Dreamfusion, SJC, and ProlificDreamer, with quantitative and qualitative evaluation supporting its effectiveness.
Takeaways & Limitations
The framework provides a practical approach for adding 3D awareness to pretrained 2D diffusion models while retaining their generalization capability for text-to-3D generation.
Takeaways & Limitations
3DFuse inherits difficulty in faithfully reflecting complex prompts and may reproduce societal biases present in the pretrained diffusion model’s data.
Abstract
from arXiv · showhide
Text-to-3D generation has shown rapid progress in recent days with the advent of score distillation, a methodology of using pretrained text-to-2D diffusion models to optimize neural radiance field (NeRF) in the zero-shot setting. However, the lack of 3D awareness in the 2D diffusion models destabilizes score distillation-based methods from reconstructing a plausible 3D scene. To address this issue, we propose 3DFuse, a novel framework that incorporates 3D awareness into pretrained 2D diffusion models, enhancing the robustness and 3D consistency of score distillation-based methods. We realize this by first constructing a coarse 3D structure of a given text prompt and then utilizing projected, view-specific depth map as a condition for the diffusion model. Additionally, we introduce a training strategy that enables the 2D diffusion model learns to handle the errors and sparsity within the coarse 3D structure for robust generation, as well as a method for ensuring semantic consistency throughout all viewpoints of the scene. Our framework surpasses the limitations of prior arts, and has significant implications for 3D consistent generation of 2D diffusion models.
1 INTRODUCTION
Text-to-3D methods use pretrained 2D diffusion models to optimize 3D representations, but their lack of camera and 3D awareness can produce inconsistent geometry. 3DFuse addresses this by injecting coarse, view-specific 3D structure while preserving diffusion-model generation capabilities.
- Text-to-3D generation has expanded through diffusion models, enabling content creation without professional modeling tools across several industries.
- SDS optimizes NeRFs with gradients from pretrained 2D diffusion models, but can produce distortions and duplicated frontal features across viewpoints.This failure is known as the Janus problem and reflects missing camera-pose awareness.
- Pose-conditioned diffusion models still struggle with 3D transformations and self-occlusions, while requiring costly ground-truth 3D data that is smaller and less extensive than available 2D data.
- 3DFuse injects 3D awareness into pretrained 2D diffusion models using projected coarse depth maps, semantic coding, and a consistency injection module.The coarse geometry is generated from text and projected into arbitrary views for conditioning.
- Across Dreamfusion, SJC, and ProlificDreamer, 3DFuse improves generation quality and geometric consistency and introduces a metric for quantitative 3D-consistency assessment.
2 RELATED WORK
Related work spans diffusion-based text-to-image generation, SDS-based text-to-3D methods, and image-to-3D or 3D-native approaches. These methods differ in whether they use 2D guidance, conditioning images, or directly generated 3D representations.
- Diffusion models support image translation and conditional generation, with text-to-image systems using guidance techniques and CLIP-based conditioning.
- Text-to-3D methods progressed from CLIP-guided NeRF optimization to SDS methods using frozen diffusion models, including Dreamfusion and SJC.
- Image-to-3D methods generate scenes from conditioning images, combining 3D-native diffusion, reconstruction losses, monocular depth estimation, or textual inversion.
3 PRELIMINARY
Text-conditioned diffusion models predict noise from noisy images and prompt embeddings, while SDS uses those predictions to optimize NeRF parameters through rendered views. 3DFuse extends this architecture with semantic-code sampling and a consistency-injection pathway.
- A text-to-image diffusion model maps prompt c to embedding e = T(c) and predicts noise from noisy image x_t conditioned on that embedding.
- SDS samples camera pose π, renders x = R_Θ(π), and uses the frozen diffusion model’s score to optimize NeRF parameters Θ toward realistic images.
- The SDS gradient omits the diffusion U-net Jacobian term and uses a weighting function derived from the DDPM diffusion process.
- 3DFuse adds semantic code sampling and a consistency injection module that supplies a 3D-aware condition through a sparse depth injector.
4 METHOD
3DFuse injects 3D awareness into pretrained 2D diffusion models by conditioning them on projected depth maps from a coarse 3D structure, while semantic code sampling and LoRA promote consistency across viewpoints.
- Motivation: 3DFuse addresses distorted, viewpoint-inconsistent geometry in SDS-based text-to-3D generation caused by 2D diffusion models lacking explicit 3D awareness.Prior methods that add camera-view text prompts are described as ad hoc and limited.
- Semantic Code Sampling: Semantic code sampling generates an image and optimizes its text embedding to secure a consistent semantic identity for the scene.The generated image and optimized embedding form a semantic code used by the consistency injection module.
- Coarse 3D Prior: A coarse point cloud generated from the semantic image is projected into each rendering view to provide a sparse depth map as a 3D-consistent condition.The depth injector adds features from the projected depth map to intermediate pretrained diffusion U-net features using a ControlNet-style architecture.
- Robust Depth Injection: The sparse depth injector is trained on point-cloud depths and MiDaS-predicted dense depths so it can infer robust structure from sparse, noisy, and out-of-domain geometry.Only the depth injector is trained while the diffusion model remains frozen.
- Robust Depth Injection: Training enables realistic generation from sparse and noisy depth maps without an auxiliary depth-completion network, including bird categories absent from the point-cloud training data.This result is shown in qualitative comparisons using sparse depth-map conditioning.
- Semantic Consistency: LoRA layers are tuned while the optimized embedding is fixed to strengthen semantic consistency across viewpoints while avoiding overfitting to a specific viewpoint.The framework is also reported to improve 3D consistency and fidelity when incorporated into ProlificDreamer, Stable-DreamFusion, and SJC baselines.
5 EXPERIMENTS
Across qualitative, quantitative, and user-study evaluations, 3DFuse improves geometric consistency and fidelity across SDS-based text-to-3D baselines, while supporting viewpoint control and semantic consistency.
- Qualitative evaluation: 3DFuse improves geometric consistency and high-fidelity generation when combined with DreamFusion, SJC, and ProlificDreamer.Qualitative results show more robust geometry across viewpoints, including when MCC supplies the 3D prior.
- Quantitative evaluation: 3DFuse outperforms baseline SJC by a large margin on a COLMAP-based 3D-consistency metric averaged over 42 generated scenes.The metric addresses the difficulty of evaluating zero-shot text-to-3D generation without ground-truth 3D scenes.
- User study: A user study with 102 participants finds that 3DFuse is preferred for 3D coherence, prompt adherence, and overall quality over Stable-DreamFusion and SJC.Participants selected among rendered scenes generated from identical prompts.
- View-dependent image generation: 3DFuse enables precise camera-pose control in 2D image generation, with small viewpoint changes reflected in the outputs.It performs better than naive view-augmented prompting in the reported comparison.
- Comparison with Zero-123: Compared with Zero-1-to-3, 3DFuse produces more expressive and high-fidelity scenes when all other SDS settings are held identical.The paper attributes this difference to finetuning only an external conditioning module while preserving the pretrained diffusion model.
- Ablation study: Semantic code sampling preserves geometric and semantic consistency across viewpoints, unlike the viewpoint-dependent differences observed without it.The ablation uses prompts describing a pumpkin and a toy tank.
6 CONCLUSION
The paper concludes that 3DFuse addresses SDS-based 3D inconsistency by conditioning pretrained 2D diffusion models on coarse-structure depth maps and adding semantic-consistency mechanisms.
- 3DFuse incorporates 3D awareness into a pretrained 2D diffusion model for SDS-based text-to-3D generation.
- Viewpoint-specific depth maps from coarse 3D structures provide geometric conditions, while sparse depth injection and semantic code sampling support consistency.
- Experimental results report improvements over previous models in quantitative metrics and qualitative human evaluation.
A.1 TRAINING DETAILS
The implementation trains external conditioning components for a Stable Diffusion-based framework, using coarse 3D priors and semantic code sampling to guide generation.
- Training data: The sparse depth injector is trained on Co3D image-depth pairs generated from annotated point-cloud videos.The training construction samples three frames from 5,625 videos, producing 16,875 RGB-depth pairs.
- Semantic code sampling: Semantic code sampling first generates an image from the text prompt, optimizes its text embedding, and uses the image to infer an auxiliary 3D shape.
- Diffusion backbone: Stable Diffusion is adopted because DALL-E2 and Imagen are not publicly available.
- Architectural choices: The method evaluates conditioning approaches including ControlNet-style depth conditioning and attaches LoRA layers to diffusion U-Net attention layers.
- Concurrent-work comparison: Compared with explicit image-to-3D models, 3DFuse generates more detailed, high-fidelity results through looser conditioning that preserves generative freedom.
B.2 ROBUSTNESS
Additional experiments indicate that 3DFuse remains robust across random seeds and sparse point-cloud conditions, while qualitative comparisons report stronger robustness and 3D consistency than prior methods.
- Random-seed robustness: Varying random seeds from fixed prompts shows enhanced stochastic robustness for 3DFuse compared with previous works.
- Point-cloud sparsity: 3DFuse infers consistent dense structures and realistic images despite substantial changes in point-cloud sparsity.
- Qualitative comparison: Additional qualitative results report superior robustness and 3D consistency compared with previous methods.
B.5 ABLATION ON SPARSE DEPTH INJECTOR
The ablation examines sparse depth injection and compares 3DFuse with prior methods using a controlled user study. Removing the injector breaks geometric consistency, while the study evaluates coherence, prompt adherence, and overall quality.
- Ablation: Removing the sparse depth injector produces blurry and inconsistent 3D shapes, confirming its importance for geometric consistency.The ablation compares the framework with and without the injector.
- User study: The user study compares 3DFuse, SJC, and Stable-DreamFusion using identical prompts and camera poses across rendered scenes.The study uses seven prompts, three scenes per prompt, five rendered images per scene, and 102 participants.
- User study: Participants evaluate natural shape, prompt adherence, and overall quality from anonymized, randomized renderings.The natural-shape question targets 3D coherence across horizontal camera rotation from 0 to 360 degrees.
C.2 COLMAP-BASED METRIC
The COLMAP-based metric evaluates 3D consistency through camera-pose reconstruction from uniformly sampled views. Its variance measure links unstable pose predictions to distortions and inconsistent geometry, with 3DFuse producing more stable poses than SJC.
- Metric construction: The metric renders 100 views from cameras on a hemisphere and uses COLMAP to reconstruct their camera poses.Cameras share a fixed radius and 30-degree elevation, while all point toward the hemisphere center.
- Metric interpretation: Variance in predicted camera differences measures 3D inconsistency because distortions and repetitive artifacts make COLMAP optimization ambiguous.Adjacent camera differences are identical at rendering time, so higher variance indicates greater prediction inaccuracy.
- Comparison: COLMAP predicts more uniformly spaced and stable cameras for 3DFuse scenes than for inconsistent SJC scenes.For an SJC-generated horse, COLMAP incorrectly places cameras on only one side of the hemisphere.
C.3 LIMITATION OF CLIP-BASED METRIC
Standard image-generation and CLIP-based metrics do not reliably measure geometric consistency in zero-shot text-to-3D generation. A geometrically inconsistent scene can therefore receive a higher CLIP score than a consistent one.
- Evaluation gap: FID and Inception Score cannot be directly applied because zero-shot text-to-3D generation lacks ground-truth 3D scenes for comparison.This evaluation gap has led prior work to rely on qualitative results or user studies.
- CLIP limitation: CLIP-based metrics are ineffective for measuring the quality and consistency of generated 3D geometry.They evaluate projected images against text rather than directly assessing 3D structure.
- CLIP limitation: A multiple-face failure can produce misleadingly high CLIP scores because repeated frontal features match the text from many viewpoints.In the illustrated example, every image from the inconsistent scene scores higher than images from the consistent scene.
D LIMITATION
3DFuse preserves the generalization capability of a pretrained 2D diffusion model while improving text-to-3D optimization, but it remains constrained by the underlying model’s prompt-following ability and dataset biases.
- Limitations: 3DFuse may struggle to faithfully reflect complex user prompts because it relies on the pretrained diffusion model’s text-following ability.This limitation concerns prompt complexity and the inherited capabilities of the pretrained model.
- Limitations: 3DFuse may inherit societal biases present in the training dataset, as in text-to-image generation models.The paper identifies this as an additional limitation of the approach.