Source-linked AI summary
OrientDream: Streamlining Text-to-3D Generation with Explicit Orientation Control
Yuzhong Huang, Zhong Li, Zhang Chen, Zhiyuan Ren, Guosheng Lin, Fred Morstatter, Yi Xu
TL;DR
Text-to-3D lacks comprehensive, diverse 3D training data, while 2D-diffusion-prior optimization can produce Janus artifacts and take several hours per scene. OrientDream fine-tunes a 2D diffusion model with explicit orientation control using MVImgNet to promote multi-view consistency during SDS optimization. Experiments report that OrientDream circumvents the multi-head Janus issue while maintaining high quality and variety, and outperforms state-of-the-art methods in time efficiency.
Problem
Text-to-3D lacks comprehensive, diverse 3D training data, while 2D-diffusion-prior optimization can produce Janus artifacts and take several hours per scene.
Source: §1 · p. 2Method
OrientDream fine-tunes a 2D diffusion model with explicit orientation control using MVImgNet to promote multi-view consistency during SDS optimization.
Source: §1 · p. 2Results
Experiments report that OrientDream circumvents the multi-head Janus issue while maintaining high quality and variety, and outperforms state-of-the-art methods in time efficiency.
Source: §1 · p. 2Takeaways & Limitations
The authors present the work as a foundation for future research in applications of 3D technology.
Source: §5 · p. 9Frame-wise A-LPIPS similarity does not guarantee view consistency and can reward identical-looking views despite absent view diversity.
Source: §4.4 · p. 7
Abstract
from arXiv · showhide
In the evolving landscape of text-to-3D technology, Dreamfusion has showcased its proficiency by utilizing Score Distillation Sampling (SDS) to optimize implicit representations such as NeRF. This process is achieved through the distillation of pretrained large-scale text-to-image diffusion models. However, Dreamfusion encounters fidelity and efficiency constraints: it faces the multi-head Janus issue and exhibits a relatively slow optimization process. To circumvent these challenges, we introduce OrientDream, a camera orientation conditioned framework designed for efficient and multi-view consistent 3D generation from textual prompts. Our strategy emphasizes the implementation of an explicit camera orientation conditioned feature in the pre-training of a 2D text-to-image diffusion module. This feature effectively utilizes data from MVImgNet, an extensive external multi-view dataset, to refine and bolster its functionality. Subsequently, we utilize the pre-conditioned 2D images as a basis for optimizing a randomly initialized implicit representation (NeRF). This process is significantly expedited by a decoupled back-propagation technique, allowing for multiple updates of implicit parameters per optimization cycle. Our experiments reveal that our method not only produces high-quality NeRF models with consistent multi-view properties but also achieves an optimization speed significantly greater than existing methods, as quantified by comparative metrics.
1. Introduction
Text-to-3D methods using 2D diffusion priors face Janus artifacts and slow optimization, against a backdrop of limited and insufficiently diverse 3D training data. OrientDream adds camera-orientation-conditioned diffusion and decoupled NeRF updates; experiments report fewer Janus artifacts while maintaining quality and variety, with greater time efficiency than state-of-the-art methods.
1. Introduction
- Text-to-3D remains difficult partly because comprehensive 3D training data are lacking, while existing datasets are limited in scale and diversity.The introduction contrasts these limits with the extensive online image data supporting text-to-image diffusion models.Source: §1 · p. 2
- Without relative pose information, 2D-prior supervision can yield Janus duplication, including duplicated heads and other malformed body parts.Figure 2 illustrates examples involving a bunny, pig, eagle, and frog.Source: §1 · p. 2, Figure 2 · p. 2
- OrientDream fine-tunes a 2D diffusion model with explicit camera orientation, using MVImgNet data and orientation-conditioned 2D priors to supervise NeRF reconstruction.The dataset passage mentions frames and camera poses.Source: §1 · p. 2
- Decoupled back-propagation uses DDIM to resolve directly to XT-step, permitting multiple implicit-parameter updates per optimization cycle.This separates NeRF parameter updates from the diffusion process.Source: §1 · p. 2
- Experiments report that OrientDream circumvents the multi-head Janus issue while maintaining high quality and variety, and outperforms state-of-the-art methods in time efficiency.The authors also describe the method as enhancing 3D-generation efficiency while maintaining quality and variety.Source: §1 · p. 2
2. Related Works
The related work describes NeRFs as mappings from 3D locations to radiance, text-conditioned diffusion as a source of image priors, and per-prompt 2D lifting as a route to text-to-3D generation with consistency challenges.
2.1. NeRFs for Image-to-3D
- NeRFs map 3D coordinates to radiance, using an encoded location followed by a final MLP; spatial grids add spatial inductive biases.The point encoder maps locations to an encoded representation that the final MLP converts to radiance.Source: §2.1 · pp. 2–3
2.2. Text-to-Image Generation
- Text-to-image diffusion score-matches noisy images and uses text conditioning; the described setup uses T5-XXL and CLIP embeddings with cross-attention and reuses text tokens to modulate NeRF.Text conditioning enables generation of images corresponding to prompts.Source: §2.2 · p. 3
2.3. Text-to-3D Generation
- Prior per-prompt 2D-lifting methods update NeRF from diffusion noise estimates but can produce Janus inconsistencies; mitigations alter embeddings, noise, or gradients, or use multi-view and 3D-trained priors.The authors note that methods using smaller, lower-texture 3D datasets tend to reflect those datasets' average quality.Source: §2.3 · p. 3
3. Method
OrientDream conditions a 2D diffusion model on text and camera orientation, then uses its generated views to guide NeRF reconstruction. It also decouples diffusion and NeRF updates to accelerate optimization.
3.1. Camera Orientation Conditioned Diffusion Model
- Explicit camera-orientation conditioning addresses pose ambiguity in text prompts, which can leave NeRF geometry inconsistent across views.The method adds encoded camera orientations to the diffusion model rather than relying on direction-annotated prompts such as “side view.”Source: §3.1 · pp. 3–4
- The model combines the text CLIP feature with encoded orientation as c′=c+p, using quaternion representation and sine encoding to support smooth interpolation between poses.The authors report that quaternion form with sine encoding most effectively distinguishes view angles among the evaluated alternatives.Source: §3.1 · p. 4
3.2. Text to 3D generation
- The orientation-conditioned diffusion model supplies a multi-view prior for NeRF through SDS, replacing Stable Diffusion and using camera parameters rather than direction-annotated prompts.The authors also use this model to initialize the variational model in VSD.Source: §3.2 · p. 5
3.3. Decoupled Back Propagation
- Decoupled back-propagation uses DDIM to compute a diffusion state several steps ahead, then optimizes NeRF at the sampled camera pose with MSE loss.The step size is reduced during training; the authors say late-training quality approaches that of a 2D diffusion solver.Source: §3.3 · p. 5
- The method avoids SDS loss and is reported to improve diversity and texture quality while accelerating training through fewer diffusion-model evaluations.This efficiency depends on the 2D diffusion model generating views at specified locations precisely enough to guide NeRF.Source: §3.3 · p. 5
4. Experiments
The experiments evaluate training details, orientation-controlled image generation, text-to-3D lifting, and quantitative quality and speed. They report orientation control and quality advantages over selected baselines, alongside faster generation and image-synthesis results.
4.1. Training Details
- The orientation-controlled diffusion model is fine-tuned from Stable Diffusion 2.1 on 6.5 million MVImgNet frames from 219,188 videos and 238 classes for two epochs.Dreamfusion-based methods are used for 2D-to-3D lifting, implemented in Threestudio, which is also used to replicate baselines.Source: §4.1 · p. 5
4.2. Orientation-Controlled Image Generation
- The orientation-controlled diffusion model and MVDream reliably control viewing orientation, unlike text-suffix conditioning, while OrientDream produces photorealistic images with richer texture and variety.The authors note that MVDream generates four views simultaneously while their method generates them independently, so the four-view consistency comparison is not apples-to-apples.Source: §4.2 · p. 6
4.3. Orientation-Controlled Diffusion for Text to 3D
- DreamFusion shows duplicated heads in toucan and eagle outputs, while OrientDream’s results are described as free from the Janus problem and rich in detail and texture.MVDream maintains good 3D consistency but is described as lower in content richness and texture quality.Source: §4.3 · p. 6, Figure 6 · p. 7
4.4. Quantitative Evaluation
- Our method achieves comparable Zero123grad to MVDream while outperforming it in R-Precision and A-LPIPS on 64 prompts.The authors interpret this as similar 3D awareness but superior texture quality.Source: §4.4 · p. 8, Table 1 · p. 8
- Generation for 64 prompts is faster with our method than with other methods, which the authors attribute to decoupled optimization.Generation speed is reported as a usability factor; the table caption identifies the speed analysis.Source: §4.4 · p. 8, Table 3 · p. 9
4.5. Evaluation of Orientation Conditioned Image Generation.
- Stable Diffusion achieves the best FID, IS, and CLIP scores; our method’s FID of 16.93 is closer to the original model than MVDream’s post-fine-tuning FID.Scores use 397 prompts, with 10 generated images per prompt.Source: §4.5 · p. 9, Table 4 · p. 9
5. Conclusion
OrientDream’s camera orientation conditioned diffusion model targets more accurate and consistent 3D generation, while decoupled back-propagation is intended to improve efficiency and diversity.
5. Conclusion
- The camera orientation conditioned diffusion model advances 3D generation toward greater accuracy and consistency.The conclusion identifies it as a key aspect of the work.Source: §5 · p. 9
- OrientDream circumvents the multi-head Janus issue in text-to-3D generation.Source: §5 · p. 9
- Decoupled back-propagation represents an effort to improve 3D model generation efficiency and diversity.Source: §5 · p. 9
- The authors hope the work will provide a foundation for future research in 3D technology applications.Source: §5 · p. 9