Source-linked AI summary
SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D
Weiyu Li, Rui Chen, Xuelin Chen, Ping Tan
TL;DR
Lifting view-agnostic 2D diffusion priors into 3D is ambiguous and produces multi-view inconsistency, especially from misplaced geometry. SweetDreamer fine-tunes diffusion to generate camera-conditioned canonical coordinate maps from coarse 3D geometry, then integrates these aligned geometric priors into existing pipelines. The method achieves over 85% consistency in both evaluated pipelines, compared with around 30% for previous methods, while preserving appearance modeling and generalization.
Problem
2D diffusion models lack 3D knowledge during lifting, causing multi-view inconsistency; limited 3D data makes learning comprehensive 3D priors difficult.
Method
AGP fine-tunes a pre-trained 2D diffusion model to generate viewpoint-conditioned coarse geometric maps of canonically oriented objects, using them as geometry guidance in existing pipelines.
Results
Over 85% consistency was achieved in both pipelines, compared with around 30% for previous methods.
Takeaways & Limitations
AGP improves geometry modeling while retaining the pre-trained diffusion model’s ability to generate high-quality, varied objects and visual appearances.
Takeaways & Limitations
AGP does not directly model appearance, so ambiguity in mapping geometry to appearance may still rarely cause inconsistency.
Abstract
from arXiv · showhide
It is inherently ambiguous to lift 2D results from pre-trained diffusion models to a 3D world for text-to-3D generation. 2D diffusion models solely learn view-agnostic priors and thus lack 3D knowledge during the lifting, leading to the multi-view inconsistency problem. We find that this problem primarily stems from geometric inconsistency, and avoiding misplaced geometric structures substantially mitigates the problem in the final outputs. Therefore, we improve the consistency by aligning the 2D geometric priors in diffusion models with well-defined 3D shapes during the lifting, addressing the vast majority of the problem. This is achieved by fine-tuning the 2D diffusion model to be viewpoint-aware and to produce view-specific coordinate maps of canonically oriented 3D objects. In our process, only coarse 3D information is used for aligning. This "coarse" alignment not only resolves the multi-view inconsistency in geometries but also retains the ability in 2D diffusion models to generate detailed and diversified high-quality objects unseen in the 3D datasets. Furthermore, our aligned geometric priors (AGP) are generic and can be seamlessly integrated into various state-of-the-art pipelines, obtaining high generalizability in terms of unseen shapes and visual appearance while greatly alleviating the multi-view inconsistency problem. Our method represents a new state-of-the-art performance with an 85+% consistency rate by human evaluation, while many previous methods are around 30%. Our project page is https://sweetdreamer3d.github.io/
1 INTRODUCTION
Text-to-3D methods often lift view-agnostic 2D diffusion priors into 3D, where geometric ambiguity causes multi-view inconsistency. SweetDreamer aligns those priors with coarse, well-defined 3D geometries while preserving their generalizability, achieving over 85% consistency in human evaluation.
- Motivation: 2D diffusion lacks 3D knowledge, making lifting inherently ambiguous and causing multi-view failures such as the multi-face Janus problem.The paper frames limited 3D data and abundant image data as motivating the transfer of 3D priors into generalizable 2D priors.
- Motivation: Geometric inconsistency is identified as the primary source of inconsistent results, while appearance inconsistency usually appears only in extreme cases.Geometric errors arise when structures can occupy different positions or orientations despite identical 2D projections.
- Contribution: AGP fine-tunes 2D diffusion to produce camera-conditioned coordinate maps aligned with canonical 3D geometries, using coarse geometry rather than detailed appearance.This targets the dominant geometric inconsistency while retaining the diffusion model’s ability to generate varied content.
- Results: 85+% consistency was achieved in human evaluation, while the method remains compatible with multiple pipelines and generalizes to unseen geometries and appearances.The authors describe this as state-of-the-art performance and report the result in both qualitative and quantitative evaluations.
2 RELATED WORK
Related work primarily lifts 2D diffusion priors into 3D through score-based optimization, multi-view diffusion, or view-conditioned image translation. SweetDreamer instead uses viewpoint-aware coarse geometry to support consistency without requiring intensive joint training on synthetic and real images.
- Text-to-3D using 2D Diffusion: Score Distillation Sampling uses diffusion priors as score functions to supervise optimization of a 3D representation.This line of work bypasses the need for large-scale text-3D datasets for training scalable 3D diffusion models.
- Text-to-3D using 2D Diffusion: MVDream trains a dedicated multi-view diffusion model on real and synthetic multi-view images to generate consistent sparse views.The cited comparison contrasts this with SweetDreamer’s use of low-resolution geometric information rather than computationally intensive renderings and joint fine-tuning.
- Generative Novel Views with Diffusion Models: Existing view-conditioned methods may capture view specifications inaccurately, whereas AGP uses camera conditions only to generate coarse geometries that later evolve into consistent 3D objects.The comparison distinguishes AGP’s purpose from methods requiring accurate viewpoint awareness for direct consistent generation.
3 METHOD
SweetDreamer aligns 2D diffusion geometric priors with canonical 3D coordinate maps and injects camera information to produce viewpoint-aware coarse geometry. AGP is then added to existing text-to-3D pipelines as geometry supervision while leaving appearance modeling intact.
- Method motivation: The method targets geometric inconsistency because consistent geometry supports later modeling of detailed geometry and visual appearance.The paper distinguishes misplaced geometric structures from incorrect appearance modeling on those structures.
- 3.1 ALIGNNING GEOMETRIC PRIORS IN 2D DIFFUSION: Training renders depth maps from canonically oriented 3D models and converts them into canonical coordinate maps, using coarse geometry only for alignment.The 3D data aligns geometric priors rather than supplying detailed geometric content.
- 3.1 ALIGNNING GEOMETRIC PRIORS IN 2D DIFFUSION: Canonical Coordinates Maps represent normalized objects in a canonical space contained within a unit cube, reducing ambiguity from object orientation.Objects are uniformly scaled so the maximum extent of their tight bounding box has length 1 and is centered at the origin.
- 3.1 ALIGNNING GEOMETRIC PRIORS IN 2D DIFFUSION: Camera extrinsic parameters are injected into the diffusion model to improve viewpoint awareness, but only for roughly generating coarse geometries.The resulting geometry is intended to evolve into a consistent 3D object during subsequent optimization.
- 3.1 ALIGNNING GEOMETRIC PRIORS IN 2D DIFFUSION: Fine-tuning preserves the 2D diffusion architecture while conditioning it on camera parameters, transferring generalization to varied subjects unseen in the 3D dataset.The implementation uses Stable Diffusion v2.1 as the pre-trained text-to-image model.
- 3.2 INTEGRATION INTO TEXT-TO-3D: AGP integrates into existing pipelines through an additional geometry-supervision branch, including DMTet-based and NeRF-based representations.The DMTet integration applies aligned-prior supervision during coarse and fine geometry stages, while appearance modeling remains untouched.
4 TEXT-TO-3D GENERATION
The evaluation compares AGP-integrated text-to-3D pipelines with baselines using quantitative consistency tests, visual comparisons, and a user study. Across these evaluations, AGP substantially improves multi-view consistency while remaining compatible with multiple pipelines and representations.
- Evaluation setup: The evaluation compares DMTet- and NeRF-based pipelines, including NeRF variants using DeepFloyd IF and Stable Diffusion, against several text-to-3D baselines.The baselines include Fantasia3D, DreamFusion-IF, SJC, Magic3D-IF, TextMesh-IF, and MVDream.
- Qualitative evaluation: Visual comparisons show high-fidelity, highly consistent outputs from the proposed pipelines, while baseline results exhibit 3D inconsistency issues.The figure highlights baseline inconsistencies with red arrows.
- Quantitative evaluation: The quantitative evaluation generates 80 results per method from randomly selected DreamFusion prompts and manually counts 3D inconsistencies.The reported success rate is the number of 3D-consistent objects divided by the total number generated.
- Quantitative evaluation: 85+% success rates are achieved in both pipelines, compared with around 30% for previous methods.The success rate measures the proportion of generated objects judged 3D-consistent.
- Qualitative evaluation: Integrating AGP into Fantasia3D significantly improves its results, while the NeRF-based pipeline produces realistic appearances with high 3D consistency.The reported appearance benefit is attributed to preserving Stable Diffusion’s visual priors during appearance modeling.
- User study: In a user study of 30 generated results with 36 participants, the proposed method had the highest preference rate when participants judged only 3D consistency.Participants viewed videos rendered from models generated from the same text prompts.
5 CONCLUSION
The paper introduces Aligned Geometric Priors (AGP), which add viewpoint-conditioned coarse geometric guidance to 2D diffusion models for consistent text-to-3D generation. AGP improves geometry modeling without compromising appearance modeling, while direct appearance modeling remains an unresolved limitation.
- Conclusion: AGP fine-tunes a pre-trained 2D diffusion model to generate viewpoint-conditioned coarse geometric maps of canonically oriented objects.The resulting priors confer 3D awareness and can be integrated into existing text-to-3D pipelines.
- Conclusion: AGP improves geometry modeling without compromising appearance modeling guided by strong priors learned from billions of real images.The method is designed to preserve the 2D diffusion model’s appearance capability while aligning geometric priors.
- Limitations: The method does not directly address appearance modeling, so ambiguity between geometric structure and appearance can still cause rare appearance inconsistencies.An attempted appearance generator overfit to renderings derived from 3D data and produced insufficiently photorealistic results.
A.1 GEOMETRIC PRIORS IN 2D DIFFUSION
Stable Diffusion’s latent representations visibly retain geometric structures, revealing geometric priors learned during 2D diffusion training. These priors are not inherently 3D-aware.
- Stable Diffusion compresses images into a latent space using a variational autoencoder before diffusion modeling.The latent diffusion model generates images from text in this compressed space.
- Latent images preserve geometric cues such as head contours, cheek surfaces, eye edges, and whisker patterns.These structures indicate that geometric priors emerge as a by-product of diffusion-model training.
- Although these priors contain geometric information, they are not 3D-aware and can contribute to multi-view inconsistency during lifting.
- Two rendered views of a 3D head are encoded into the diffusion model’s latent space, producing latent images with clear geometric structures.
A.2 MORE TEXT-TO-3D RESULTS
The appendix presents additional text-to-3D synthesis results from the proposed methods across Figures 6–9.
- Additional text-to-3D synthesis results are presented in Figure 6.
- Additional text-to-3D synthesis results are presented in Figure 7.
- Additional text-to-3D synthesis results are presented in Figures 8 and 9.
A.3 MORE IMPLEMENTATION DETAILS OF OUR PIPELINES
The proposed method is implemented in both DMTet-based and NeRF-based text-to-3D pipelines, using established frameworks and distinct 3D representations.
- DMTet-based pipeline: The DMTet-based implementation integrates Aligned Geometric Priors into Fantasica3D using the original paper’s parameters.Geometry and appearance learning are disentangled.
- DMTet-based pipeline: The DMTet-based pipeline takes about 12 minutes for fine geometry and 8 minutes for corresponding PBR materials per object.
- NeRF-based pipeline: The NeRF-based implementation uses threestudio with Instant-NGP as the optimized 3D representation.Instant-NGP uses a multi-resolution hash-grid to predict sampled ray colors and densities.
A.4 MORE COMPARISON RESULTS USING PROMPTS FROM MVDREAM
The appendix compares the method with MVDream using prompts listed on MVDream’s website and reports a generalizability trade-off for that concurrent method.
- Because MVDream’s official implementation was unavailable, the comparison uses prompts listed on its website.
- The visual comparisons are presented in Figure 10.
- MVDream can resolve multi-view inconsistency but is observed to overfit limited 3D data, compromising generalizability in the original 2D diffusion model.
A.5 MORE DISCUSSION
The discussion reports additional generated results from the proposed DMTet- and NeRF-based models and compares them with competing methods. It also notes a potential generalizability risk because AGP training uses no regularization constraint.
- AGP training introduces no regularization constraint, creating a potential risk of degrading generalizability for highly diverse geometries.
- The proposed models include DMTet-based and NeRF-based variants, with additional generated results presented for both.
- The comparison includes DreamFusion, Magic3D, TextMesh, ProlificDreamer, MVDream, and the proposed variants.
- MVDream results omit prompt concepts such as a rocket, backpack, and squirrel in some side-by-side comparisons.