SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D

1 INTRODUCTION

• Because high-quality 3D data is scarcer and costlier to collect than paired text-image data, learning comprehensive text-to-3D priors faces a substantial data challenge.The introduction contrasts efforts to collect text-3D pairs with the extensive paired text-image corpora used to train scalable diffusion models.Source: §1 · p. 1
• 2D lifting optimizes differentiable 3D representations with Score Distillation Sampling from pretrained 2D diffusion models, producing high-fidelity objects and scenes from text prompts [Poole et al., 2022; Lin et al., 2023; Chen et al., 2023; Wang et al., 2023b; Shi et al., 2023].The approach is promising, but its 2D priors lack 3D knowledge to disambiguate lifted results.Source: §1 · pp. 1, 3
• Geometric inconsistency is the primary cause of multi-view failures: misplaced or repetitive structures dominate, while appearance inconsistency alone is less common and can persist in exceptional cases.The authors argue that supplying hints from 3D-consistent geometry also aids appearance modeling, including in pipelines that optimize geometry and appearance together or separately [Poole et al., 2022; Chen et al., 2023; Richardson et al., 2023].Source: §1 · p. 3
• AGP fine-tunes 2D diffusion to generate canonical-space coordinate maps conditioned on camera specifications, aligning geometric priors to coarse 3D shapes during lifting.Using only coarse geometry avoids reliance on detailed 3D appearance information while addressing most inconsistency and retaining the ability to generate vivid, diverse objects unseen in 3D datasets.Source: §1 · p. 3
• 85+% consistency rate in human evaluation establishes AGP as a new state-of-the-art result, while it significantly alleviates multi-view inconsistency across state-of-the-art pipelines.AGP is described as compatible with different 3D representations and generalizable to unseen geometries and appearances.Source: §1 · p. 3

Text-to-3D using 2D Diffusion

• Text-to-3D methods use diffusion priors to optimize 3D representations without text-3D training data, but often remain inconsistent despite attempts to improve score functions or prompts.Poole et al. (2022) introduced Score Distillation Sampling, while Wang et al. (2023a) backpropagated diffusion scores through a differentiable renderer; later work explored sampling schedules, 3D representations, and score distillation.Source: §2 · p. 4
• MVDream (Shi et al., 2023) fine-tunes on real and synthetic multi-view images, whereas SweetDreamer uses low-resolution geometry maps for efficient alignment and integrates into existing pipelines.The paper contrasts this generic compatibility with MVDream, and says SweetDreamer avoids computationally intensive renderings and fine-tuning on both synthetic and real images.Source: §2 · p. 4

Generative Novel Views with Diffusion Models

• View-conditioned diffusion models often fail to accurately capture view specifications, producing multiple views that are only approximately 3D consistent; AGP also takes camera specifications but generates coarse geometries that subsequently evolve into a 3D consistent object.AGP does not use appearance information from synthetic renderings in 3D datasets, which could compromise pretrained diffusion-model visual priors and degrade visual quality.Source: §2 · p. 4

3 METHOD

• The method targets geometric inconsistency as a main source of 3D-inconsistent results, using consistent geometry to support later modeling of intricate details and appearance.It aligns the 2D priors with well-defined geometries from canonically oriented 3D assets, using coarse shapes for alignment rather than detail generation.Source: §3 · pp. 4–5

3.1 ALIGNNING GEOMETRIC PRIORS IN 2D DIFFUSION

• Canonical coordinate maps use category-level canonical orientation and unit-cube normalization; anisotropic scaling makes thin-structure coordinate differences across views easier to perceive.The maps are rendered from uniformly normalized objects, with each coordinate component scaled into the range 0 to 1.Source: §3.1 · p. 5
• Camera extrinsics, passed through an MLP, condition fine-tuning of the pretrained 2D diffusion model to generate canonical coordinate maps for specified views.The camera condition need only provide rough viewpoint awareness for coarse geometries; retaining the pretrained architecture enables transfer of generalizability to varied subjects unseen in the 3D dataset.Source: §3.1 · pp. 5–6

3.2 INTEGRATION INTO TEXT-TO-3D

• In the DMTet-based pipeline, a parallel AGP branch supervises geometry in both coarse and fine stages, while leaving appearance modeling untouched.The added supervision is combined with the original geometry SDS loss, and the integration avoids the original pipeline’s need for carefully designed initialization shapes.Source: §3.2 · p. 7
• In the NeRF-based pipeline, AGP supplies geometry SDS alongside the original RGB-image SDS to update the geometry branch, while appearance modeling remains untouched.The two losses are combined with weights balancing their effects, and AGP continues to model coarse, 3D-consistent geometry.Source: §3.2 · p. 7

Baselines

• The evaluation compares DMTet- and NeRF-based AGP pipelines against Fantasia3D, DreamFusion-IF, SJC, Magic3D-IF, TextMesh-IF, and MVDream.The NeRF variants use DeepFloyd IF alone or DeepFloyd IF followed by Stable Diffusion; baselines generally use implementations from Guo et al. (2023), and Stable Diffusion is the default prior except for IF-suffixed methods.Source: §4 · pp. 7–8

Qualitative Evaluation

• AGP integration improved Fantasia3D’s coarse, inaccurate outputs, while the NeRF-based pipeline produced consistent, more realistic results without sacrificing appearance modeling.The authors attribute the DMTet improvement to addressing a gap between rendered normal maps and Stable Diffusion’s latent geometry, and say AGP guides geometry without compromising appearance priors.Source: Figure 4 · p. 8, §4 · p. 9

Quantitative Evaluation

• Over 85+% success rates in both pipelines, versus around 30% for previous methods, were reported after manual consistency checks on 80 DreamFusion-gallery prompts.Because comprehensive evaluation metrics are lacking, the study focuses on multi-view consistency and counts consistent objects among 80 results per method.Source: §4 · pp. 8–9, Table 1 · p. 9

Qualitative Evaluation

• Although MVDream also resolves multi-view inconsistency, the authors report limited-data overfitting, reduced generalizability, a missing backpack, and less photorealistic appearance.They attribute the appearance limitation to its use of synthetic multi-view renderings for fine-tuning.Source: §4 · p. 9

User Study

• In a 36-participant study over 30 rounds, participants preferred the authors’ method by a large margin when judging consistency in rendered videos.Participants chose their favored 3D result based only on consistency.Source: §4 · p. 9

• AGP improves geometry modeling without compromising appearance modeling guided by strong priors learned from billions of real images.It fine-tunes a pretrained 2D diffusion model to generate viewpoint-conditioned coarse geometric maps of canonically oriented objects and integrates into existing pipelines.Source: §5 · p. 9

Discussion

• Appearance inconsistency may still arise rarely because the mapping from geometric structure to associated appearance remains ambiguous.The work does not directly consider appearance modeling.Source: §5 · p. 9
• Fine-tuning an appearance generator on 3D renderings led to overfitting and results lacking the desired photorealism, so the authors leave this direction for future work.Source: §5 · pp. 9–10
• The authors propose a “less is more” direction: use relatively limited 3D data to enhance 2D diffusion priors for text-to-3D synthesis.Source: §5 · p. 10

A.1 GEOMETRIC PRIORS IN 2D DIFFUSION

• Stable Diffusion generates text-conditioned images by applying diffusion in a latent space produced by a variational autoencoder.It compresses images into latent space rather than operating directly in high-dimensional image space.Source: §A.1 · p. 12
• These structures indicate that training the diffusion model to generate latent images also teaches it geometric priors.Source: §A.1 · p. 12
• The success of coarse geometry modeling under diffusion-model supervision indirectly supports the presence of these priors (Chen et al., 2023).Source: §A.1 · p. 12
• However, geometric priors in 2D diffusion are not 3D-aware.Source: §A.1 · p. 12
• This lack of 3D awareness leads to multi-view inconsistency when 2D diffusion results are lifted to 3D.Source: §A.1 · p. 12
• The VAE latents show geometric structures such as head contours, flat cheek surfaces, sharp eye edges, and dense whisker dots.Figure 5 describes latent images encoded from renderings of a 3D head from two views.Source: Figure 5 · p. 12, §A.1 · p. 12

A.2 MORE TEXT-TO-3D RESULTS

• Additional text-to-3D synthesis results obtained with the authors’ methods are presented in Figures 6–9.Source: §A.2 · p. 12

Ours (DMTet-based)

• The DMTet pipeline integrates aligned geometric priors into Fantasica3D and disentangles geometry learning from appearance generation.It follows Fantasica3D’s original parameters; geometry takes about 12 minutes and PBR materials about 8 minutes per object. The SDS time-step range anneals uniformly from [0.5, 0.98] to [0.05, 0.5].Source: §A.3 · p. 13

Ours (NeRF-based full)

• The NeRF-based pipeline is implemented in threestudio with Instant-NGP as its optimized 3D representation, and sampled camera views follow the render-dataset protocol used to fine-tune the UNet.It uses time annealing, negative prompts, CFG rescaling factor 0.7, and generates each object in about 1 hour with 10,000 steps on 4 V100 GPUs. For SDS, the maximum time step decreases linearly from 0.98 to 0.5, while the maximum time step is kept at 0.02.Source: §A.3 · p. 13

A.4 MORE COMPARISON RESULTS USING PROMPTS FROM MVDREAM

• MVDream also resolves multi-view inconsistency, but the comparison suggests it overfits limited 3D data and compromises the original 2D diffusion model’s generalizability.The authors use the prompts listed on MVDream’s website because its official implementation was unavailable at submission; visual comparisons are shown in Figure 10.Source: §A.4 · p. 13
• MVDream omits the “backpack” from the generated pig prompted to carry one, and its results lack desired photorealism.The authors attribute the appearance limitation to fine-tuning with synthetic multi-view renderings.Source: §A.4 · p. 13

More discussion

• Despite training AGP without regularization, the model retains strong generalizability, as evidenced by successful results.The authors identify a potential risk that fine-tuning could degrade the integrity of geometric priors learned by the pretrained diffusion model, particularly for highly diverse geometries.Source: §A.5 · p. 13
• Figures 6 and 7 show additional generated results from the proposed DMTet-based model.These figures provide further examples of generated results.Source: Figure 6 · p. 14, Figure 7 · p. 15
• Figures 8 and 9 show additional generated results from the proposed NeRF-based model.These figures provide further examples of generated results.Source: Figure 8 · p. 16, Figure 9 · p. 17
• The side-by-side comparison notes that MVDream results omit prompt concepts including a rocket, backpack, and squirrel.The comparison uses prompts from MVDream.Source: Figure 10 · p. 18