PaperlayerView exact revision on arXiv

Source-linked AI summary

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, Jun Zhu

arXiv:2305.16213v2cs.LGcs.CV

TL;DR

Text-to-3D generation with SDS suffers from over-saturation, over-smoothing, and low diversity. ProlificDreamer replaces SDS's fixed 3D parameter with a particle-based variational distribution and adds orthogonal representation and initialization improvements. It yields high-fidelity, diverse NeRFs and photo-realistic textured meshes, while generation remains time-intensive and can fail on complex prompts.

  • Problem

    SDS-based text-to-3D generation suffers from over-saturation, over-smoothing, and low diversity, motivating methods that address these issues.

  • Method

    VSD models the 3D parameter as a random variable and uses particle-based variational optimization, alongside improved distillation scheduling, resolution, and density initialization.

  • Results

    ProlificDreamer generates high-fidelity and diverse 3D results, including 512 × 512 NeRFs with rich structures and complex effects and photo-realistic textured meshes.

  • Takeaways & Limitations

    VSD generalizes SDS while enabling diverse, high-fidelity NeRF and textured-mesh generation within the ProlificDreamer approach.

  • Takeaways & Limitations

    Generation takes hours, camera poses are independent of scene structure, and complex prompts may fail or produce multi-face Janus artifacts.

Abstract

from arXiv · show

Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., $7.5$). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., $512\times512$) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page and codes: https://ml.cs.tsinghua.edu.cn/prolificdreamer/

1 Introduction

Text-to-3D methods aim to reduce the substantial effort required to create premium 3D content. ProlificDreamer introduces VSD and complementary training and initialization choices to produce high-fidelity, diverse scenes, NeRFs, and textured meshes.

  • Premium 3D content remains time- and effort-intensive, motivating methods that automate 3D generation from textual descriptions.
  • VSD models the 3D scene as a random variable and optimizes a distribution whose rendered images align with a pretrained 2D diffusion model.
  • SDS is a special case of VSD using a single-point Dirac distribution, helping explain SDS's restricted diversity and fidelity.
  • At CFG weight 7.5, VSD produces realistic samples, whereas SDS exhibits over-saturation and over-smoothing.
  • ProlificDreamer combines VSD with 512 × 512 training, an annealed distilling-time schedule, and scene initialization for high-fidelity and diverse 3D results.
  • The method generates high-fidelity NeRFs with rich structures and complex effects, complex multi-object scenes in 360◦ views, and detailed photo-realistic textured meshes.

2 Background

Diffusion models learn to generate data by adding and then removing noise, while classifier-free guidance trades off text-image alignment against diversity. SDS distills these models into a 3D optimization procedure but has recognized quality and diversity issues.

  • Diffusion models use a forward process to add noise and a reverse process to denoise and generate data.
  • A trained noise-prediction network approximates the diffusion model's score function, enabling sampling from the learned data distribution.
  • Classifier-free guidance combines conditional and unconditional noise predictions to control text-image generation.
  • Larger guidance scales usually improve text-image alignment but reduce sample diversity.
  • SDS distills a pretrained diffusion model by optimizing a single 3D representation whose rendered views achieve high likelihood under the text condition.
  • Empirical observations report that SDS often suffers from over-saturation, over-smoothing, and low diversity, without a thorough explanation or adequate remedy.

3 Variational Score Distillation

VSD formulates text-to-3D generation as sampling from a distribution of 3D scenes rather than optimizing one scene, using particle-based variational inference and learned score estimation. It explains SDS as a restrictive special case and supports realistic, diverse sampling across CFG settings.

  • 3.1 Sampling from 3D Distribution as Variational Inference: VSD treats the 3D scene as a random variable and optimizes its distribution so rendered images align with the pretrained diffusion model across views.The target alignment is formulated through KL divergence between rendered-image and diffusion-model distributions.
  • 3.2 Update Rule for Variational Score Distillation: VSD represents the target distribution with 3D parameter particles and updates them through Wasserstein gradient flow.The method maintains particles as samples and alternates their optimization with score-model fitting.
  • 3.1 Sampling from 3D Distribution as Variational Inference: VSD introduces diffused intermediate distributions because directly matching the complex target distribution is difficult in high-dimensional spaces.Larger diffusion times make optimization easier as distributions approach a standard Gaussian, while annealing can later recover details.
  • 3.2 Update Rule for Variational Score Distillation: A learned score model, parameterized by a small U-Net or LoRA, estimates noisy rendered-image scores for the VSD update.LoRA is trained on rendered images and can incorporate camera conditioning and the text prompt.
  • 3.3 Comparison with SDS: SDS is a single-point Dirac-distribution special case of VSD, whereas VSD can use multiple particles and learn a parametric score even with one particle.The learned score may provide better updating directions in low-density regions and use text information through LoRA.
  • 3.3 Comparison with SDS: Under the same CFG weight of 7.5, VSD produces realistic 2D samples like ancestral sampling, while SDS fails with over-saturation and over-smoothing.The 2D experiment uses an identity rendering function to isolate the sampling algorithm from 3D representation factors.

4 ProlificDreamer

ProlificDreamer combines VSD with high-resolution rendering, annealed distillation times, and scene initialization in a two-stage NeRF-to-mesh pipeline. These components improve fidelity and details while supporting complex scene generation.

  • 4 ProlificDreamer: ProlificDreamer combines VSD with a high-resolution NeRF stage and optional DMTet mesh extraction and fine-tuning.The NeRF stage targets complex geometry, while the mesh stage adds high-resolution textured details.
  • 4.1 Design Space: Training at up to 512 × 512 rendering resolution is used to obtain high-fidelity NeRFs across resolutions from 64 to 512.Instant NGP enables the high-resolution NeRF rendering used during training.
  • 4.2 3D Representation and Training: Scene initialization uses object-centric density for objects and a hollow density enclosing the camera to support complex scenes.For complex scenes, the specified settings are λσ = −10 and r = 2.5.
  • 4.1 Design Space: An annealed time schedule samples t from U(0.02, 0.98) initially and later narrows it to U(0.02, 0.50).The schedule uses larger times for early optimization and smaller times later to align more closely with the clean target distribution and add details.
  • 4.2 3D Representation and Training: The NeRF ablation shows successive improvements from higher resolution, annealed time scheduling, and VSD over the common 64-resolution SDS setting.Each added component improves the generated result in the elephant-skull comparison.
  • 4.2 3D Representation and Training: For mesh refinement, geometry is optimized with SDS while texture is supervised by VSD with CFG = 7.5 and the annealed schedule.The authors report that this texture optimization provides more details than SDS.

5 Experiments

Experiments show that ProlificDreamer produces diverse, high-fidelity 3D content, while VSD and orthogonal design choices improve quality and diversity over SDS-based settings.

  • VSD generates diverse results from the same text, with different particles producing diverse scenes when using n = 4.
  • Object-centric generation: ProlificDreamer generates higher-fidelity, more detailed 3D objects than DreamFusion, Magic3D, and Fantasia3D.
  • Large scene generation: ProlificDreamer generates high-fidelity 360° scenes with fine details and actual geometry rather than merely textured spheres.
  • Ablation on NeRF training: Increasing rendering resolution, adding an annealed time schedule, and adding VSD each improve generated NeRF results over the 64-resolution SDS baseline.
  • Ablation on mesh fine-tuning: VSD provides higher-fidelity mesh textures than SDS during mesh fine-tuning, although quality depends strongly on the initial NeRF.
  • Ablation on CFG: Smaller CFG encourages diversity; VSD remains effective at small CFG, whereas SDS cannot produce plausible results at CFG 7.5.

6 Related Works

Related work spans CLIP- and diffusion-guided text-to-3D methods, higher-resolution mesh approaches, and text-driven large-scene generation. ProlificDreamer differs through VSD and an annealed time schedule while using only a text-to-image diffusion model.

  • Text-to-3D generation: Prior text-to-3D methods use CLIP guidance, 2D diffusion models, latent-space optimization, or mesh-based representations.
  • Text-driven large scene generation: Text-driven large-scene methods include systems using monocular depth priors and contemporaneous pipelines with different designs.
  • Text-driven large scene generation: ProlificDreamer uses VSD with a two-stage annealed time schedule and only a text-to-image diffusion model, producing high-fidelity NeRFs and photo-realistic meshes.

7 Conclusion

The paper introduces VSD as a variational generalization of SDS and combines it with other 3D-representation improvements. ProlificDreamer produces high-fidelity NeRFs and photo-realistic textured meshes, with additional mesh results shown in Figures 6 and 7.

  • VSD treats the 3D parameter as a random variable and infers its distribution, generalizing SDS within a variational formulation.
  • Combined with orthogonal improvements to 3D representations, ProlificDreamer generates high-fidelity NeRFs and photo-realistic textured meshes.
  • Limitations and broader impact: The limitations include hours-long generation, camera poses independent of scene structure, failures on complex prompts, and occasional multi-face Janus artifacts.
  • Figures 6 and 7 provide additional ProlificDreamer results for 3D textured meshes.
  • Figure 8 compares ProlificDreamer with DreamFusion and reports better NeRF fidelity and detail for ProlificDreamer.

B Additional Experiment Results of 3D NeRF

Additional experiments compare ProlificDreamer with SDS and analyze VSD’s variational formulation, particle updates, and CFG behavior. The results support higher-fidelity NeRFs, multiple 3D samples, and improved sampling relative to SDS.

  • Additional NeRF results: ProlificDreamer generates NeRF results with better fidelity and details than DreamFusion in the reported comparison.The comparison uses NeRF representations and is presented as evidence for VSD’s effectiveness against SDS.
  • Variational formulation: VSD optimizes a distribution of 3D scenes whose rendered images align with a pretrained diffusion-model distribution across views.The objective is formulated through KL divergence in Wasserstein space, rather than optimizing only one parameter point.
  • Comparison with SDS and SJC: VSD supports multiple 3D objects and common CFG values such as 7.5, whereas SDS and SJC optimize one object and typically require CFG around 100.SDS and SJC are represented as the single-point Dirac special case of VSD.
  • Comparison with SDS and SJC: SDS is recovered when VSD uses a single-point empirical distribution, while VSD additionally trains a neural score model to generalize across the underlying scene distribution.The paper attributes poor low-density updates in the single-point approximation to the limited generalization of its score estimate.
  • Sampling behavior: Under the 2D identity-rendering special case and CFG 7.5, VSD produces good samples while SDS produces poor sample quality.The analysis contrasts VSD’s sampling objective with SDS’s mode-seeking behavior.

C.4.2 Proof of Theorem 3

Theorem 3 derives VSD’s Wasserstein gradient flow and connects it to particle updates. The proof uses functional derivatives and the chain rule to express the optimization direction through rendered-image score differences.

  • Proof setup: Particle-based variational inference minimizes KL divergence by evolving particles along a vector field toward a target distribution.The cited proof context identifies this as a standard particle-based variational-inference result.
  • Functional derivative: The functional derivative combines the rendered-image Jacobian with the difference between variational and pretrained diffusion-model scores.The expression uses x_t = α_t g(θ,c) + σ_t ε and differentiates through the rendering function.
  • Theorem 3: The theorem starts from an initial distribution and states a gradient flow that minimizes the VSD energy functional in Wasserstein space.The derivation is framed as the Wasserstein gradient flow of score distillation.

D.3 Ablation on Number of Particles

The particle-count ablation finds slightly greater diversity with more particles while preserving similar quality. Additional ablations examine mesh refinement, resolution, initialization, and implementation choices.

  • Number of particles: Increasing particles from 1, 2, 4, to 8 slightly increases diversity, while generated quality changes little.The VSD CFG is set to 7.5, and the authors note computational overhead when selecting particle counts.
  • Mesh fine-tuning: Textured-mesh fine-tuning further improves quality over NeRF, and VSD texture fine-tuning provides higher fidelity than SDS.Both mesh fine-tuning methods rely on the high-fidelity NeRF produced by VSD.
  • Algorithm: The implementation initializes n 3D structures and a learned noise-prediction model, then renders sampled structures from random camera poses for VSD optimization.The algorithm backpropagates rendered-image gradients into NeRF parameters while training the additional score model.
  • Rendering resolution: Higher NeRF training resolution produces finer details, while VSD remains competitive at 128 or 256 resolution with lower computational cost than 512.The resolution ablation directly compares detail quality and computational efficiency.

G 2D Experiments of Variational Score Distillation

2D experiments show that VSD produces diverse, high-fidelity samples and remains plausible with many particles. Compared with SDS, VSD yields more detailed results and more natural optimization gradients.

  • Scalability: VSD produces plausible 2D results with 2048 particles, demonstrating scalability to a large particle population.The experiment uses a smaller U-Net and reports approximately 20k optimization steps.
  • VSD results: With LoRA, annealed time scheduling, six particles, and CFG 7.5, VSD provides high-fidelity and diverse results.The same experiments also report results with 2048 particles.
  • SDS comparison: Compared with VSD, SDS results are smoother and lack details in the reported 2D comparison.The comparison is shown in the SDS result figure.
  • Gradient visualization: SDS produces over-saturated and over-smooth gradients, whereas VSD produces more natural-looking gradients with more details.The authors connect these gradient differences to better final VSD results.
  • CFG analysis: Smaller CFG increases diversity, but CFG 1 lacks sufficient guidance for plausible results; the 3D experiments therefore use CFG 7.5 as a trade-off.The CFG study compares four VSD particles with repeated SDS generations using different random seeds.

I Limitations and Discussions

The paper identifies limitations in runtime, camera positioning, prompt correspondence, and baseline comparisons, while reporting strong user preference results under constrained evaluation conditions.

  • Runtime: Generation takes hours, particularly for high-resolution NeRF training, and remains much slower than vanilla diffusion-model image generation.The authors leave speeding up text-to-3D generation for future work.
  • Camera positioning: Scene initialization uses a fixed camera view toward the scene center, which may be suboptimal for intricate geometry and detailed textures.The authors suggest improved camera poses as a future direction.
  • Prompt and view limitations: Prompt-to-result correspondence is sometimes insufficient for complex prompts, and the multi-face Janus problem also persists.The authors associate complex-prompt limitations with the Stable Diffusion text encoder and suggest stronger or more 3D-aware diffusion models.
  • Evaluation scope: The user study compares ProlificDreamer with three baselines across 15 prompts using 1,635 pairwise comparisons from 109 volunteers.Because baseline code was unavailable, the study used figures copied from the corresponding papers, limiting the number of comparable results.

K Quantitative Results

Quantitative experiments compare VSD and SDS in 3D and 2D settings, showing that VSD outperforms SDS and that increasing particle counts yields modest additional gains.

  • 3D sample quality: VSD with n=1 outperforms SDS in 3D experiments using 512 resolution and an annealed distilling-time schedule.The 3D evaluations use FID comparisons between rendered samples and diffusion-model samples.
  • Particle ablation: VSD with 4 particles slightly outperforms VSD with 1 particle in 3D, while VSD with 8 particles slightly outperforms VSD with 4 particles in 2D.The comparisons are constrained by time and computational resources.
  • 2D sample quality: VSD obtains much better FID than SDS in 2D experiments that isolate the sampling algorithm from 3D representations.The authors note that VSD remains worse than state-of-the-art diffusion samplers but generalizes to 3D cases.
  • Geometry optimization: VSD can obtain reasonable mesh geometry, but its overall geometry result is similar to SDS when mesh triangles are too large to represent fine details.VSD remains important for quality in the first and third pipeline stages, while geometry is optimized with SDS for efficiency.
  • Baseline comparisons: Additional figures compare ProlificDreamer with baselines, although unavailable baseline code and missing prompts limit direct comparisons.The comparisons use copied baseline figures and different visualization choices for geometry.
Loading 2305.16213v2…