PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamFusion: Text-to-3D using 2D Diffusion

Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall

arXiv:2209.14988v1cs.CVcs.LGstat.ML

TL;DR

Creating 3D models from text lacks the large labeled 3D datasets and efficient 3D denoising architectures available for 2D diffusion. DreamFusion uses Score Distillation Sampling to optimize a NeRF with a pretrained 2D diffusion model, producing coherent 3D scenes across diverse prompts and outperforming prior work.

  • Problem

    Text-to-3D synthesis lacks large-scale labeled 3D datasets and efficient architectures for denoising 3D data.

  • Method

    DreamFusion distills a pretrained 2D diffusion model into a NeRF by optimizing differentiable renderings with Score Distillation Sampling.

  • Results

    DreamFusion generates high-fidelity coherent 3D objects and scenes for diverse text prompts, outperforming prior work and achieving 58.5% caption consistency on textureless geometry renders.

  • Takeaways & Limitations

    Pretrained 2D image diffusion models can serve as priors for text-to-3D synthesis without 3D or multi-view training data.

  • Takeaways & Limitations

    SDS often produces oversaturated, oversmoothed, and less diverse results, while 3D outputs can vary little across random seeds.

Abstract

from arXiv · show

Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.

1 INTRODUCTION

DreamFusion addresses the scarcity of labeled 3D data by using a pretrained 2D diffusion model as a prior for optimizing a NeRF from text. Its Score Distillation Sampling loss replaces CLIP-based guidance with probability density distillation from the diffusion model.

  • Motivation: Diffusion models achieve high-quality, diverse, and controllable text-conditioned image synthesis using large aligned image-text datasets and scalable architectures.The passage identifies stable, scalable denoising as a particular strength of diffusion models.
  • Motivation: DreamFusion learns 3D structure from a 2D diffusion model trained on images, avoiding the relatively scarce 3D data required by explicit 3D generative models.The contrast is between plentiful 2D images and comparatively limited 3D data.
  • 3D Representation: Neural Radiance Fields combine volumetric ray tracing with a neural mapping from spatial coordinates to color and volumetric density for inverse rendering.NeRFs were originally effective for reconstructing geometry from many images of a scene.
  • Prior Work: DreamFusion follows Dream Fields’ optimization-based NeRF approach but replaces CLIP with a loss distilled from a 2D diffusion model.Dream Fields demonstrated that pretrained 2D image-text models could support 3D synthesis.
  • Method: Score Distillation Sampling minimizes a KL divergence between Gaussian distributions with shared diffusion-process means and the pretrained model’s learned score functions.This probability density distillation loss enables sampling through optimization in differentiable image parameterizations.

2 DIFFUSION MODELS AND SCORE DISTILLATION SAMPLING

Diffusion models learn score functions through a forward noising process and a reverse denoising process, including text-conditioned models with classifier-free guidance. Score Distillation Sampling uses these learned scores to optimize a differentiable generator in parameter space, enabling sampling beyond directly generating pixels.

  • Diffusion models: Diffusion models gradually add noise to data and learn reverse transitions that remove noise, equivalently learning score functions for increasingly noisy data distributions.The forward marginals interpolate between the data distribution and Gaussian noise, while the reverse model starts from random noise and adds structure.
  • Diffusion models: Text-to-image diffusion models condition their predicted noise on text embeddings and use classifier-free guidance to favor text-consistent regions.Guidance combines conditional and unconditional predictions through a guidance scale parameter ω.
  • Score Distillation Sampling: Rather than sampling pixels directly, the method optimizes parameters θ of a differentiable generator x = g(θ) so its outputs resemble samples from a frozen diffusion model.This optimization follows a DeepDream-like objective in which plausible images have lower loss than implausible images.
  • Score Distillation Sampling: Omitting the expensive and poorly conditioned U-Net Jacobian yields an effective optimization gradient for generator parameters.The Jacobian requires backpropagation through the diffusion U-Net and is poorly conditioned at small noise levels.
  • Score Distillation Sampling: Score Distillation Sampling is the gradient of a weighted probability density distillation loss based on the diffusion model’s learned score functions.The loss perturbs x with timestep-dependent noise and follows the diffusion score toward higher-density regions.
  • Score Distillation Sampling: SDS generates constrained images with reasonable quality, produces detail comparable to ancestral sampling, and supports transfer learning by operating in parameter space.The authors report that large classifier-free guidance weight ω improves quality and that the loss is relatively robust to the weighting function w(t).

3 THE DREAMFUSION ALGORITHM

DreamFusion synthesizes 3D assets by optimizing a randomly initialized NeRF with gradients from a pretrained 2D text-to-image diffusion model. Each iteration samples a camera and light, renders a shaded view, computes SDS gradients, and updates the NeRF.

  • 3.1 Components: DreamFusion combines a pretrained text-to-image diffusion model, a differentiable NeRF image parameterization, and a loss whose minima correspond to good samples.The algorithm uses Imagen’s 64 × 64 base model without its super-resolution cascade.
  • 3.2 Optimization: For each prompt, DreamFusion trains a randomly initialized NeRF from scratch without 3D data through camera-light sampling, rendering, SDS gradient computation, and parameter updates.These four operations define each DreamFusion optimization iteration.
  • 1. Random camera and light sampling: Cameras are sampled in spherical coordinates with elevation φcam ∈[−10°, 90°], azimuth θcam ∈[0°, 360°], and distance from the origin in [1, 1.5].The method also samples a look-at point, an up vector, and a focal length multiplier to construct the camera pose.
  • 2. Rendering: Given each sampled pose and light, DreamFusion renders the shaded NeRF at 64 × 64 resolution and randomly selects illuminated, textureless, or unshaded-albedo output.Textureless shading is produced by randomly replacing albedo with white, helping avoid degenerate texture-on-flat-geometry solutions.

4 EXPERIMENTS

DreamFusion is evaluated for coherent zero-shot 3D scene generation across text prompts, using CLIP R-Precision because reference-based 3D and view-synthesis metrics require unavailable ground truth. Experiments compare against existing methods and progressively ablate viewpoint, prompting, lighting, and textureless-render optimization choices.

  • Evaluation Scope: The experiments assess DreamFusion’s ability to generate coherent 3D scenes from varied text prompts and examine components enabling accurate 3D geometry.The evaluation also explores compositional generation and presents galleries, videos, and meshes.
  • Evaluation Metric: CLIP R-Precision is used because zero-shot text-to-3D prompts lack a corresponding true 3D scene for reference-based metrics such as Chamfer Distance or PSNR.Chamfer Distance compares recovered geometry with ground truth, while PSNR compares rendered views with held-out photographs.
  • Baselines: Table 1 compares DreamFusion with Dream Fields, CLIP-Mesh, an MS-COCO captioned-image oracle, and an enhanced Dream Fields reimplementation using DreamFusion’s 3D representation.Dream Fields and CLIP-Mesh have an evaluation advantage because they use CLIP during training.
  • Ablations: The ablation study progressively adds ViewAug, ViewDep, Lighting, and Textureless optimization choices to a simplified DreamFusion model.CLIP R-Precision is measured on albedo, full shaded, and textureless renders to assess geometry-related effects.
  • Ablations: Without view-dependent prompts, the base method produces a multi-faced dog with flat geometry in the illustrated bulldog-and-pirate-hat example.Figure 6 visualizes ablation effects on albedo, shaded, and textureless renderings using CLIP L/14 on object-centric COCO.

5 DISCUSSION

DreamFusion transfers a pretrained 2D diffusion prior to text-to-3D synthesis without 3D or multi-view training data. The method remains limited by SDS artifacts, missing fine detail, ill-posed optimization, and inherited ethical risks.

  • Contributions: DreamFusion enables text-to-3D synthesis across a wide range of prompts using Score Distillation Sampling and a NeRF-like rendering engine.It uses only a pretrained 2D diffusion model and requires no 3D or multi-view training data.
  • Limitations: SDS often produces oversaturated and oversmoothed results relative to ancestral sampling, and dynamic thresholding only partially ameliorates this issue.The issue remains unresolved when SDS is applied in a NeRF context.
  • Limitations: Using the 64 × 64 Imagen model causes synthesized 3D models to lack fine details, while higher-resolution diffusion and larger NeRFs would make synthesis impractically slow.The paper anticipates that efficiency improvements in diffusion and neural rendering could enable tractable high-resolution synthesis.
  • Limitations: Because many possible 3D worlds can produce identical 2D images, the reconstruction problem is highly ill-posed and creates a highly non-convex optimization landscape.The method’s design specifically aims to sidestep local minima arising from this ambiguity.
  • Ethical concerns: DreamFusion inherits problematic biases and limitations from Imagen, including risks associated with undesirable training images and unwanted biases in its pretrained language model.The paper emphasizes careful attention to the contents of datasets used for generative models.
  • Ethical concerns: Generative 3D models could be used to create more convincing disinformation or displace creative workers, while also potentially improving accessibility and growth in the creative industry.Renderings of DreamFusion’s synthesized models are less realistic than state-of-the-art 2D image synthesis.

A APPENDIX · A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.

The appendix gives pseudocode for ancestral DDPM sampling and Score Distillation Sampling, including timestep scheduling, conditioning, and gradient computation without differentiating through the diffusion U-Net.

  • A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.: Ancestral sampling uses DDPM pseudocode with an optional conditioning signal y, such as a caption.The timestep t decreases monotonically during sampling.
  • A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.: Typically, ancestral sampling sets t_max = 1 and t_min = 1/n_step.These values define the usual timestep range in the pseudocode.
  • A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.: Score Distillation Sampling uses an application-specific generator mapping parameters differentiably to images.The generator provides the parameterized image representation optimized by the sampling procedure.
  • A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.: The Score Distillation Sampling gradient g is computed without backpropagating through the diffusion model’s U-Net.This avoids differentiating through the diffusion network during parameter optimization.
  • A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.: The pseudocode expresses the loss with a stopgradient operator, separating the diffusion-model computation from parameter updates.The stopgradient formulation makes the intended gradient flow explicit.
  • A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.: An explicit VJP can compute the parameter update as g = matmul(weight(t) * (-epshat_t - eps), grad(x, params)).This provides an alternative to the stopgradient formulation for evaluating the update.

A.2 NERF DETAILS AND TRAINING HYPERPARAMETERS

The model builds on mip-NeRF 360 with integrated positional encoding and uses several architectural, shading, sampling, regularization, and optimization choices tailored to DreamFusion training. Key schedules include covariance annealing over the first 5k of 15k steps and a long learning-rate warmup.

  • Architecture: The model builds on mip-NeRF 360 and replaces NeRF’s sinusoidal positional encoding with integrated positional encoding.mip-NeRF 360 is an improved version of NeRF.
  • Training schedules: Covariance scale λΣ is linearly annealed from 5 × 10−2 to 2 × 10−3 during the first 5k of 15k optimization steps.This coarse-to-fine schedule uses integrated positional encoding and is analogous in effect to annealing used by Park et al. (2021).
  • Architecture: The NeRF MLP uses 5 ResNet blocks with 128 hidden units, Swish/SiLU activations, layer normalization, exponential density activation, and sigmoid RGB activation.The exponential activation produces density τ, while the sigmoid activation produces RGB albedo ρ.
  • Shading: Diffuse shading is disabled for the first 1k steps, then enabled with probability 0.75 using ℓa = [0.1, 0.1, 0.1] and ℓρ = [0.9, 0.9, 0.9].When shading is enabled, textureless shading with ρ = 1 is selected with probability 0.5.
  • Regularization and optimization: The method uses Ref-NeRF orientation regularization with stop-gradient rendering weights and opacity regularization, while Distributed Shampoo warms the learning rate from 10−9 to 10−4 over 3000 steps before cosine decay to 10−6.The orientation-loss weight is typically 10−2 and is annealed from 10−4 over the first 5k of 15k steps; the optimizer uses β1 = 0.9, β2 = 0.9, and ϵ = 10−6.

A.4 DERIVING THE SCORE DISTILLATION SAMPLING LOSS AND GRADIENTS

The appendix derives the SDS gradient from probability density distillation, showing that it matches diffusion-loss optimization while omitting the diffusion U-Net Jacobian term. It further motivates including the predicted noise as a control variate, which yields lower-variance gradients and can improve optimization outcomes.

  • Derivation: SDS finds score-function modes shared across diffusion noise levels, and its gradient matches optimizing LDiff without the diffusion U-Net Jacobian term.The derivation is inspired by probability density distillation.
  • Derivation: The second term is related to ˆϵ through the chain rule using sφ(zt|y) ≈∇zt log pφ(zt|y).This connects the score-function formulation to the predicted-noise representation.
  • Derivation: The first gradient term has zero total gradient because the forward process has fixed variance and therefore constant entropy for each t.Its gradient can still be expressed using score-function and path-derivative terms.
  • Variance reduction: Including ϵ as a control variate for ˆϵ follows a sticking-the-landing-style gradient and discards the score-function gradient while retaining the path derivative.The predicted noise is correlated with the other loss term, motivating this variance-reduction construction.
  • Variance reduction: Including ϵ produces lower-variance gradients that speed optimization and can yield better final results.This is the reported practical effect of the modified gradient.
  • Related work: Unlike SDS, the related divergence-based sampler requires costly backpropagation through the diffusion model, while DDPM-PnP uses an auxiliary classifier and SDS uses CFG.These related approaches provide comparisons in optimization cost and conditioning mechanism.

A.5 IMPACT OF SEED AND GUIDANCE WEIGHT

DreamFusion requires unusually large guidance weights to learn high-quality 3D models, yet produces limited diversity across random seeds. The authors attribute this limitation to LSDS’s mode-seeking behavior and smoothed densities with few distinct modes at high noise levels.

  • Guidance weight: DreamFusion uses guidance weights up to ω = 100, with even larger scales working without severe artifacts.This exceeds the ω ∈[5, 30] weights used by image synthesis models.
  • Guidance weight: Large guidance weights are important for learning high-quality 3D models.The authors suggest this may reflect constrained optimization, because the MLP bounds colors to [0, 1] with a sigmoid while image samplers require clipping.
  • Random seed: DreamFusion does not yield large amounts of diversity across random seeds.The authors link this to LSDS’s mode-seeking properties and the possibility that smoothed densities have few distinct modes at high noise levels.
  • Open direction: The relationship among guidance strength, diversity, and loss functions remains an open direction for future research.The paper identifies understanding this interplay as an important unresolved problem.
Loading 2209.14988v1…