DreamFusion: Text-to-3D using 2D Diffusion

1 INTRODUCTION

• Using a pretrained 2D diffusion model lets DreamFusion learn 3D structure without comparatively scarce 3D training data.Manual asset creation requires substantial time and expertise; text-to-3D generation could lower barriers for novices and improve experienced artists’ workflows.Source: §1 · p. 1
• Earlier GAN-based approaches learned controllable 3D generators from single-category photographs, but had not demonstrated arbitrary-text support.These methods apply adversarial losses to rendered 2D images (Henzler et al., 2019; Nguyen-Phuoc et al., 2019; Or-El et al., 2022).Source: §1 · p. 1
• NeRF (Mildenhall et al., 2020) combines volumetric ray tracing with a neural mapping from spatial coordinates to color and volumetric density.Originally used for reconstruction from multiple scene images, an optimized NeRF can synthesize views from previously unobserved angles.Source: §1 · p. 1
• Dream Fields (Jain et al., 2022) optimizes NeRFs using frozen CLIP image-text embeddings, demonstrating 2D-to-3D transfer but producing objects with limited realism and accuracy.DreamFusion adopts a similar optimization-based approach but replaces CLIP with diffusion-model distillation.Source: §1 · p. 3
• DreamFusion combines Score Distillation Sampling with a task-tailored NeRF variant to generate high-fidelity, coherent 3D objects and scenes from diverse text prompts.The approach replaces Dream Fields’ CLIP guidance with a loss derived from distilling a pretrained 2D diffusion model.Source: §1 · p. 3
• Score Distillation Sampling enables sampling through optimization of differentiable image parameterizations using a probability density distillation loss.The loss minimizes a KL divergence involving shared-mean forward-diffusion Gaussian distributions and score functions learned by the pretrained diffusion model.Source: §1 · p. 3

2 DIFFUSION MODELS AND SCORE DISTILLATION SAMPLING

• Diffusion models reverse progressive noising, with predicted noise defining score functions for smoothed data distributions.Ho et al. (2020) uses a noise-predicting U-Net to approximate the denoiser. Training with a weighted evidence lower bound reduces to weighted denoising score matching (Ho et al., 2020; Kingma et al., 2021).Source: §2 · p. 3
• Classifier-free guidance (Ho & Salimans, 2022) strengthens text conditioning, improving sample fidelity at the cost of diversity.Guidance combines conditional and unconditional noise predictions to favor high conditional-to-unconditional density ratios. Empirically, large guidance weights improve SDS quality.Source: §2 · p. 3, §2.1 · p. 5

2.1 HOW CAN WE SAMPLE IN PARAMETER SPACE, NOT PIXEL SPACE?

• SDS produces detail comparable to ancestral sampling while supporting constrained images and parameter-space transfer learning.Figure 2 illustrates a symmetry-constrained generator, x = (flip(theta), theta), producing images described as having reasonable quality.Source: Figure 2 · p. 4, §2.1 · p. 5
• Differentiable image parameterizations replace pixel sampling with optimization of generator parameters, enabling constraints, compact representations, and 3D rendering.Following differentiable image parameterizations, an image is generated as x = g(theta). For 3D synthesis, theta specifies a volume and g is a volumetric renderer, targeting plausible images from random viewpoints.Source: §2.1 · p. 4
• Directly minimizing the diffusion training loss failed to produce realistic samples, even when optimizing pixels directly.Its U-Net Jacobian is expensive to compute and poorly conditioned at low noise levels; omitting it yields effective optimization updates. Although Graikos et al. (2022) demonstrated success with carefully chosen timestep schedules, the authors found this objective brittle and difficult to tune.Source: §2.1 · p. 4
• Score Distillation Sampling (SDS) optimizes image-generator parameters using diffusion-predicted updates without backpropagating through the frozen diffusion model.It perturbs generated images with random noise and follows diffusion scores toward higher-density regions, implementing the gradient of a timestep-weighted KL distillation loss. The objective is relatively robust to the choice of weighting w(t).Source: §2.1 · pp. 4–5

3 THE DREAMFUSION ALGORITHM

• DreamFusion optimizes a randomly initialized NeRF with score distillation from the unmodified 64 × 64 Imagen base model of Saharia et al. (2022).Each prompt is optimized from scratch without 3D data by sampling a camera and light, rendering, computing SDS gradients, and updating parameters. Renderings alternate between illuminated color, textureless shading, and unshaded albedo.Source: §3 · p. 5, §3.2 · pp. 6–7

3.1 NEURAL RENDERING OF A 3D MODEL

• DreamFusion builds on mip-NeRF 360 (Barron et al., 2022), using density and surface albedo rather than view-dependent radiance for controllable diffuse shading.Surface normals follow normalized negative density gradients (Yariv et al., 2020; Srinivasan et al., 2021). A bounding sphere and separate directional-background MLP prevent near-camera density buildup while allowing a backdrop; smaller bounds can help single objects.Source: §3.1 · p. 6
• Randomly whitening albedo discourages flat, texture-painted geometry, while opacity and orientation penalties constrain empty-space filling and backward-facing normals.The orientation penalty modifies Ref-NeRF (Verbin et al., 2022), preventing normals from turning away from the camera to darken textureless renderings.Source: §3.1 · p. 6

3.2 TEXT-TO-3D SYNTHESIS

• Broad camera-location sampling is critical for coherent 3D scenes, while varied camera distances improve learned scene resolution.Camera elevation spans −10° to 90°, azimuth 0° to 360°, and distance 1 to 1.5; focal length also varies. Look-at points and up vectors are sampled, with point lights distributed around the camera.Source: §3.2 · p. 7
• View-dependent text conditioning adapts prompts to sampled viewpoints instead of describing every rendering with the same canonical view.Above 60° elevation, prompts receive “overhead view”; otherwise, azimuth determines a weighted combination of front-, side-, and back-view text embeddings.Source: §3.2 · p. 7
• SDS uses classifier-free guidance weight 100, with higher guidance improving sample quality and smaller weights producing oversmoothing.Noise timesteps are sampled uniformly from 0.02 to 0.98 to avoid numerical instability at the extremes. The weighting function is w(t) = σ_t^2, although uniform weighting performed similarly.Source: §3.2 · p. 7
• 15,000 optimization iterations take around 1.5 hours on four TPUv4 chips, with compute split evenly between NeRF rendering and diffusion evaluation.Each chip processes one separate view per iteration, and parameters are updated using Distributed Shampoo (Anil et al., 2020).Source: §3.2 · p. 7

Evaluation protocol

• 153 object-centric COCO prompts from Dream Fields support evaluation with CLIP R-Precision, including textureless renders to assess geometry without misleading texture.Following Jain et al. (2022), R-Precision measures whether CLIP (Radford et al., 2021) retrieves the correct caption among distractors. Reference-based metrics are difficult to apply because text prompts have no corresponding ground-truth 3D scene.Source: §4 · p. 8

Benchmark comparison

• 58.5% geometry (Geo) R-Precision contrasts with near-chance reimplemented Dream Fields; DreamFusion also outperforms Dream Fields and CLIP-Mesh on color images.Color-image performance approaches that of ground-truth images, despite the CLIP-based baselines benefiting from using CLIP during training.Source: §4 · p. 8

Ablations

• Albedo scores can conceal poor geometry: the highest-scoring base model produces a multi-headed dog, while textureless optimization smooths surfaces but can carve color details.In the bulldog example, the skull-and-crossbones color pattern becomes carved into the geometry.Source: §4 · p. 8, Figure 6 · p. 9
• +12.5% improvement in full-rendering CLIP R-Precision accompanies progressively broader viewpoints, view-dependent prompts, lighting, and textureless-render optimization, each improving geometry.Accurate geometry requires view-dependent prompts, illumination, and textureless renders.Source: §4 · p. 8, Figure 6 · p. 9

5 DISCUSSION

• DreamFusion outperforms prior text-to-3D work using Score Distillation Sampling and a NeRF-like renderer to transfer pretrained 2D diffusion models without 3D or multi-view training data.The approach supports a wide range of text prompts.Source: §5 · p. 9
• SDS often produces oversaturated, oversmoothed images and less diversity than ancestral sampling, while DreamFusion’s 3D outputs vary little across random seeds.Dynamic thresholding (Saharia et al., 2022) partially helps images but does not resolve these artifacts in NeRFs. The authors suggest reverse KL divergence’s mode-seeking properties may explain limited diversity.Source: §5 · p. 9
• 64 × 64 Imagen limits fine detail in DreamFusion’s synthesized models; higher-resolution diffusion and larger NeRFs would presumably help but make synthesis impractically slow.The authors identify improvements in diffusion and neural-rendering efficiency as a potential route to tractable high-resolution synthesis.Source: §5 · p. 9
• Ambiguous reconstruction from 2D images creates a highly non-convex optimization landscape, sometimes yielding degenerate 3D scenes with all content painted onto one flat surface.Multiple 3D worlds can produce identical 2D images; more robust 3D priors may help address this ambiguity.Source: §5 · p. 9

ETHICS STATEMENT

• DreamFusion inherits potential biases and limitations from Imagen’s image-training data and pretrained language-model conditioning, motivating careful dataset selection to avoid propagating hateful media.Despite partial filtering, Imagen’s LAION-400M (Schuhmann et al., 2022) training subset contained undesirable images (Birhane et al., 2021).Source: ETHICS STATEMENT · p. 10
• 3D generation could enable convincing disinformation and displace creative workers, while also potentially supporting creative-industry growth and accessibility.The authors caution that 3D disinformation may be more convincing than 2D imagery, although DreamFusion renderings remain less realistic than state-of-the-art 2D synthesis.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Imagen is not publicly available, but DreamFusion provides algorithm schematics, SDS pseudocode, hyperparameters, evaluation details, and loss derivations to aid reproducibility.Its mip-NeRF 360 foundation is publicly available through MultiNeRF. Other conditional diffusion models may produce similar results, but this is presented as a possibility.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 PSEUDOCODE FOR ANCESTRAL SAMPLING AND OUR SCORE DISTILLATION SAMPLING.

• Ancestral DDPM sampling accepts an optional conditioning signal y, such as a caption.Source: Figure 7 · p. 15
• The timestep t decreases monotonically during ancestral DDPM sampling.Typical bounds are tmax = 1 and tmin = 1/nstep.Source: Figure 7 · p. 15
• Score Distillation Sampling uses an application-specific generator that differentiably maps parameters to images.Source: Figure 8 · p. 15
• SDS computes the parameter gradient without backpropagating through the diffusion model’s U-Net.Source: Figure 8 · p. 15
• The SDS pseudocode expresses its loss using a stopgradient operator.Source: Figure 8 · p. 15
• The SDS parameter update can alternatively be computed using an explicit vector–Jacobian product.Source: Figure 8 · p. 15

NeRF representation

• DreamFusion builds on mip-NeRF 360 (Barron et al., 2022), using integrated positional encoding to represent ray intervals as Gaussian distributions.The Gaussian mean and covariance approximate each interval’s 3D volume, accounting for ray width and interval length in mip-NeRF (Barron et al., 2021).Source: §A.2 · p. 15

Covariance annealing and optimization

• 5 × 10^-2 to 2 × 10^-3 is the representative covariance-scale annealing schedule over the first 5,000 of 15,000 optimization steps.Rather than deriving covariance from camera geometry, training anneals its scale for coarse-to-fine integrated positional encoding with L = 8. Distributed Shampoo (Anil et al., 2020) uses β1 = β2 = 0.9, exponent override 2, block size 128, SQRT N grafting, and ϵ = 10^-6; learning rate warms linearly from 10^-9 to 10^-4 over 3,000 steps, then cosine-decays to 10^-6, with the long warmup helping geometry coherence.Source: §A.2 · pp. 15–17

MLP architecture changes

• 5 ResNet blocks with 128 hidden units form the NeRF MLP, which predicts density and RGB albedo using exponential and sigmoid activations, respectively.The network uses Swish/SiLU activations and layer normalization between blocks. An added Gaussian density blob centered at the origin concentrates content away from cameras, with representative scale 5 and width 0.2.Source: §A.2 · p. 16

Shading, camera, light, and prompting

• 75% of steps after the first 1,000 use diffuse shading; initial steps disable it, while camera, light, and prompt sampling vary rendering conditions.Diffuse shading uses ambient [0.1, 0.1, 0.1] and diffuse [0.9, 0.9, 0.9], becoming textureless with probability 0.5; elevation sampling equally mixes pole-biased angular and uniform-area half-sphere distributions, with camera-position, look-at, and up-vector perturbations augmenting viewpoints. Light directions are sampled from N(pcam, I), favoring the camera side, with light radius U(0.8, 1.5) and camera radius U(1.0, 1.5); azimuth quadrants select the nearest front/side/back prompt embedding.Source: §A.2 · p. 16

Regularizer hyperparameters

• Orientation and opacity regularization encourage camera-facing visible normals and discourage unnecessary density, with stopped rendering-weight gradients helping prevent object shrinkage or disappearance.The orientation loss follows Ref-NeRF (Verbin et al., 2022), typically annealing its weight from 10^-4 to 10^-2 over 5,000 steps; excessive weight oversmooths surfaces. Opacity regularization improves foreground/background separation, with reasonable weights from 10^-3 to 5 × 10^-3.Source: §A.2 · p. 16

A.3 EXPERIMENTAL SETUP

• 30° elevation is used for DreamFusion’s R-Precision evaluation, averaging over multiple azimuths to reduce variance because it is not prone to CLIP-based evaluation overfitting.CLIP-based systems can overfit R-Precision because their training model is similar to the evaluation model.Source: §A.3 · p. 17
• 45°-elevation held-out views mitigate evaluation overfitting in Dream Fields (Jain et al., 2022) and CLIP-Mesh (Khalid et al., 2022), whose training elevations reach at most 30°.Each baseline evaluates renderings at a single held-out view.Source: §A.3 · p. 17
• 2 generation seeds per caption are used for the main results in Table 1 unless otherwise noted.This evaluation covers all captions.Source: §A.3 · p. 17

• SDS seeks modes shared across diffusion noise levels and yields the diffusion-training update without the diffusion U-Net Jacobian.The loss draws on probability density distillation (van den Oord et al., 2018). Unlike Graikos et al. (2022), SDS avoids costly backpropagation through the diffusion model and uses classifier-free guidance rather than an auxiliary classifier.Source: §A.4 · p. 17
• Fixed-variance forward-process entropy has zero mean-parameter gradient, while the other KL term relates to predicted noise through the chain rule.The entropy gradient can nevertheless be decomposed into score-function and path-derivative terms.Source: §A.4 · p. 17
• Following Sticking-the-Landing (Roeder et al., 2017), SDS retains the entropy’s path derivative, discards its score-function gradient, and uses sampled noise as a control variate.The retained path-derivative term is correlated with the predicted-noise term, enabling variance reduction.Source: §A.4 · p. 17
• Including sampled noise in the gradient reduces variance, speeds optimization, and can improve final results.Source: §A.4 · p. 17
• Entropy-gradient estimation uses one noise level (Lim et al., 2020), while Stein-discrepancy amortized samplers have score-difference-like optimal critics (Hu et al., 2018; Grathwohl et al., 2020).Lim et al. (2020) do not use SDS’s noise-subtraction control variate. SDS could also ease optimization by using multiple noise levels.Source: §A.4 · p. 18

A.5 IMPACT OF SEED AND GUIDANCE WEIGHT

• 100 is a guidance weight used by DreamFusion, versus 5–30 for image synthesis models; even larger weights work without severe artifacts.Large guidance weights are important for learning high-quality 3D models.Source: §A.5 · p. 18
• Sigmoid-bounded MLP colors may explain tolerance for strong guidance: outputs stay within [0, 1], whereas image samplers require clipping.The authors offer this constrained optimization setting as a possible explanation, not an established cause.Source: §A.5 · p. 18
• DreamFusion produces limited diversity across random seeds, possibly because LSDS is mode-seeking and high-noise smoothed densities may contain few distinct modes.The interplay between guidance strength, diversity, and loss functions remains an open research direction.Source: §A.5 · p. 18