Noise-Free Score Distillation

1 INTRODUCTION

• SDS uses pretrained text-to-image diffusion models to optimize NeRFs and other representations, including meshes, texture maps, fonts, and SVG.Source: §1 · p. 1
• SDS-generated results tend to be over-smoothed and over-saturated, with limited fine detail, and the loss remains intriguing and not fully understood.Source: §1 · p. 1
• The score decomposition motivates NFSD, which the authors report outperforms SDS without additional computational costs.The decomposition separates condition alignment, domain correction, and denoising.Source: §1 · p. 1
• The proposed framework also provides a more concise explanation of VSD and DDS, methods that have shown improvements over SDS.Source: §1 · p. 1
• NFSD addresses some issues present in SDS and leads to improved visual results for images and NeRFs using Stable Diffusion.Figure 1 shows two learned NeRFs and a gallery of images optimized with NFSD.Source: §1 · p. 2, Figure 1 · p. 2

2 BACKGROUND

• Diffusion models predict noise, and their predictions are proportional to a score function that points toward higher data density at a given noise level.Source: §2 · p. 2
• CFG extrapolates the conditioned prediction away from the unconditioned one to steer toward the condition, but trades sample fidelity for diversity.Source: §2 · pp. 2–3
• SDS leverages pretrained text-to-image models to support text-conditioned generation in domains such as 3D, where comparable extensive training data are unavailable.Source: §2 · p. 3
• SDS optimizes a differentiable generator by adding timestep-specific noise to its rendered image and backpropagating the loss gradient through the generator.Source: §2 · p. 3
• The SDS gradient moves a noised rendered image toward a condition-aligned dense region, then subtracts the added noise to adapt the direction to clean images.Source: §2 · p. 3
• SDS can be over-saturated and lack realistic fine detail; related approaches include time annealing, VSD, HiFA, and DDS, while the paper proposes a three-component score interpretation.Source: §2 · p. 3

3 SCORE DECOMPOSITION

• The condition direction δC is the difference between conditioned and unconditioned noise predictions, steering generated images toward alignment with the condition.It is empirically observed to be uncorrelated with added noise and concentrated around condition-specific image regions across timesteps.Source: §3 · p. 4, Figure 2 · p. 4
• The unconditional prediction is decomposed into domain correction δD, which is expected to correlate with rendered-image content, and denoising δN, which is not.Matched-noise in-domain and out-of-domain examples visualize δN and δD, respectively; adding δD to an out-of-domain image improves its realism.Source: §3 · p. 4, Figure 3 · p. 5
• The three components yield a score comprising δD, δN, and sδC, with δC visualized as clean and concentrated around the main object.The score function is proportional to the gradient of the log density of samples noised to timestep t.Source: §3 · pp. 3, 5, Figure 2 · p. 4
• The SDS direction includes a noisy residual, δN − ϵ, that becomes more pronounced at smaller timesteps and can drive optimization in varying directions, averaging away fine and medium details.The authors note that higher-level semantics are less affected and cite prior observations linking subtraction of ϵ to blurry results.Source: §3 · p. 5, Figure 4 · p. 5
• Large CFG scales let the image-correlated sδC term dominate the noisy residual, but high scales are associated with oversaturated, less realistic results.Figure 5 compares ancestral samples made with the same prompt and seed at different CFG values.Source: §3 · pp. 5–6, Figure 5 · p. 6

4 NOISE FREE SCORE DISTILLATION

• For t < 200, NFSD approximates δD with the unconditional prediction because the denoising component is small enough to be negligible.This threshold is reported for the authors’ experiments.Source: §4 · p. 6
• For t ≥ 200, NFSD estimates δD by subtracting the prediction for a negative prompt from the unconditional prediction, assuming δC for that prompt is approximately −δD.The negative prompt describes unrealistic, blurry, low-quality, out-of-focus, ugly, low-contrast, dull, dark, low-resolution, and gloomy results.Source: §4 · p. 6
• The estimated δD and condition direction δC define the NFSD loss, whose gradients replace SDS gradients when optimizing θ.The approach aims to use δD and sδC while excluding the noisy denoising residual.Source: §4 · p. 6
• NFSD yields noticeable improvements in generated image and NeRF quality while using CFG s = 7.5 rather than SDS’s s = 100.The authors attribute the lower CFG requirement to approximately eliminating δN, so δC no longer needs a large scale to dominate.Source: §4 · p. 6

• The score decomposition provides a framework for explaining prior methods proposed to improve SDS.The paper applies it to DDS and VSD.Source: §5 · p. 6

DDS

• DDS cancels the residual component linked to SDS’s over-smoothed and over-saturated results, while its in-domain initialization also makes the domain-correction component unnecessary.The remaining direction corresponds to the difference between the original and edited prompts.Source: §5 · p. 7

ProlificDreamer

• VSD’s fine-tuned diffusion model approximates the noise component, leaving the same terms as NFSD, but requires additional diffusion-model fine-tuning.The model is fine-tuned on rendered images during optimization.Source: §5 · p. 7

Implementation

• The experiments use Stable Diffusion 2.1-base; the described 3D models run for 25,000 iterations, with rendering resolution increased from 64×64 to 512×512 after 5,000 iterations.The paper specifies AdamW with learning rate 0.01 for the 3D optimization.Source: §6 · p. 7

3D generation

• NFSD-optimized NeRFs are described as sharp and highly intricate.The paper also presents additional NFSD NeRF examples in Figure 6.Source: §6 · p. 7

2D image generation

• At CFG 7.5, NFSD produces clearer objects, more detailed backgrounds, and more realistic images than SDS at CFG 7.5 or 100.The comparison describes artifacts in SDS-100 dog fur and less detailed, painted-looking fire; NFSD-7.5 shows more realistic fur and more detailed fire.Source: §6 · p. 8, Figure 7 · p. 8

Text-to-NeRF synthesis

• NFSD produces more detailed NeRF results with ordinary CFG values by attempting to eliminate noise and explicitly approximate δD.The comparison omits diffusion-time annealing, which the paper says slightly hinders output quality but makes method differences clearer.Source: §6 · p. 8, Figure 8 · p. 9

Comparison with related methods

• NFSD achieves comparable or better results than the related methods and is simpler to implement and faster than ProlificDreamer.The paper notes that the compared methods differ in implementation details and guiding diffusion models; Appendix A.4 compares results using threestudio for all methods.Source: §6 · pp. 8–9

7 CONCLUSION AND FUTURE WORK

• NFSD uses a three-component score interpretation to prevent noise distillation, requiring minimal SDS changes and a nominal CFG scale.The paper reports improvements over SDS and existing approaches for 3D object generation.Source: §7 · p. 9
• The Janus problem and low diversity remain drawbacks of SDS, including after NFSD improves score distillation.The authors suggest more principled noise scheduling might improve diversity; they describe diffusion-time annealing as helpful.Source: §7 · p. 9
• Text-generated NeRF evaluation lacks suitable quality metrics and benchmarks for comprehensive comparisons across methods.The authors identify developing such evaluation tools as future work.Source: §7 · p. 9

A.1 VISUALIZATION OF SCORE COMPONENTS

• Stable Diffusion latent visualizations are decoded to RGB, and brown denotes a zero latent code.The paper says a pixel’s RGB value can be approximated by a fixed linear transformation of the corresponding latent pixel.Source: §A.1 · p. 12
• Figure 3 visualizes δD and δN using DDIM-generated in-domain cat and cow images and filtered out-of-domain versions.The cat image is Gaussian-filtered and the cow image bilateral-filtered.Source: §A.1 · p. 12

A.2 PROMPTS USED IN THE PAPER

• The passage reports the prompts used in several figures, presented in a left-to-right, top-to-bottom order.Source: §A.2 · p. 12
• Figure 6’s prompts include a hedgehog, an iron soldier pen holder, an elephant statue, sunflowers, a dragonfly statue, and a parrot.The listed prompts specify varied objects and materials.Source: §A.2 · p. 12
• Figure 8’s prompts include an eagle catching a snake, a fairy angel, a gargoyle, a running horse, a Hystrix, and a phoenix.The listed prompts describe varied figures and scenes.Source: §A.2 · p. 12

A.3 ADDITIONAL RESULTS

• Figures 10 and 11 present additional text-to-NeRF results optimized with NFSD.Their prompts include zoomed-out DSLR views.Source: §A.3 · p. 12, Figure 10 · p. 12, Figure 11 · p. 13

A.4 MORE COMPARISONS WITH RELATED METHODS

• Under a unified threestudio implementation using Stable Diffusion 2.1-base on one GPU, NFSD is reported comparable or better than the compared methods, with high resolution and detailed features.The authors note implementation differences; Fantasia3D’s reported results used eight GPUs, while this comparison used one.Source: §A.4 · p. 14, Figure 12 · p. 14
• Figures 13–17 add comparisons using results reported in the methods’ original papers.The appendix does not state specific outcomes for these additional comparisons in the supplied passage.Source: §A.4 · p. 14, Figure 13 · p. 15, Figure 14 · p. 15