PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-to-3D with Classifier Score Distillation

Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song-Hai Zhang, Xiaojuan Qi

arXiv:2310.19415v2cs.CVcs.AIcs.GR

TL;DR

Text-to-3D generation with SDS relies on classifier-free guidance, whose classifier component has generally been treated as auxiliary. The paper introduces CSD, using that component alone, and reports state-of-the-art results across multiple text-to-3D tasks. Its findings motivate a different account of what drives score-distillation optimization in this setting.

  • Problem

    SDS-based text-to-3D methods rely on classifier-free guidance, but its classifier component has been regarded as auxiliary despite its recognized importance in practical optimization.

    Source: §1 · p. 1

  • Method

    CSD performs text-to-3D optimization using only the implicit classifier component of score distillation.

    Source: §1 · p. 3

  • Results

    CSD achieves state-of-the-art results across multiple text-to-3D tasks, including generation, texture synthesis, and editing.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The findings suggest that text-to-3D score distillation draws its efficacy from implicit classifier scores rather than the generative prior, and connect this view to existing techniques.

    Source: §1 · p. 3, §6 · p. 9

    CSD produces artifacts when applied to 2D image optimization, and the authors have not formulated a distribution-based objective for the optimization.

    Source: §6 · p. 9

Abstract

from arXiv · show

Text-to-3D generation has made remarkable progress recently, particularly with methods based on Score Distillation Sampling (SDS) that leverages pre-trained 2D diffusion models. While the usage of classifier-free guidance is well acknowledged to be crucial for successful optimization, it is considered an auxiliary trick rather than the most essential component. In this paper, we re-evaluate the role of classifier-free guidance in score distillation and discover a surprising finding: the guidance alone is enough for effective text-to-3D generation tasks. We name this method Classifier Score Distillation (CSD), which can be interpreted as using an implicit classification model for generation. This new perspective reveals new insights for understanding existing techniques. We validate the effectiveness of CSD across a variety of text-to-3D tasks including shape generation, texture synthesis, and shape editing, achieving results superior to those of state-of-the-art methods. Our project page is https://xinyu-andy.github.io/Classifier-Score-Distillation

1 INTRODUCTION

The introduction reframes classifier-free guidance as central to score-distillation-based text-to-3D generation: its implicit classifier component alone is sufficient for optimization. The paper names this approach Classifier Score Distillation and reports strong results across existing 3D-generation applications.

1 INTRODUCTION

  • SDS uses a pretrained 2D diffusion model to optimize 3D representations toward high-probability image regions conditioned on text, without requiring 3D training data.The introduction describes SDS as producing high-quality, intricate results from diverse prompts.Source: §1 · p. 1
  • Classifier-free guidance combines a data-density gradient with an implicit-classifier gradient, although the latter had been treated as an auxiliary optimization trick.The data-density term moves synthesized images toward high-density regions conditioned on the prompt; the classifier term corresponds to the gradient of the posterior over text given the image.Source: §1 · pp. 1, 3
  • Classifier Score Distillation (CSD) shows that the classifier component of SDS is essential and sufficient for text-to-3D generation, with efficacy attributed to the implicit classifier rather than the generative prior.The paper also connects this interpretation to negative prompts, text-guided editing, and Variational Score Distillation.Source: §1 · p. 3
  • CSD integrates into SDS-based NeRF generation, textured-mesh generation, and texture synthesis pipelines, and the paper reports state-of-the-art results.Figure 1 illustrates text-to-NeRF generation, subsequent mesh refinement, and texture synthesis on user-specified meshes.Source: Figure 1 · p. 2, §1 · p. 3

2 DIFFUSION MODELS

Diffusion models learn data distributions by progressively corrupting samples with noise and then reversing that process. Their denoising objective predicts the added noise, while the resulting score estimates provide directions toward high-density regions.

2 DIFFUSION MODELS

  • The forward process adds Gaussian noise over time to data samples, yielding latent variables whose noise level increases until the distribution approaches a standard Gaussian.The noised sample is expressed as x_t = α_t x + σ_t ε, with ε drawn from a standard normal distribution.Source: §2 · p. 3
  • The reverse process reconstructs the original signal from the terminal latent through a Markov generative process.The paper describes the reverse process as the generative process.Source: §2 · p. 3
  • Training maximizes a variational lower bound on log data likelihood, then uses a denoising network and an MSE objective to predict the noise corrupting the data.The predicted noise is related to the score function of the perturbed data distribution, defined as the log-density gradient with respect to the data point.Source: §2 · p. 3
  • The diffusion score estimates a direction toward high-density regions, a principle used by SDS to optimize a 3D scene.The paper notes that repeatedly moving through high-density regions is expected to yield good-quality samples.Source: §2 · p. 3
  • Classifier-free guidance combines conditioned and unconditioned diffusion estimates, and their difference is proportional to the gradient of an implicit classifier over text given the noisy image.The guidance scale controls a fidelity-diversity trade-off during sampling.Source: §2 · p. 4

3 WHAT MAKES SDS WORK?

SDS optimizes a 3D representation through rendered images, but classifier-free guidance changes the practical optimization direction. The authors find that the classifier score dominates when guidance is large, motivating CSD.

3 WHAT MAKES SDS WORK?

  • SDS uses a loss gradient to optimize a 3D representation through rendered images, with θ denoting the representation parameters and g the differentiable renderer.The optimization seeks modes of score functions across diffusion noise levels.Source: §3 · p. 4
  • The image-space update δx is the difference between the text-conditioned predicted noise and the noise ϵ, encouraging rendered images toward high-density regions conditioned on the prompt.In practice, CFG with a large guidance weight makes this update deviate from the theoretical formulation.Source: §3 · p. 4
  • The update separates into a generative prior associated with the conditional data-density score and a classifier score associated with aligning the image to the text.The classifier score corresponds to the direction from an implicit classifier model.Source: §3 · p. 4
  • With guidance weight ω = 40, the classifier score dominates the optimization direction despite the generative prior having a several-times-larger gradient norm; optimization fails with only the generative component at ω = 0.Figure 2 compares gradient norms during optimization and results at different guidance weights.Source: §3 · p. 5, Figure 2 · p. 5

4 CLASSIFIER SCORE DISTILLATION (CSD)

Classifier Score Distillation uses an implicit classifier derived from text-conditioned diffusion models to update 3D scenes. The section extends this view to negative prompts, text-guided editing, and an interpretation of VSD as using a negative classifier score.

4 CLASSIFIER SCORE DISTILLATION (CSD)

  • CSD updates a 3D scene by aligning rendered images at different noise levels with noise-aware implicit classifiers, using the classifier score rather than the generative prior.The classifier score is defined as the difference between text-conditioned and unconditioned predicted noise.Source: §4 · p. 5
  • Negative prompts turn CSD into a dual objective that pulls rendered images away from unwanted states, but may trade fidelity for quality by causing prompt mismatch.The section also notes that negative prompts can accelerate training and enhance generated results.Source: §4 · pp. 5–6
  • Using separate positive- and negative-prompt weights, gradually decreasing the negative-prompt weight improves texture quality, fidelity, and alignment with the target prompt.The modified formulation is presented as an extension based on the CSD interpretation, not part of original SDS.Source: §4 · p. 6
  • For text-guided 3D editing, CSD uses a target prompt and an edit prompt to modify or remove an attribute while balancing prompt alignment against fidelity to the original scene.Text-to-3D generation is a special case with ω2 = 0 and no specified original attribute to edit.Source: §4 · p. 6
  • VSD can be interpreted as using a negative classifier score that adapts to current rendered images; the authors report CSD with general negative prompts achieves comparable texture quality.The adaptive score is associated with training inefficiency and instabilities.Source: §4 · p. 6

5 EXPERIMENTS

The experiments evaluate CSD on text-guided 3D generation, texture synthesis, and editing, with comparisons to existing methods and user studies for the first two tasks.

5.1 IMPLEMENTATION DETAILS

  • The generation pipeline uses CSD for low-resolution NeRF generation and high-resolution mesh refinement, while texture synthesis applies CSD to a texture field with edge and depth ControlNets.The two generation stages use DeepFloyd-IF stage I and Stable Diffusion 2.1; texture synthesis uses Stable Diffusion 1.5 ControlNets at scale 0.5.Source: §5.1 · p. 7

5.2 TEXT-GUIDED 3D GENERATION

  • CSD significantly outperforms DreamFusion and Magic3D on CLIP Score and CLIP R-Precision; 59.4% of user-study responses prefer its text-to-3D results.Against ProlificDreamer, CSD achieves competitive visual quality in 1 hour on a single A800 GPU, versus 8 hours.Source: Table 2 · p. 8, §5.2 · p. 8, Table 1 · p. 8

5.3 TEXT-GUIDED TEXTURE SYNTHESIS

  • CSD produces detailed, photorealistic textures without seam artifacts commonly seen in TEXTure; 57.7% of texture-study responses prefer its results.The texture task uses 20 diverse meshes from Objaverse.Source: Figure 4 · p. 7, §5.3 · p. 8, Table 1 · p. 8
  • Negative prompts can improve visual quality but may weaken alignment with the original prompt; reducing their weight improves quality while retaining prompt alignment.The described setting uses ω1 = 1 and decreases ω2.Source: §5.3 · pp. 8–9, Figure 5 · p. 9

5.4 TEXT-GUIDED 3D EDITING

  • For existing NeRF scenes, CSD edits prompted attributes while maintaining other attributes, using the portion to edit as a negative prompt.The examples use ω1 = 1 and ω2 = 0.5.Source: §5.4 · p. 9, Figure 6 · p. 9

6 CONCLUSION, DISCUSSION, AND LIMITATION

The paper presents CSD as a framework that achieves state-of-the-art results across multiple tasks and argues that classifier scores are central to score-distillation-based text-to-3D generation. It also identifies unresolved questions about the optimization objective and CSD’s performance on 2D image optimization.

6 CONCLUSION, DISCUSSION, AND LIMITATION

  • CSD achieves state-of-the-art results across multiple tasks and frames classifier scores as potentially the most essential component driving score-distillation optimization.The authors say their results motivate rethinking what works in text-to-3D generation and offer new insights into existing techniques.Source: §6 · p. 9
  • The authors have not formulated a distribution-based objective to guide CSD optimization, despite empirical results showing superior generation quality to SDS.This remains an open question for future study.Source: §6 · p. 9
  • CSD produces artifacts in 2D image optimization, unlike its photo-realistic results in 3D tasks.The authors hypothesize that the difference may arise from implicit fields and multi-view optimization strategies in 3D, and leave the explanation for future work.Source: §6 · p. 9

A.1 RELATED WORKS

The related work describes the development of text-guided 3D generation using pretrained 2D diffusion models and positions CSD as a plug-and-play alternative to SDS. It also situates CSD alongside prior work in generation, editing, and texture synthesis.

A.1 RELATED WORKS

  • SDS, introduced in DreamFusion, distills 3D information from pretrained 2D diffusion models and has become widely used in text-guided 3D generation.Subsequent work explored mesh optimization, multi-view diffusion, timestep annealing, and Variational Score Distillation; VSD improves generation quality and diversity but requires training another 2D diffusion model and has long training time.Source: §A.1 · p. 13
  • Prior work also applies 2D diffusion models to text-guided 3D editing and texture synthesis.The cited examples include Haque et al. (2023), Shao et al. (2023), Zhuang et al. (2023), Richardson et al. (2023), and Michel et al. (2022).Source: §A.1 · p. 13
  • CSD is presented as a plug-and-play replacement for SDS that improves text alignment and appearance realism in 3D generation, with reported strength in texture synthesis and potential for 3D editing.The authors characterize CSD as a new perspective on the SDS optimization process.Source: §A.1 · p. 13

A.2 CSD FOR EDITING

The editing discussion compares CSD with DDS, which uses a reference-image score in its editing formulation. In the reported ablations, removing classifier guidance does not support effective optimization, whereas using the classifier-score component alone can achieve superior results.

A.2 CSD FOR EDITING

  • DDS editing uses an initial image and prompt plus a target prompt, and subtracts the initial image’s SDS score to remove a component described as undesired.The authors say the DDS paper associates relying only on the target score with blurred outputs and loss of detail.Source: §A.2 · p. 13
  • Using only the classifier-score component can achieve superior editing results, while removing that component from DDS does not yield successful optimization.The comparison concerns editing a panda into a squirrel; Fig. 7 compares DDS variants with CSD.Source: §A.2 · p. 14, Figure 7 · p. 14
  • The authors describe CSD’s editing insight as more straightforward than DDS’s and say it does not need a reference image to provide the score treated as undesired.They also state that CSD and DDS are fundamentally different formulations despite their resemblance.Source: §A.2 · pp. 13–14

A.3 ADDITIONAL RESULTS

The supplementary material provides galleries of text-guided 3D generation and texture synthesis results, with comparisons against baseline methods.

A.3 ADDITIONAL RESULTS

  • The supplementary gallery includes 81 text-guided 3D generation results and 20 text-guided texture synthesis results.The authors direct readers to geometry.html and texture.html for the respective comparisons.Source: §A.3 · p. 14
  • The supplementary files include comparisons between the authors’ method and baseline methods for both listed tasks.The passages identify geometry.html for 3D generation comparisons and texture.html for texture synthesis comparisons.Source: §A.3 · p. 14
  • The additional results are presented as video galleries in the supplementary material.The stated gallery coverage is text-guided 3D generation and text-guided texture synthesis.Source: §A.3 · p. 14

Related papers

Loading 2310.19415v2…