PaperlayerView exact revision on arXiv

Source-linked AI summary

Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation

Susung Hong, Donghoon Ahn, Seungryong Kim

arXiv:2303.15413v5cs.CVcs.CLcs.GRcs.LG

TL;DR

Score-distilled text-to-3D generation often suffers from view inconsistency, including the Janus problem, and the paper traces this to bias in 2D diffusion scores. It proposes score clipping and prompt adjustments to debias score-distillation frameworks. The authors report fewer artifacts and improved realism and 3D consistency, while noting limits in Stable Diffusion’s understanding of view-conditioned prompts.

  • Problem

    Score-distilled text-to-3D generation often has view inconsistency, including the Janus problem, while its causes have received limited analysis.

    Source: §1 · p. 2

  • Method

    The paper analyzes score-distillation gradients and combines dynamic score clipping with language-model-based prompt debiasing and adjusted view-prompt ranges.

    Source: §1 · p. 2

  • Results

    The authors report that score debiasing reduces artifacts and improves 3D consistency, while prompt debiasing refines user and view prompts to produce more realistic, view-consistent objects.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The methods aim to balance faithfulness to 2D diffusion models with 3D consistency, without significantly compromising faithfulness or adding substantial overhead.

    Source: §1 · p. 2

    Stable Diffusion’s limited comprehension of view-conditioned prompts constrains outputs for some user prompts despite debiasing.

    Source: §B.1 · p. 16

Abstract

from arXiv · show

Existing score-distilling text-to-3D generation techniques, despite their considerable promise, often encounter the view inconsistency problem. One of the most notable issues is the Janus problem, where the most canonical view of an object (\textit{e.g}., face or head) appears in other views. In this work, we explore existing frameworks for score-distilling text-to-3D generation and identify the main causes of the view inconsistency problem -- the embedded bias of 2D diffusion models. Based on these findings, we propose two approaches to debias the score-distillation frameworks for view-consistent text-to-3D generation. Our first approach, called score debiasing, involves cutting off the score estimated by 2D diffusion models and gradually increasing the truncation value throughout the optimization process. Our second approach, called prompt debiasing, identifies conflicting words between user prompts and view prompts using a language model, and adjusts the discrepancy between view prompts and the viewing direction of an object. Our experimental results show that our methods improve the realism of the generated 3D objects by significantly reducing artifacts and achieve a good trade-off between faithfulness to the 2D diffusion models and 3D consistency with little overhead. Our project page is available at~\url{https://susunghong.github.io/Debiased-Score-Distillation-Sampling/}.

1 Introduction

Zero-shot score-distilled text-to-3D methods face view inconsistency, especially the Janus problem, which is rarely carefully analyzed. The paper traces the issue to bias in estimated scores and proposes score and prompt debiasing.

1 Introduction

  • View inconsistency, especially the Janus multi-face problem, constrains zero-shot text-to-3D methods but has rarely been carefully formulated or analyzed.These methods use score distillation and diffusion models to optimize neural radiance fields without 3D supervision.Source: §1 · p. 2
  • The analysis attributes biased 3D gradients to both the unconditional score and the pose-prompt gradient.It examines how user prompts and camera-dependent view prompts affect score-distilling frameworks.Source: §1 · p. 2
  • Score debiasing dynamically clips scores estimated by 2D diffusion models, gradually increasing the truncation value to reduce artifacts while balancing 3D consistency and faithfulness.The gradual increase follows the coarse-to-fine nature of 3D object generation and improves the consistency trade-off without significantly compromising faithfulness.Source: §1 · p. 2
  • Prompt debiasing uses an MLM-trained language model and point-wise mutual information to identify contradictions between user and view prompts, then adjusts view-prompt ranges to camera pose.The authors report that these changes help text-to-image models predict accurate 2D scores and yield more realistic, consistent structures.Source: §1 · p. 2

2 Background

The background describes diffusion models as denoisers, explains classifier-free text guidance, and reviews score-distillation methods that use 2D diffusion scores to optimize 3D scenes. View prompts are commonly appended to user prompts to support view consistency.

Diffusion models

  • DDPMs generate images through progressive denoising, with inference sampling a previous image state from the current noisy state.The model predicts noise for timesteps t, while x_0 is the original image and x_t its timestep-perturbed version.Source: §2 · p. 2
  • Denoising diffusion models have a tight relationship with continuous score-based models and can be refactored into denoising score matching.With preconditioning, a diffusion model can be converted into a denoiser Dϕ trained for each noise level σ.Source: §2 · pp. 2–3

Diffusion guidance

  • Classifier-free guidance combines conditional and unconditional diffusion-model predictions using a guidance scale s.The conditional prediction is based on the user-given text prompt.Source: §2 · p. 3

Score distillation for text-to-3D generation

  • DreamFusion’s score-distillation sampling uses text-to-image diffusion models to optimize neural fields through a gradient that omits the diffusion backbone’s Jacobian.The omission makes optimization tractable in differentiable parameterizations.Source: §2 · p. 3
  • SJC’s perturb-and-average scoring uses score estimation directly and provides a strong baseline with publicly available Stable Diffusion.Its score estimate is practically obtained by Monte Carlo sampling and plugged into the 2D-to-3D chain rule; the resulting gradient matches SDS with a different weighting rule and sampler.Source: §2 · pp. 3–4

View prompts

  • Score-distillation frameworks commonly concatenate view prompts with user prompts to encourage view consistency, though this practice is rarely discussed.Examples include appending prompts such as “back view of.”Source: §2 · p. 4

3 Score Distillation and the Janus Problem

The section formulates 3D scene scores from 2D diffusion scores across viewpoints and identifies biases in those scores as a source of view inconsistency. It distinguishes unconditional-score bias from pose-prompt guidance that can conflict with the requested view.

3 Score Distillation and the Janus Problem

  • The 3D parameter density is defined as a product of conditional 2D image likelihoods over uniformly sampled viewpoints, whose log-gradient is estimated using diffusion-model scores.This generalizes SJC and DreamFusion formulations, while avoiding the use of Jensen’s inequality.Source: §3 · p. 4
  • The unconditional score can bias typical-view renders early in optimization, contributing to Janus artifacts when facial views dominate some objects’ 2D data.Source: §3 · p. 4
  • Pose-prompt guidance steers renders toward a camera pose and user prompt, but contradictory conditions can make its gradient terms adverse to view-consistent optimization.Source: §3 · p. 4

4 Methodology

The methodology debiases score-distillation through dynamic clipping of 2D scores and prompt adjustments that address conflicts between user descriptions and view directions.

4.1 Score debiasing

  • The magnitude of the estimated score reflects scaled deviation between the rendered image and the denoised image, motivating direct clamping to suppress large deviations.The authors relate such deviations to artifacts that can ignore geometry or appearance.Source: §4.1 · p. 5
  • Dynamic score clipping limits erroneous 2D scores and artifacts, while increasing the threshold during optimization balances early structural consistency against later detail and 2D fidelity.Naive clipping trades 2D fidelity for artifact reduction as its threshold decreases.Source: §4.1 · pp. 5–6, Figure 3 · p. 5

4.2 Prompt debiasing

  • Contradictions between view and user prompts can make diffusion models follow a user word such as “smiling” instead of the requested view.Source: §4.2 · p. 6, Figure 4 · p. 6
  • The method uses masked-language-model probabilities to identify low-PMI user-prompt words for removal, and adjusts view-prompt regions, including halving the front-view azimuth range.PMI is computed from view-prompt and user-word probabilities; words below a threshold are omitted in practice.Source: §4.2 · pp. 6–7

5 Experiments

The experiments evaluate view consistency, prompt faithfulness, and quality, and report that the debiasing methods improve these outcomes while reducing artifacts. Ablations show that prompt and score debiasing contribute differently, while dynamic clipping balances consistency with visual detail.

5.1 Implementation details

  • The experiments use 10,000 optimization steps and take about 20 minutes on one NVIDIA 3090 RTX GPU, with almost no overhead over the baseline.The SJC hyperparameters remain fixed throughout the experiments.Source: §5.1 · p. 7

5.2 Evaluation Metrics

  • A-LPIPS averages perceptual differences between adjacent rendered views, while azimuth-wise CLIP similarities assess faithfulness to view-augmented prompts.The evaluation renders 100 views from uniformly spaced camera poses on an upper hemisphere; larger adjacent-view perceptual loss indicates artifacts or inconsistency near those views.Source: §5.2 · p. 7

5.3 Comparison with the baseline

  • Debiased results show stronger view-prompt alignment by azimuth, and the user study favors the method over the baseline across all human-evaluation criteria.The study evaluates view consistency, faithfulness to user prompts, and overall quality; the prompt-alignment graph shows peak similarity in the desired view region.Source: §5.3 · pp. 8–9, Table 2 · p. 8
  • The combined method produces more consistent 3D objects than the SJC baseline [27], and significantly outperforms it on Janus-free success across prompts with countable faces.The success-rate evaluation covers 41 of 70 prompts with countable faces; objects count as successful when they have the accurate number of faces and no Janus problem.Source: §5.3 · pp. 8–9, Table 3 · p. 9
  • Qualitative comparisons show fewer Janus artifacts than SJC [27] and Stable-DreamFusion [18], including more consistent giraffe bodies and single-handle mugs.The method is also reported to apply to DreamFusion [18], Magic3D [10], and ProlificDreamer [28].Source: Figure 6 · p. 8, §5.3 · p. 9

5.4 Ablation study

  • Prompt debiasing alone can reduce multi-face artifacts but may leave or add ears; combining it with score debiasing removes those extra ears in the reported examples.Sequentially adding prompt and score debiasing improves view consistency and reduces artifacts.Source: §5.4 · p. 9
  • With static clipping, lowering the threshold removes more artifacts but yields pixelated, collapsed appearances; dynamic clipping eliminates artifacts, preserves intricate details without pixelation or degradation, and closely matches low-threshold static consistency.Source: §5.4 · p. 10, Figure 8 · p. 10

6 Conclusion

The conclusion presents dynamic score debiasing and prompt debiasing as approaches to reducing view inconsistency and artifacts in zero-shot text-to-3D generation. Appendix examples report that debiasing mitigates artifacts in DreamFusion results.

6 Conclusion

  • The paper concludes that dynamic score debiasing reduces artifacts and improves 3D consistency, while prompt debiasing refines user and view prompts for more realistic, view-consistent objects.The conclusion frames both methods as addressing view inconsistency, particularly the Janus problem, in zero-shot text-to-3D generation.Source: §6 · p. 10

Appendix

  • In debiased DreamFusion [18] examples, baseline artifacts such as additional necks, handles, and faces are mitigated.The examples use the threestudio [4] DreamFusion implementation with DeepFloyd-IF.Source: Figure 9 · p. 13, Figure 10 · p. 13

A.1 Qualitative results

Additional qualitative examples report improved view consistency, while clipping comparisons illustrate how dynamic clipping reduces artifacts without sacrificing object structure. A user study finds that the method outperforms the baseline across its human-evaluation criteria.

A.1 Qualitative results

  • Across sequential camera views, debiasing improves consistency when a baseline front view reappears in back or side views; an ablation example also attributes improved realism and consistency to both debiasing methods.Even when Janus artifacts occur, smooth transitions can make individual views appear reasonable.Source: §A.1 · pp. 13–14, Figure 12 · p. 14

A.2 Dynamic clipping of 2D-to-3D scores

  • Gradually reducing the clipping threshold preserves the main object while eliminating artifacts, unlike low static clipping, which can make necessary objects such as icebergs transparent.No clipping and high-threshold static clipping show artifacts in some views.Source: §A.2 · p. 14, Figure 14 · p. 15

A.3 User study

  • In a 75-participant user study, the debiased method surpassed the baseline in view consistency, faithfulness to user prompts, and overall quality.The study compares the baseline and debiased results.Source: §A.3 · pp. 14–15, Figure 13 · p. 15

A.4 Results on other text-to-3D frameworks

The method is presented as applicable to score-distillation text-to-3D frameworks that use text-to-image scores and view-augmented prompts. Examples across several frameworks show improvements, while optimization speed remains nearly unchanged.

A.5 Visualization of optimization process and convergence speed

  • Geometry is mostly formed during Magic3D's first 4,000 optimization steps, and debiasing leaves convergence speed nearly unchanged at approximately 20 minutes for SJC and the proposed method.The optimization-step counts are comparable to the baseline.Source: Figure 15 · p. 16, §A.5 · p. 16

A.4 Results on other text-to-3D frameworks

  • Score debiasing alone improves results on DreamFusion [18], Magic3D [10], and ProlificDreamer [28], including examples with complex geometries susceptible to Janus artifacts.The comparisons use the same seed and the threestudio implementations.Source: §A.4 · pp. 15–16

B.1 Limitations

The authors attribute constrained outputs for some prompts to Stable Diffusion’s limited comprehension of view-conditioned prompts. They describe the strategy as applicable to SDS methods that use view prompts and report no identified deleterious impact.

B.1 Limitations

  • Stable Diffusion’s limited comprehension of view-conditioned prompts constrains results for some prompts, even with the debiasing methods.The failure cases occur where Stable Diffusion has a severely limited ability to generate view-conditioned images.Source: §B.1 · p. 16, Figure 16 · p. 16

B.2 Broader impact

  • The authors say the debiasing strategy can be integrated into SDS techniques that deploy view prompts.They cite this as a capability of the strategy, given that those methods universally use view prompts.Source: §B.2 · p. 17
  • The procedure is not identified as having deleterious impact because it is designed to debias the existing framework.Source: §B.2 · p. 17

C Implementation Details

The implementation uses fixed SJC hyperparameters and 10,000 optimization steps, with separate schedules for score clipping and prompt debiasing. Prompt debiasing uses BERT-based PMI and modified view-prompt assignments.

C.1 Common settings

  • Both SJC and the proposed methods use 10,000 optimization steps with fixed SJC hyperparameters; Stable Dreamfusion is omitted from prompt comparisons because background-only outputs could skew metrics.The prompts used in the experiments are listed in Table 4.Source: §C.1 · p. 17, Table 4 · p. 18

C.2 Score debiasing

  • For Stable Diffusion, the score-clipping threshold increases linearly from 2.0 to 8.0; for Deepfloyd-IF, it increases from 0.5 to 2.0.Both schedules linearly increase from one-fourth of the predefined threshold to the predefined threshold over optimization steps.Source: §C.2 · p. 17

C.3 Prompt debiasing

  • BERT [1] supplies conditional probabilities for PMI; words below 0.95 normalized PMI for a view prompt are removed, with P(u) set to 1 or 1/2 based on omission risk.The PMI evaluation prepends “This image is depicting a” to the view and user prompts.Source: §C.3 · p. 17

Related papers

Loading 2303.15413v5…