PaperlayerView exact revision on arXiv

Source-linked AI summary

LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching

Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, Yingcong Chen

arXiv:2311.11284v3cs.CVcs.GRcs.MM

TL;DR

Text-to-3D generation often produces over-smoothed, low-detail models because SDS supplies inconsistent and low-quality guidance. The paper introduces ISM with invertible diffusion trajectories and interval-based matching, and combines it with 3D Gaussian Splatting. LucidDreamer reportedly produces detailed results, surpasses state-of-the-art methods, and does so without extra computational costs or increased training burden.

  • Problem

    SDS provides inconsistent and low-quality pseudo-ground truths, causing feature averaging and over-smoothed 3D results.

  • Method

    ISM uses DDIM inversion and interval-based score matching, while LucidDreamer integrates ISM with 3D Gaussian Splatting.

  • Results

    LucidDreamer significantly surpasses current state-of-the-art methodologies, while ISM yields highly detailed results without extra computational costs.

  • Takeaways & Limitations

    ISM supports high-fidelity 3D generation and editing across NeRF and 3D Gaussian Splatting, as well as 2D editing tasks.

Abstract

from arXiv · show

The recent advancements in text-to-3D generation mark a significant milestone in generative models, unlocking new possibilities for creating imaginative 3D assets across various real-world scenarios. While recent advancements in text-to-3D generation have shown promise, they often fall short in rendering detailed and high-quality 3D models. This problem is especially prevalent as many methods base themselves on Score Distillation Sampling (SDS). This paper identifies a notable deficiency in SDS, that it brings inconsistent and low-quality updating direction for the 3D model, causing the over-smoothing effect. To address this, we propose a novel approach called Interval Score Matching (ISM). ISM employs deterministic diffusing trajectories and utilizes interval-based score matching to counteract over-smoothing. Furthermore, we incorporate 3D Gaussian Splatting into our text-to-3D generation pipeline. Extensive experiments show that our model largely outperforms the state-of-the-art in quality and training efficiency.

1 HKUST (GZ) 2 HKUST 3 Zhejiang Lab 4 Zhejiang University

The paper presents LucidDreamer as a framework for distilling high-fidelity 3D textures and shapes from pretrained 2D diffusion models.

  • LucidDreamer combines Interval Score Matching with an advanced 3D distillation pipeline.The framework is designed to distill high-fidelity textures and shapes from pretrained 2D diffusion models.
  • The authors report photorealistic 3D generation results with short training time.
  • The paper identifies Yixun Liang and Xin Yang as contributing to conceptualization and methodology.The listed conceptualization contributions are 60% and 40%, while methodology contributions are 60% and 40%, respectively.

1. Introduction

Text-to-3D generation uses diffusion-model guidance but SDS can produce inconsistent, low-quality pseudo-ground truths that average away detail. ISM addresses this through invertible trajectories and interval-based matching, while LucidDreamer combines ISM with 3D Gaussian Splatting for efficient, high-quality generation.

  • Motivation: High-quality 3D content generation remains time-consuming, labor-intensive, and dependent on skilled expertise.The paper motivates text-to-3D generation as a response to the broad use of digital 3D assets.
  • Text-to-3D Generation: Text-to-3D methods use pretrained text-to-image diffusion models as image priors to supervise neural 3D models from text descriptions.The goal is to render 3D-consistent images aligned with the text.
  • Limitations of SDS: SDS produces over-smoothed, detail-deficient results because its pseudo-ground truths are inconsistent and visually low quality.Applying their varying update directions to the same 3D model creates an averaging effect.
  • Interval Score Matching: ISM uses DDIM inversion to create an invertible diffusion trajectory and matches two interval steps instead of relying on one-step pseudo-ground-truth reconstruction.These mechanisms target pseudo-ground-truth inconsistency and high reconstruction error.
  • LucidDreamer: Integrating ISM with 3D Gaussian Splatting yields state-of-the-art performance with less training cost than multi-stage competitors.The reported comparisons include Magic3D, Fantasia3D, and ProlificDreamer.

2. Related Works

Prior text-to-3D work progressed from CLIP-guided NeRF optimization to diffusion-based SDS, while alternative improvements often required longer training. The paper positions ISM as a systematic analysis and efficient alternative, supported by differentiable 3D representations.

  • Text-to-3D Generation: DreamField pioneered text-to-3D distillation by training NeRF with CLIP guidance, but its results were unsatisfactory because CLIP supervision was weak.
  • Text-to-3D Generation: DreamFusion introduced SDS to distill 3D assets from pretrained 2D text-to-image diffusion models.SDS facilitates 3D distillation by seeking modes in a text-guided diffusion model.
  • SDS Improvements: Concurrent analyses such as CSD and NFSD sought empirical improvements to SDS, while other approaches required much longer training stages.
  • SDS Improvements: This work differs by systematically analyzing inconsistent, low-quality SDS pseudo-ground truths and introducing ISM without increasing computational burden.
  • Differentiable 3D Representations: Differentiable 3D representations use trainable parameters and differentiable rendering to optimize rendered images against conditioning signals.The rendering equation g(θ, c) produces an image from representation parameters θ and camera pose c.

3. Methodology

LucidDreamer replaces SDS’s inconsistent, low-quality pseudo-ground-truth guidance with deterministic DDIM inversion and interval score matching, then uses an efficient 3D Gaussian Splatting pipeline for high-fidelity text-to-3D generation.

  • Revisiting SDS: SDS can produce inconsistent, feature-inconsistent, blurry pseudo-ground truths because random noise and camera poses perturb diffusion inputs and single-step predictions lose detail.These guidance signals are averaged during optimization, leading to over-smoothed 3D results.
  • DDIM Inversion: LucidDreamer uses DDIM inversion to generate deterministic noisy latent trajectories from rendered views, increasing pseudo-ground-truth consistency with the original views.The invertible trajectory provides latents such as xs and xt for subsequent interval-score computation.
  • Interval Score Matching: A naive multi-step denoising alternative could improve guidance quality but is too time-consuming, motivating ISM’s more efficient interval-based formulation.ISM retains the benefit of multi-step reasoning while avoiding the full computational cost of repeatedly generating denoised pseudo-ground truths.
  • Interval Score Matching: ISM replaces SDS’s single-step pseudo-ground-truth matching with interval score matching between noisy latents generated through DDIM inversion.The method disregards the bias term ηt because neighboring interval scores are expected to cancel, improving efficiency without compromising distillation quality.
  • Advantages: ISM provides consistent, high-quality pseudo-ground truths that yield detailed, fine-structured outputs with marginal computational overhead compared with SDS-based alternatives.The method also avoids the multi-stage training required by cited competitors while maintaining a simple training pipeline.
  • The Advanced Generation Pipeline: The advanced pipeline combines 3D Gaussian Splatting with text-to-point initialization to support efficient high-resolution rendering and better initialization for non-axial-symmetric objects.3DGS is selected for efficient rendering and optimization, while text-to-point models provide richer starting geometry than simple boxes, spheres, or cylinders.

4. Experiments

Experiments compare LucidDreamer and ISM across representations, hyperparameters, initialization, and applications. The results emphasize detailed, realistic generation, user preference, and a tradeoff between structure, detail, and computational cost.

  • Text-to-3D Generation: LucidDreamer generates semantically consistent 3D content with realistic, intricate appearances while avoiding excessive smoothness and over-saturation.The examples include detailed character portraits, hair textures, common objects, and creative concepts.
  • Generalizability of ISM: ISM provides fine-grained details with both NeRF and 3D Gaussian Splatting under normal CFG, outperforming SDS qualitatively.The comparison is presented as evidence of ISM’s generalizability across explicit and implicit 3D representations.
  • User Study: Users consistently favored models generated by LucidDreamer in a study averaging preference rankings across 28 text-to-3D result sets.The table caption states that smaller rankings are better and that the proposed result was preferred by most users.
  • Applications: ISM supports zero-shot 2D and 3D editing, personalized text-to-3D generation with LoRA, and 3D avatar generation.The applications figure presents these tasks as examples of the framework’s broader use.
  • Ablation Studies: Increasing δT produces more natural colors and simpler structures but reduces detail, whereas smaller δT values increase detail alongside visual anomalies.The authors describe this as a tradeoff and propose annealing the interval from coarse structure toward fine-grained features.
  • Ablation Studies: Increasing δS has marginal influence on results for fixed δT while significantly reducing the computational cost of DDIM inversion.Figure 6 qualitatively compares the effects of δT and δS.

5. Applications

LucidDreamer extends its framework to pose-specific avatar generation by combining a geometry prior with DensePose-conditioned supervision. This produces high-fidelity avatars from advanced control signals.

  • Zero-shot Avatar Generation: SMPL initializes the 3D Gaussian point cloud as a geometry prior for pose-specific avatar generation.The framework uses the Skinned Multi-Person Linear Model to initialize the explicit 3D representation.
  • Zero-shot Avatar Generation: ControlNet conditioned on DensePose signals provides robust supervision for rendering high-fidelity avatars.The human mesh is rendered into 2D and passed through a pretrained DensePose model to obtain body-part segmentation conditions.
  • Zero-shot Avatar Generation: The pipeline renders sampled-camera human meshes with PyTorch3D before generating DensePose conditioning maps.These maps are then supplied to the pretrained DensePose model as control signals.

6. Conclusions

The paper attributes SDS over-smoothing to inconsistent, low-quality pseudo-ground truths and introduces ISM for more reliable guidance. Combined with 3D Gaussian Splatting, LucidDreamer achieves detailed results efficiently across diverse applications.

  • Conclusions: SDS over-smoothing is attributed to inconsistent and low-quality pseudo-ground truths that provide unreliable updating directions.The authors identify this issue as the root cause of the over-smoothing effect.
  • Conclusions: ISM addresses the problem by providing consistent and reliable guidance for 3D generation.The method is introduced as the paper’s response to SDS’s deficient guidance.
  • Conclusions: ISM overcomes over-smoothing and yields highly detailed results without extra computational costs.The conclusion also reports compatibility with NeRF, 3D Gaussian Splatting, 3D editing, and 2D editing.
  • Conclusions: LucidDreamer combines ISM with 3D Gaussian Splatting and significantly surpasses current state-of-the-art methodologies.The reported applications include text-to-3D generation and editing, zero-shot avatar creation, and personalized text-to-3D conversion.

7. Appendix

The appendix details LucidDreamer’s 3DGS optimization, classifier-free guidance, and analyses of SDS pseudo-ground-truth inconsistency. It also reports faster convergence for ISM than SDS.

  • Implementation Details: LucidDreamer optimizes an explicit 3D Gaussian Splatting representation with ISM while densifying and pruning Gaussians every 300 iterations through 3000 iterations.The implementation follows most training hyperparameters from the original 3DGS paper.
  • Implementation Details: Classifier-free guidance combines positive and negative prompt predictions using guidance scale gs.The appendix denotes y and y_n as the positive and negative prompts and defines gs as the guidance scale of y.
  • SDS Analysis: SDS pseudo-ground truths vary because timestep, noise, and camera pose are random.These three properties are identified as sources of inconsistency in the SDS algorithm.
  • SDS Analysis: With noise fixed, SDS pseudo-ground truths remain inconsistent across camera poses and timesteps, producing feature-averaged results.Figure 9 visualizes rendered views and pseudo-ground truths under these changes.

7.3. Complementary Experiments of ISM

The experiments show that ISM improves distillation efficiency and detail by using deterministic DDIM inversion and interval-based updates, while also converging faster than SDS. They additionally expose a tradeoff between timestep spacing, detail, and visual artifacts.

  • DDIM Inversion: DDIM inversion produces pseudo-GTs more consistent in feature and style across views and timesteps than vanilla noise addition.This consistency persists when δT increases from 20 to 200, although inversion alone does not resolve single-step pseudo-GT quality.
  • Naive Objective: Multi-step denoising replaces SDS’s single-step pseudo-GT estimation, and combining it with DDIM inversion yields the naive distillation objective.The construction follows the same-step-size inversion and denoising process.
  • ISM Objective: Omitting ηt produces the ISM objective, which avoids computing update directions at neighboring timesteps and substantially improves efficiency.ηt contains neighboring interval scores that can leak timestep-dependent residuals; computing the corresponding denoised estimate requires equivalent inversion and denoising steps.
  • Distillation Results: ISM distillation is more efficient than the naive objective and produces 3D objects with enhanced details.The authors attribute the detail improvement to focusing updates solely at timestep t rather than potentially inconsistent directions at other timesteps.
  • Limitations: The analysis leaves the behavior of update directions at neighboring timesteps for future work.ISM focuses on timestep t rather than s, s−δT, and other neighboring steps.
  • Convergence Speed: ISM converges faster than SDS, generating a clear and reasonable structure in 1000 iterations while SDS remains noisy.The comparison fixes the noise and hyperparameters for both methods.

7.4. Zero-shot Avatar Generation

The framework supports zero-shot, pose-specific avatar generation by initializing a 3D representation with SMPL and conditioning ControlNet on DensePose signals.

  • Zero-shot Avatar Generation: SMPL initializes the 3D representation, while a pretrained DensePose predictor supplies segmentation signals to condition ControlNet.The workflow uses an open-source controlnetseg model for the conditional generation stage.

7.5. Details of User Study

The user study compares rendered videos from six text-to-3D methods across 28 prompt sets using 50 internet-collected questionnaires, alongside additional visual results from LucidDreamer.

  • User Study: Participants compared 360° rendered videos from six methods across 28 prompt sets, producing 50 questionnaires summarized in the main paper.The study investigates user preferences among current state-of-the-art text-to-3D methods.
  • More Visual Results: Additional LucidDreamer results are reported as having high visual quality and 3D consistency.These results appear in Fig. 11.
Loading 2311.11284v3…