LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching

1. Introduction

• Text-to-3D uses pretrained text-to-image diffusion models as image priors to train neural 3D models from text, addressing the time, effort, and expertise needed to create high-quality 3D content.The paper notes that 3D content is used across domains including architecture, gaming, virtual and augmented reality, retail, and education.Source: §1 · p. 2
• SDS produces inconsistent, low-quality pseudo-GTs through one-step reconstruction and random diffusion trajectories; applying their averaged updates to one 3D model causes over-smoothing and detail loss.The paper attributes the averaging effect to semantically variant pseudo-GTs and the high reconstruction errors to one-step diffusion-model estimates.Source: §1 · p. 2
• ISM uses DDIM inversion for invertible diffusion trajectories and matches two interval steps, mitigating pseudo-GT inconsistency and avoiding high-error one-step reconstruction.The paper reports that its ISM loss consistently outperforms SDS with realistic, detailed results.Source: §1 · pp. 2–3
• With 3D Gaussian Splatting, LucidDreamer reports state-of-the-art results over Magic3D [21], Fantasia3D [5], and ProlificDreamer [47], without their multi-stage training.The paper says this reduces training cost while maintaining a simple pipeline.Source: §1 · pp. 2–3

Text-to-3D Generation

• DreamField [17] used CLIP-guided NeRF, while DreamFusion [34] introduced SDS, which later methods build on despite over-smoothing [21] [31] [34] [50] and saturation.Other approaches address visual quality or Janus problems, but continue to rely heavily on SDS. LucidDreamer analyzes SDS’s inconsistent pseudo-ground-truths and uses ISM to improve results without added computational burden.Source: §2 · p. 3

Differentiable 3D Representations

• NeRF’s heavy implicit rendering hampers diffusion-resolution output, motivating explicit meshes [40] and 3D Gaussian Splatting [19], which this work adopts.The passage describes Gaussian Splatting as efficient in reconstruction tasks.Source: §2 · p. 3

Diffusion Models

• DDPMs [43] are widely adopted for text-guided 2D generation, modeling transitions by a predefined β_t schedule and predicting the mean and variance given x_t.Source: §2 · p. 3

3.1. Revisiting the SDS

• SDS’s noise and camera-pose randomness produce inconsistent pseudo-ground-truths, while low-quality single-step predictions can be blurry or detail-less, contributing to over-smoothed 3D results.The paper attributes the inconsistencies to the DDPM’s sensitivity to its input and notes that optimizing toward inconsistent pseudo-ground-truths produces feature-averaged outcomes.Source: §3.1 · p. 4

3.2. Interval Score Matching

• ISM uses DDIM inversion for consistent pseudo-ground-truths and interval scores instead of single-step predictions, omitting a bias term to improve efficiency while retaining distillation quality.DDIM inversion makes pseudo-ground-truths more consistent with the rendered view; the omitted bias term comprises neighboring interval scores with opposing scales and depends on a hyperparameter unrelated to the 3D representation.Source: §3.2 · pp. 4–5
• ISM produces high-fidelity results with rich details and fine structure without requiring large conditional guidance scales, and typically converges in fewer iterations despite DDIM inversion costs.The paper says the transition from SDS to ISM adds marginal computational overhead and does not compromise overall efficiency.Source: §3.2 · p. 5

3.3. The Advanced Generation Pipeline

• 3D Gaussian Splatting enables efficient rendering and optimization, supporting high-resolution rendering and larger training batches with limited computational resources; text-to-point models provide coarse initialization that improves convergence.The point-cloud initialization uses human priors and addresses the limitations of initializing with simple geometries for non-axial-symmetric objects.Source: §3.3 · p. 6

4. Experiments

• ISM produces finer-grained detail than SDS [34] in both NeRF [32] and 3D Gaussian Splatting [20], even at normal CFG 7.5.The NeRF comparison follows ProlificDreamer’s hyperparameter design.Source: Figure 5 · p. 7, §4 · p. 7
• Against SoTA baselines [5] [21] [34] [47] reimplemented by Three-studio using Stable Diffusion 2.1 on A100, LucidDreamer achieves high fidelity and geometry consistency with less time and resource consumption.Its crown has more precise geometric structures and realistic colors than baseline results.Source: §4 · p. 7
• Across 28 prompt sets, LucidDreamer achieved the highest average user ranking among six methods, with users consistently favoring its generated 3D models.The study asked users to rank results by fidelity and alignment with the text prompt; the average ranking favored LucidDreamer.Source: Table 1 · p. 7, §4 · p. 7

4.1. Ablation Studies

• The paper demonstrates ISM applications in zero-shot 2D and 3D editing, personalized text-to-3D generation with LoRA, and 3D avatar generation.The figure caption describes the proposed ISM and advanced 3D generation pipeline as performing well across these tasks.Source: Figure 8 · p. 8
• Increasing δT produces more natural colors and simpler structures but sacrifices detail, while lower δT preserves detail at the risk of visual anomalies; increasing δS has marginal visual impact but reduces DDIM inversion costs.The authors propose annealing the interval to build overall structure first and add fine-grained features later; δT can also tune smoothness to user preference.Source: §4.1 · pp. 7–8
• Starting from a prompt-generated point cloud rather than random initialization improves 3D Gaussian geometry, while the Point Generator’s geometry prior reduces the Janus problem.The initialization comparison holds parameters and random seed constant.Source: §4 · p. 7, Figure 7 · p. 7, §4.1 · p. 8

Zero-shot Avatar Generation

• LucidDreamer generates pose-specific avatars using SMPL as a geometry prior and ControlNet conditioned on DensePose [10] for supervision.The framework renders a 3D human mesh into a 2D image, then uses a pretrained DensePose model to produce the body-part segmentation condition; the paper reports a high-fidelity avatar in Fig. 8.Source: §5 · p. 8

Personalized Text-to-3D

• With LoRA [15], LucidDreamer ties subjects or styles to identifier strings for personalized 3D generation, producing humans or objects with fine-grained details.The authors present this combination as having potential for controllable text-to-3D generation.Source: §5 · p. 8

Zero-shot 2D and 3D Editing

• ISM enables zero-shot 2D and 3D editing by directing input-image-conditioned updates toward a target condition, while further customization is left for future work.The paper demonstrates editing in Fig. 8 and notes that further exploration of ISM for these editing tasks is reserved due to space limitations.Source: §5 · p. 8

6. Conclusions

• ISM addresses SDS over-smoothing by targeting inconsistent, low-quality pseudo ground truth with consistent and reliable guidance.The paper identifies pseudo-ground-truth inconsistency and low quality as the root cause of the over-smoothing effect.Source: §6 · p. 8
• LucidDreamer combines ISM with 3D Gaussian Splatting and significantly surpasses current state-of-the-art methodologies in extensive experiments.The framework is presented for text-to-3D generation and editing, zero-shot avatar creation, and personalized text-to-3D conversions.Source: §6 · p. 9
• ISM produces highly detailed results without extra computational costs and is compatible with NeRF, 3D Gaussian Splatting, and 2D editing tasks.The paper describes these applications across 3D generation and editing as evidence of ISM’s versatility.Source: §6 · p. 9

7.1. Implementation details

• LucidDreamer optimizes 3D Gaussian Splatting toward diffusion-model pseudo-GT using its proposed Interval Score Matching objective.The implementation follows most training hyperparameters from the original 3DGS paper.Source: §7.1 · p. 9
• 3DGS Gaussians are densified and pruned every 300 iterations until iteration 3000, while training runs for around 5000 iterations rather than 10,000 in previous works [34].The paper attributes the streamlined training to ISM’s precise gradients and the resulting high coverage speed.Source: §7.1 · p. 9
• LucidDreamer initializes 3DGS from a pretrained Point-E [33] checkpoint and uses camera-dependent prompts for some asymmetrical objects to further reduce Janus problems, following Perp-Neg.Source: §7.1 · p. 9
• In some cases, LucidDreamer uses negative prompts [18] with classifier-free guidance, where gs is the positive-prompt guidance scale.The negative prompts are reported to further improve generation quality.Source: §7.1 · p. 9

7.2. Inconsistency in SDS pseudo-GT

• With noise fixed, SDS pseudo-GTs vary across camera poses and timesteps, yielding feature-averaged results; Figure 9 contrasts random-noise and DDIM-inversion pseudo-GTs.The figure includes views from four camera poses and shows pseudo-GTs at timesteps 500 and 200, including DDIM inversion with step sizes 20 and 200.Source: Figure 9 · p. 9, §7.2 · p. 9
• The identified sources of SDS pseudo-GT inconsistency are randomness in timestep t, noise ϵ, and camera pose c.Source: §7.2 · p. 9

7.3.1 Benefits of DDIM inversion

• DDIM inversion makes pseudo-GTs more like input views and more consistent across views and timesteps, even as δT increases from 20 to 200, but does not fix single-step prediction quality.The comparison replaces vanilla noise addition with DDIM inversion.Source: §7.3.1 · pp. 9–10
• SDS shares SDEdit’s [28] random-noise timestep tradeoff, while ISM’s inversion-then-denoising bridges align with DDIB [44] and frame its objective as entropy-regularized optimal transport.The paper describes ISM as bridging rendered-image and diffusion-latent distributions, then the latent and target distributions.Source: §7.3.1 · p. 10

7.3.2 Discussion of ηt

• ISM produces more detailed results and runs more efficiently than the naive objective by focusing updates on timestep t rather than potentially inconsistent directions at other timesteps.The comparison uses the naive objective in Eq. (19) and ISM with accelerated DDIM inversion.Source: Figure 10 · p. 10, §7.3.2 · p. 10
• Because ηt leaks hyperparameter-dependent neighboring interval scores and requires redundant denoising steps for timesteps outside t, ISM omits it.The authors argue that ISM should distill update directions at t rather than at other timesteps it will traverse; they leave investigation of the leakage to future work.Source: §7.3.2 · pp. 10–11

7.3.3 The convergence speed of ISM v.s. SDS

• ISM generates a clear, reasonable structure in 1,000 iterations, while SDS remains noisy at the same stage, demonstrating faster convergence.Noise and hyperparameters were fixed for the comparison.Source: Figure 12 · p. 11, §7.3.3 · p. 11

7.4. Zero-shot Avatar Generation

• For pose-specific avatar generation, SMPL initializes Gaussian point-cloud positioning, while DensePose produces a body segmentation map conditioned into ControlNet using open-source controlnetseg.The workflow is depicted in Fig. 13.Source: §7.4 · p. 11

7.5. Details of User Study

• Participants compared 360° rendered videos of generated assets from six methods, including LucidDreamer.The study aimed to assess users’ preferences among current state-of-the-art text-to-3D methods.Source: §7.5 · p. 12
• The study presented 28 video sets generated from different prompts and collected 50 questionnaires online.The collected preferences were summarized in the main paper.Source: §7.5 · p. 12

• LucidDreamer’s additional generated 3D assets demonstrate high visual quality and 3D consistency (Fig. 11).Source: §7.6 · p. 12