Source-linked AI summary
GaussianDiffusion: 3D Gaussian Splatting for Denoising Diffusion Probabilistic Models with Structured Noise
Xinhai Li, Huaibin Wang, Kuo-Kun Tseng
TL;DR
Text-to-3D generation faces multi-view inconsistency, slow rendering, and artifacts in vanilla Gaussian models. GaussianDiffusion combines Gaussian Splatting with structured noise and variational splatting; the authors report faster rendering and improved realism over prior methods.
Problem
Text-to-3D generation from 2D diffusion models faces multi-view geometric inconsistency, slow NeRF rendering, and artifacts in vanilla 3D Gaussian models.
Source: §1 · p. 2Method
GaussianDiffusion combines Gaussian Splatting with score-function and Langevin diffusion, viewpoint-structured noise, and variational perturbations of Gaussian parameters.
Source: §1 · p. 2, §3.3 · p. 6Results
The authors report faster rendering and improved realism over prior methods, while structured noise addresses multi-view inconsistency and variational splatting mitigates artifacts.
Source: §5 · p. 9Takeaways & Limitations
The authors present GaussianDiffusion as a text-to-3D framework that improves appearance while addressing multi-view inconsistency and rendering speed.
Source: §5 · p. 9The SJC formulation assumes the asset density is proportional to expected multiview-rendering densities over camera poses.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
Text-to-3D, known for its efficient generation methods and expansive creative potential, has garnered significant attention in the AIGC domain. However, the pixel-wise rendering of NeRF and its ray marching light sampling constrain the rendering speed, impacting its utility in downstream industrial applications. Gaussian Splatting has recently shown a trend of replacing the traditional pointwise sampling technique commonly used in NeRF-based methodologies, and it is changing various aspects of 3D reconstruction. This paper introduces a novel text to 3D content generation framework, Gaussian Diffusion, based on Gaussian Splatting and produces more realistic renderings. The challenge of achieving multi-view consistency in 3D generation significantly impedes modeling complexity and accuracy. Taking inspiration from SJC, we explore employing multi-view noise distributions to perturb images generated by 3D Gaussian Splatting, aiming to rectify inconsistencies in multi-view geometry. We ingeniously devise an efficient method to generate noise that produces Gaussian noise from diverse viewpoints, all originating from a shared noise source. Furthermore, vanilla 3D Gaussian-based generation tends to trap models in local minima, causing artifacts like floaters, burrs, or proliferative elements. To mitigate these issues, we propose the variational Gaussian Splatting technique to enhance the quality and stability of 3D appearance. To our knowledge, our approach represents the first comprehensive utilization of Gaussian Diffusion across the entire spectrum of 3D content generation processes.
1. Introduction
GaussianDiffusion combines Gaussian Splatting with diffusion-based text-to-3D generation to target multi-view inconsistency, slow rendering, and artifacts. The authors report faster rendering and the most realistic appearance currently achievable in text-to-3D tasks.
1. Introduction
- Automating 3D asset creation requires high-quality geometry consistent across views, alongside rich materials and textures.Source: §1 · p. 2
- Text-prompted 3D generation faces multi-view geometric inconsistency, slow NeRF rendering, and artifacts from vanilla 3D Gaussian models.Source: §1 · p. 2
- GaussianDiffusion combines a Gaussian Splatting renderer with score-function and Langevin diffusion, and claims the most realistic appearance currently achievable in text-to-3D tasks.Source: §1 · p. 2
- Structured noise from multiple viewpoints targets multi-view geometric inconsistency, including the problem of multifaceted structures.Source: §1 · p. 2
- Variational Gaussian Splatting is proposed to mitigate local-minimum artifacts such as floaters, burrs, and proliferative elements.Source: §1 · p. 2
2. Related Works
Related work reviews diffusion models and Gaussian Splatting as foundations for 3D generation, alongside image-to-3D and text-to-3D approaches. It contrasts prior methods with GaussianDiffusion's structured-noise approach to the multiface problem.
2.1. Diffusion models
- 2D diffusion models' stability, versatility, and scalability underpin distillation-based 3D generation, which uses them as teacher models.Source: §2.1 · p. 2
2.2. 3D Gaussian Splatting
- 3D Gaussian Splatting represents scenes with projected and α-composited spheres; the structured-noise figure links pose- and pixel-related noise to a shared source.Source: §2.2 · pp. 2–3, Figure 3 · p. 3
2.3. Image-to-3D generation
- Single-image-to-3D draws on image data and mature diffusion models, but limits computers' generation and imagination; text prompts better fit human-computer interaction.Source: §2.3 · p. 3
2.4. Text-to-3D generation
- DreamFusion uses SDS, SJC adds secondary noise, and 3DFuse uses staged depth and ControlNet guidance; TANGO needs a 3D input, while GaussianDiffusion uses score-function Langevin sampling.Source: §2.4 · p. 3
2.5. Multiface Problem
- Unlike prior viewpoint-feature methods that fine-tune on Objaverse, structured noise avoids fine-tuning and combines random Gaussian-sphere noise with the target scene before denoising.Source: §2.5 · p. 3
3. Method
GaussianDiffusion builds text-to-3D generation around 3D Gaussian parameters and viewpoint-conditioned noise, then learns distributions over those parameters rather than only fixed values.
3.1. 3D Gaussian Splatting for SJC
- SJC models a 3D asset through expected densities of its multiview renderings, motivating viewpoint-conditioned noise projected from a shared source.The paper identifies constructing this noise and using it to perturb rendered views as the central challenge.Source: §3.1 · p. 4
3.2. 3D Noise Generation and Projection
- For positive semi-definite Gaussian covariances, scale s and rotation R define the 3D covariance, which is projected using viewpoint W and Jacobian J.The projected ellipsoid is represented as a bivariate Gaussian on the image plane.Source: §3.2 · p. 4
- Semantic Code Sampling narrows the scene to one semantic identity; Point-E supplies a sparse cloud, while ControlNet, LoRA, Gaussian Splatting, and SDS guide generation.The cloud is pose-projected into a depth map for ControlNet, and serves as Gaussian Splatting’s initial input.Source: Figure 5 · p. 5
- With fixed camera and noise positions, α-weighted color blending yields normally distributed pixel color C, which is standardized to N(0, I) for diffusion noise.The resulting noise is associated with viewpoint π and is added to a novel Gaussian Splatting view for the diffusion process.Source: §3.2 · p. 5
3.3. Variational Gaussian Spaltting
- Variational Gaussian Splatting learns a distribution over parameters by perturbing position z and scale s, with perturbation variance synchronized to diffusion noise.Higher diffusion noise calls for more perturbation and a blurrier model; lower noise calls for less perturbation as the model moves from coarse to fine.Source: §3.3 · p. 6
- Perturbing θ expands the represented scene space to help escape local minima and avoid artifacts, while gradients remain equivalent to those for θ.The method propagates gradients to the distribution mean and retains the diffusion model’s noise level in the variance.Source: Figure 4 · p. 4, §3.3 · pp. 6–7
4. Experiment
The experiments report faster convergence and stronger geometric consistency for GaussianDiffusion, alongside qualitative gains and ablations that expose weaknesses when either proposed technique is removed.
4.1. Overall Pipeline
- The pipeline uses Semantic Code Sampling, LoRA adaptation, Point-E initialization, and depth conditioning, with 2,000 overall training steps and 4,096 initial cloud points.The experiments use 512-resolution rendering and randomly sampled camera poses.Source: §4.1 · p. 7
4.2. Comparision of Convergence Speed
- The method reaches a satisfactory state around 2,000 iterations, compared with about 10,000 for 3DFuse, while maintaining geometric consistency and avoiding artifacts.It also enhances rendering speed and integrates with traditional rendering pipelines.Source: §4.2 · pp. 7–8
4.3. Structured Noise Generation Details
- Structured noise combines viewpoint-based Gaussian noise with SJC noise, gradually reducing its proportion from 0.3 to 0.05 over 2,000 iterations.The construction is intended to preserve standard-normal noise per pixel and numerical stability.Source: §4.3 · p. 8
4.4. Variational Gaussian Splatting
- Variational Gaussian Splatting perturbs Gaussian positions and scales with noise, allowing the model to learn their distribution rather than fixed values.The reported perturbation uses noise variance matched to the frozen diffusion model and γ = 0.15.Source: §4.4 · p. 8
4.5. Quantitative Comparison
- GaussianDiffusion significantly outperforms SJC, 3DFuse, and DreamGaussian on mean variance across 50 generated 3D scenes.The evaluation uses COLMAP-based mean variance; the supplied text gives no numeric scores.Source: §4.5 · p. 8, Table 1 · p. 8
4.6. Qualitative Comparision
- Comparisons after fine-tuning report realistic, visually appealing results with multiview geometric consistency, while SJC and 3DFuse retain appearance or consistency issues and DreamGaussian has limited later optimization.The comparisons use 10,000 iterations for SJC and 3DFuse, 600 for DreamGaussian, and 2,000 for GaussianDiffusion.Source: §4.6 · p. 8
4.7. Ablation Study
- Removing structured noise produces multiview consistency problems and multifaceted structures; replacing variational splatting with vanilla splatting yields floaters, burrs, or proliferative elements.The quantitative variance change after removing structured noise is described as a sharp decline; the change after removing variational splatting is described as slight.Source: §4.7 · pp. 8–9, Figure 8 · p. 8
5. Discussion and Conclusion
GaussianDiffusion combines Gaussian Splatting with structured noise and variational Gaussian Splatting for text-to-3D generation. The paper reports faster rendering and more realistic appearance, while noting residual blurriness and haze.
5. Discussion and Conclusion
- The Gaussian Splatting framework is reported to accelerate rendering and achieve realistic appearance in text-to-3D tasks.The paper describes these results as improved realism compared with prior methods.Source: §5 · p. 9
- Structured noise from multiple viewpoints addresses the challenge of multiview geometric inconsistency.Source: §5 · p. 9
- Variational Gaussian Splatting mitigates floaters, burrs, and proliferative elements, but introduces some blurriness and haze.Source: §5 · p. 9