Source-linked AI summary
3D Gaussian Splatting as Markov Chain Monte Carlo
Shakiba Kheradmand, Daniel Rebain, Gopal Sharma, Weiwei Sun, Jeff Tseng, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, Kwang Moo Yi
TL;DR
Existing 3D Gaussian Splatting methods depend on heuristic Gaussian placement and good initialization, complicating quality and resource control. This paper reformulates training as MCMC implemented with SGLD, replacing those heuristics with probability-preserving state transitions and reporting broader robustness and performance.
Problem
Existing methods rely on carefully engineered cloning, splitting, pruning, opacity resets, and good point-cloud initialization, making Gaussian counts difficult to control and sometimes producing poor renderings.
Method
The method models 3D Gaussians as MCMC samples, implements updates with SGLD noise, replaces heuristic densification and pruning with deterministic state transitions, and adds a regularizer for parsimonious Gaussian use.
Results
The strategy generalizes across various scenes, outperforms original 3D Gaussian Splatting, and beats NeRF backbones on the challenging MipNeRF3360 dataset.
Takeaways & Limitations
The reformulation eliminates point-cloud initialization and heuristic densification, pruning, and reset while improving robustness to initialization and enabling more parsimonious Gaussian use.
Takeaways & Limitations
Classical cloning and centre-corrected cloning can change Gaussian likeness so that P(g_new) ≠ P(g_old), leading to sub-optimal training.
Abstract
from arXiv · showhide
While 3D Gaussian Splatting has recently become popular for neural rendering, current methods rely on carefully engineered cloning and splitting strategies for placing Gaussians, which can lead to poor-quality renderings, and reliance on a good initialization. In this work, we rethink the set of 3D Gaussians as a random sample drawn from an underlying probability distribution describing the physical representation of the scene-in other words, Markov Chain Monte Carlo (MCMC) samples. Under this view, we show that the 3D Gaussian updates can be converted as Stochastic Gradient Langevin Dynamics (SGLD) updates by simply introducing noise. We then rewrite the densification and pruning strategies in 3D Gaussian Splatting as simply a deterministic state transition of MCMC samples, removing these heuristics from the framework. To do so, we revise the 'cloning' of Gaussians into a relocalization scheme that approximately preserves sample probability. To encourage efficient use of Gaussians, we introduce a regularizer that promotes the removal of unused Gaussians. On various standard evaluation scenes, we show that our method provides improved rendering quality, easy control over the number of Gaussians, and robustness to initialization.
1 Introduction
The paper reframes 3D Gaussian Splatting as MCMC sampling to replace heuristic Gaussian placement with principled updates and relocation. This improves rendering quality and initialization robustness while enabling control over Gaussian usage.
- Motivation: Existing 3DGS methods depend on engineered initialization, cloning, splitting, pruning, and opacity-reset heuristics that can produce poor placement and wasted computation.These choices also make Gaussian counts difficult to predict and can require careful hyperparameter tuning.
- MCMC formulation: The method interprets 3D Gaussians as MCMC samples from a distribution favoring collections that faithfully reconstruct the scene.Conventional 3DGS updates become SGLD updates by adding noise to promote exploration.
- Relocation: Densification, pruning, and opacity resetting are reformulated as deterministic state transitions rather than heuristic operations.The approach relocates Gaussians between configurations while treating changes in cardinality as transitions between equivalent sample states.
- Relocation: A revised cloning strategy moves low-opacity Gaussians to live Gaussian locations while approximately preserving the rendered image and sample probability.The whole Gaussian, rather than only its center, is considered when preserving the representation.
- Regularization: An added regularizer encourages unused Gaussians to disappear, promoting parsimonious use of the representation.This targets efficient use of Gaussian memory and computation.
- Results: The method provides higher rendering quality and robustness to initialization, including competitive results with random initialization, across standard evaluation scenes.The authors report high-quality renderings whether Gaussians start randomly or from Structure-from-Motion points.
2 Related Work
Related work spans NeRF-based novel-view synthesis, efficient Gaussian Splatting, and extensions for aliasing, reflections, dynamics, generation, compactness, and few-shot reconstruction. The paper distinguishes its SGLD use from prior NeRF soft mining and positions error-based densification as compatible but separate.
- Novel-view synthesis via Neural Radiance Fields: NeRF represents scenes with neural fields encoding radiance values and renders images through volume rendering along light rays.The related work includes extensions for few-view settings and generalization to new scenes.
- Related MCMC-based work: Prior SGLD work applies the MCMC framework to soft-mine promising samples for faster NeRF training, whereas this paper uses it to rethink 3DGS representation training.The authors emphasize that the application contexts differ despite the shared SGLD paradigm.
- Gaussian Splatting: 3DGS instead stores Gaussians and uses differentiable rasterization, enabling efficient image rendering by processing the few Gaussians associated with each pixel.The passage reports 1080p rendering at 130 frames per second on modern GPUs.
- Gaussian Splatting: Subsequent Gaussian Splatting work addresses aliasing, reflections, dynamic scenes, content generation, avatars, few-shot reconstruction, and compact representations.Compact-representation methods prune or cluster Gaussians for mobile rendering.
- Concurrent work: Concurrent work modifies densification to address cloning and high-error locations, but its error-based densification is presented as orthogonal and incorporable into this method.The paper states that the cloning issue is only partially addressed by that approach.
3 Method
The method reframes 3D Gaussian Splatting as MCMC sampling, replacing heuristic optimization and cardinality changes with SGLD updates and probability-preserving relocation transitions.
- 3.1 Brief review of 3D Gaussian Splatting: 3D Gaussian Splatting represents scenes as rasterized 3D Gaussians trained with an L1/SSIM rendering loss.Gaussians are ordered by camera distance and composited through α-blending.
- 3.2 3D Gaussian Splatting as Markov Chain Monte Carlo (MCMC): The method defines a distribution G ∝ exp(−Ltotal) that assigns high probability to Gaussian collections faithfully reconstructing training images.This sampling view permits discrete parameter-space changes while retaining a probabilistic formulation.
- 3.2 3D Gaussian Splatting as Markov Chain Monte Carlo (MCMC): SGLD extends the conventional Gaussian update with a noise term, adding exploration to the loss-gradient update.The standard update corresponds to SGLD with no noise when the loss is the negative log likelihood.
- 3.3 Updating with Stochastic Gradient Langevin Dynamics: The added noise perturbs Gaussian locations anisotropically according to covariance and opacity, while reducing noise for opaque Gaussians.The sigmoid transition uses k=100 and t=0.005, centered near the default pruning threshold.
- 3.4 Heuristics as state transitions via relocation: Densification, pruning, splitting, cloning, and adding are reformulated as deterministic MCMC state transitions between Gaussian configurations.These operations relocate samples rather than relying on the original heuristic framework.
- 3.4 Heuristics as state transitions via relocation: Existing cloning strategies broaden the selected Gaussian’s extent, causing P(gnew) ≠ P(gold) and leading to sub-optimal training.The proposed relocation instead moves low-opacity Gaussians to live Gaussian locations while approximately preserving the rendering distribution.
4 Experiments
Across synthetic and real scenes, the method delivers higher-quality rendering than 3DGS, remains robust to initialization, and performs well under constrained Gaussian budgets. Ablations show that location noise, regularization, and noise scheduling materially affect reconstruction quality, while added sampling overhead is small.
- Same number of Gaussians: The method outperforms 3DGS with the same Gaussian count and shows only minor performance variation between random and SfM initialization.Experiments use multiple synthetic and real datasets, with qualitative comparisons in Figure 2.
- Limited budget: With a limited Gaussian budget, the performance gap between the method and 3DGS increases.The comparison averages results across datasets except NeRF Synthetic.
- Initialization sensitivity: The method is robust to initialization, whereas 3DGS requires careful initialization and differs significantly across strategies.The initialization ablation compares random and SfM-based setups.
- Ablations: Noise is critical for exploration, while the opacity and scale regularizers prevent stray Gaussians; these regularizers are harmful when added to classical 3DGS.Without noise, Gaussians cannot explore the full scene extent.
- Ablations: Adding noise to scale, rotation, and opacity slightly worsens performance, suggesting that exploration is most needed for Gaussian locations.The additional noise uses exponentially decayed Gaussian perturbations.
- Ablations: Exponential noise scheduling achieves 24.21 PSNR, compared with 17.64 for linear scheduling and 22.46 for the scheduler from.The exponential schedule is the one used by the method.
- Runtime: At 1M Gaussians, one optimization iteration takes 80 milliseconds for the method and 76 milliseconds for 3DGS.The added sampling and noise computation is reported as not substantial.
5 Conclusion
The paper reformulates 3DGS training as MCMC implemented through SGLD, replacing heuristic initialization and Gaussian population management. Across varied scenes, it reports improved performance over original 3DGS and a result surpassing NeRF backbones on challenging MipNeRF 360 scenes.
- 5 Conclusion: The method reformulates 3DGS training as MCMC and implements it through Stochastic Gradient Langevin Dynamics.The reformulation targets the training process rather than changing the final 3D Gaussian Splat representation.
- 5 Conclusion: The reformulation eliminates the need for point-cloud initialization and avoids heuristic densification, pruning, and opacity resetting.These changes are presented as consequences of the MCMC formulation.
A Derivation for the cloning strategy
The cloning strategy seeks to preserve rasterization while relocating a Gaussian into multiple copies. The derivation uses slice-based constraints to obtain an analytically simple covariance update, while noting that direct error minimization is intractable.
- Cloning objective: The strategy minimizes the difference between a Gaussian's rasterization before and after cloning into multiple copies.The cloned Gaussians retain the original center-related parameters and are adjusted to preserve rendering impact.
- Resulting update: The resulting update equation agrees with a prior method when N=2.This establishes a special-case correspondence for two cloned Gaussians.
- Cloning objective: Directly minimizing rasterization mean squared error produces a complex equation without a simple analytical form.This motivates the alternative slice-based derivation.
- Slice-based derivation: Random 1D slices through the shared Gaussian center impose equal pre- and post-cloning rasterization integrals.The approach is inspired by sliced Wasserstein methods and applies the constraint to arbitrary slices.
- Slice-based derivation: The slice-based equation has a simple analytical solution for the cloning update.The resulting expression is then used to derive how the Gaussian scales are modified.
- Resulting update: With equal Gaussian counts, the proposed method outperforms 3DGS across the reported evaluation results.The comparison uses the same number of Gaussians for both methods.
B Detailed results
The reported standard deviations are computed from averages across all scenes and three independent runs for each dataset and seed.
- Evaluation protocol: Standard deviations are computed from averages across all scenes and three different runs for each dataset and seed.The reported numbers are listed for all scenes in Table 5.
C Limitations and future work
The method improves initialization robustness and rendering quality but retains 3DGS's modelling-capacity limitations, including aliasing and reflections.
- Limitations: The method remains subject to the same modelling-capacity limitations as 3DGS.The authors identify aliasing and reflection modelling as examples of affected capabilities.
- Future work: Compatibility with future Gaussian Splatting advances is presented as a potential way to address these limitations.The method is described as a training framework intended to enhance other Gaussian Splatting methods.
D Broader impact
The work focuses on 3D reconstruction but may affect downstream applications by reducing reliance on Gaussian Splatting initialization heuristics.
- Downstream applications: Reduced initialization reliance may benefit downstream 3D content generation and controllable human modelling.The authors identify potential effects on generative methods and human avatars.
- Risks: The authors caution that these downstream applications may be misused and urge users to consider the implications of applying the method.The concern is framed as applying broadly to the technology's potential uses.
E Dataset licenses
The paper lists the datasets used and records their stated licensing information or the absence of license terms.
- The evaluation datasets listed are NeRF Synthetic, Mip-NeRF 360, OMMO, Deep Blending, and Tank & Temples.
- NeRF Synthetic is made available under a Creative Commons Attribution 3.0 License.
- Mip-NeRF 360, OMMO, and Deep Blending have no license terms provided in the listed sources.
- Tank & Temples is described as made available under Creative Commons Attribution-, with the license description continuing beyond the passage.