Source-linked AI summary
Taming 3DGS: High-Quality Radiance Fields with Limited Resources
Saswat Subhajyoti Mallick, Rahul Goel, Bernhard Kerbl, Francisco Vicente Carrasco, Markus Steinberger, Fernando De La Torre
TL;DR
3DGS uses redundant, unbounded Gaussian sets that increase memory, training, and rendering costs, limiting constrained-device use. This paper introduces exact-budget, score-guided densification and faster training alternatives, achieving improved resource efficiency and new opportunities for constrained environments.
Problem
Redundant, unbounded Gaussian distributions increase 3DGS memory, training, and rendering costs, limiting use on constrained devices and fixed-size downstream tasks.
Method
The method combines exact budget-constrained optimization with score-guided densification and alternative training computations, including per-splat backpropagation and optimized differentiable loss kernels.
Results
4–5× average speedup reduces training to a few minutes on consumer-grade hardware, while the approach provides improved resource efficiency for high-quality radiance fields.
Takeaways & Limitations
Exact primitive budgeting and flexible sample steering create opportunities for novel-view synthesis on hardware-constrained and edge devices, including latency-constrained streaming.
Takeaways & Limitations
Optimal quality still requires a substantial sample count and meandering exploration as Gaussians move across the scene.
Abstract
from arXiv · showhide
3D Gaussian Splatting (3DGS) has transformed novel-view synthesis with its fast, interpretable, and high-fidelity rendering. However, its resource requirements limit its usability. Especially on constrained devices, training performance degrades quickly and often cannot complete due to excessive memory consumption of the model. The method converges with an indefinite number of Gaussians -- many of them redundant -- making rendering unnecessarily slow and preventing its usage in downstream tasks that expect fixed-size inputs. To address these issues, we tackle the challenges of training and rendering 3DGS models on a budget. We use a guided, purely constructive densification process that steers densification toward Gaussians that raise the reconstruction quality. Model size continuously increases in a controlled manner towards an exact budget, using score-based densification of Gaussians with training-time priors that measure their contribution. We further address training speed obstacles: following a careful analysis of 3DGS' original pipeline, we derive faster, numerically equivalent solutions for gradient computation and attribute updates, including an alternative parallelization for efficient backpropagation. We also propose quality-preserving approximations where suitable to reduce training time even further. Taken together, these enhancements yield a robust, scalable solution with reduced training times, lower compute and memory requirements, and high quality. Our evaluation shows that in a budgeted setting, we obtain competitive quality metrics with 3DGS while achieving a 4--5x reduction in both model size and training time. With more generous budgets, our measured quality surpasses theirs. These advances open the door for novel-view synthesis in constrained environments, e.g., mobile devices.
1 Introduction
3DGS offers high-quality novel-view synthesis but can require unpredictable, redundant Gaussian distributions that strain constrained devices and fixed-size downstream applications. The paper introduces budget-controlled densification and faster training operations to reduce resources while preserving or improving quality.
- 3DGS can produce redundant Gaussians, leaving some regions under-reconstructed or blurry.
- Several million Gaussians and over one gigabyte of storage can complicate real-time rendering on low-end devices.
- Gaussian counts can vary by one order of magnitude across scenes, hindering downstream applications requiring fixed-size inputs.
- The method uses purely constructive, score-based densification to meet an exact user-defined Gaussian budget without substantial pruning or culling.
- Alternative training operations, including revised backpropagation parallelization and equivalent or approximate substitutes, reduce training time by 4–5× on average.
2 Related work
Prior work spans neural, image-, mesh-, and point-based approaches to novel-view synthesis, alongside methods compressing or pruning 3DGS models. These approaches reduce storage or improve rendering, but existing compression methods provide limited control over primitive counts and are compatible with the paper’s budgeted models.
- Novel-view synthesis methods range from image- and mesh-based representations to purely neural representations.
- NeRF-based techniques can reconstruct unbounded scenes at interactive frame rates, but training requires significant time and compute.
- 3DGS compression methods reduce storage by clustering, quantizing, encoding, pruning, or culling Gaussian parameters and primitives.
- Existing compression methods reduce storage effectively but provide little process control, while primitive decimation usually achieves only modest reductions of approximately 2×.
- Point-cloud downsampling motivates the paper’s focus on optimizing quality at low primitive counts and includes heuristic and task-specific sampling methods.
3 Method
The method makes 3DGS growth predictable and quality-directed, then uses interpretable saliency scores and less frequent densification to select useful primitives. It also analyzes the training pipeline to guide efficient updates and backpropagation.
- Predictable Model Growth: The approach replaces continuously thresholded densification with a pre-determined schedule initialized from SfM point clouds.The schedule targets a user-defined final Gaussian count while training a 3DGS model from calibrated multi-view images.
- Predictable Model Growth: A parabolic growth schedule reaches the user-defined budget, while correction for pruning keeps the final primitive count aligned with the target.The schedule uses N densification steps, final count B, and SfM initialization count S.
- Steerable Densification with Sampling: Score-based sampling combines per-Gaussian attributes with per-pixel metrics, including loss, to prioritize densification candidates.The scoring function can incorporate projected metrics and Gaussian attributes, with coefficients tunable for specific use cases.
- Steerable Densification with Sampling: Densification is performed every 500 iterations, one-fifth as frequently as in 3DGS, allowing misplaced Gaussians time to be removed through opacity reduction.The lower frequency is motivated by the interaction between loss, sample placement, and optimization.
- Steerable Densification with Sampling: Per-view saliency combines masked-region selection, L1 loss, and a Laplacian response, then aggregates scores across views for each Gaussian.The region-of-interest mask enables prioritization of selected image regions.
- Steerable Densification with Sampling: The densification procedure scores candidates using gradient, coverage, pixel-distance, opacity, scale, and blending information before cloning or splitting selected Gaussians.It uses thresholds to choose splitting for larger projected Gaussians and cloning for smaller ones.
4 3DGS Runtime Analysis and Optimization
The runtime analysis identifies backpropagation and growing optimizer updates as major 3DGS costs, then replaces them with targeted parallelization and computational optimizations. These changes substantially reduce training time while preserving equivalence to 3DGS in most cases.
- Runtime bottlenecks: Backpropagation dominates 3DGS training time, followed closely by ADAM updates as the Gaussian count increases.The benchmark breaks down high-level iteration steps across multiple scenes and training stages.
- Backpropagation: Per-pixel backward computation is replaced by per-splat parallelization, allowing threads to maintain per-splat state while exchanging smaller per-pixel states.The method reduces contention from globally accumulating many per-pixel, per-splat gradient values.
- Backpropagation: Occlusion-aware skipping and tighter culling reduce redundant rasterization work in both backward processing and the forward pass.The backward pass tracks the last contributor across each tile to skip splat–tile pairings that become redundant after saturation.
- Attribute updates: 48 of 59 optimized per-Gaussian attributes are spherical-harmonic coefficients, so higher bands receive batched ADAM updates once every 16 iterations.The 0th SH band remains separate from higher bands, which are updated through a batched schedule.
- Loss computation: Separable 11×11 Gaussian convolution and a fused SSIM kernel accelerate differentiable loss computation, especially when few Gaussians are optimized relative to image resolution.The convolution is implemented as two consecutive 1D convolutions using optimized CUDA kernels.
- Results: The optimized measures are drop-in replacements for original 3DGS, with equivalent results except for the modified SH update schedule, and outperform gsplat 1.0 by 1.5×–2×.The reported comparison concerns training-time reductions from the combined optimizations.
5 Evaluation and Discussion
The evaluation measures quality and resource efficiency across two budgeted scenarios, supported by quantitative comparisons, qualitative results, and ablations. The method achieves competitive or superior quality while controlling model size, training speed, and memory use.
- Datasets and Metrics: The evaluation covers Tanks&Temples, Deep Blending, and MipNeRF360 using PSNR, SSIM, LPIPS, training time, and Gaussian counts.Resource measurements include both final and peak numbers of Gaussians.
- Results: In the first budgeted scenario, Ours achieves outstanding reduction among splatting-based methods, with Mini-Splatting slightly outperforming it in one reported aspect.Budgets are selected per scene using spatial extent and SfM point counts.
- Results: At exactly the original 3DGS model size, Ours surpasses 3DGS and MipNeRF360 in quality, second only to the slower Zip-NeRF.This scenario demonstrates exact matching of the original 3DGS model size.
- Ablations: Removing image loss or score-based sampling significantly harms quality, even when all configurations use the same number of Gaussians.The ablation isolates densification choices under the first budgeted scenario.
- Ablations: Reverting to the original SH update frequency causes up to a 50% fitting-speed drop, while replacing the per-splat backward pass incurs an even higher performance cost.The SH change may yield minuscule quality improvements, whereas the original backward pass is less efficient.
- Ablations: Increasing the available budget consistently improves image quality, while the first budgeted scenario trains with consistently less than 10 GB VRAM.The reported memory level is described as suitable for a mid-range NVIDIA RTX 3080.
6 Conclusion
The conclusion presents an efficient splatting-based optimization technique that controls 3DGS resources through exact budgeting and flexible sample steering. It identifies constrained and latency-sensitive environments as application opportunities while noting that optimal quality still requires substantial sample counts and exploration.
- Conclusion: The approach provides exact primitive budgeting, flexible sample steering, and improved resource efficiency for high-quality radiance fields.The conclusion frames these properties as controlling the unpredictable behavior of 3DGS.
- Conclusion: The method creates opportunities for novel-view synthesis on hardware-constrained and edge devices.The stated scope includes environments with limited hardware resources.
- Conclusion: Latency-constrained streaming could steer interactive reconstructions toward salient regions such as faces.This is presented as a potential application rather than an evaluated deployment.
- Conclusion: Achieving optimal quality still requires substantial sample counts and meandering exploration as Gaussians move across the scene.The paper identifies efficient search paths, occupancy predictions, and blind-spot resolution as future-work directions.