Source-linked AI summary
PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting
Alex Hanson, Allen Tu, Vasu Singla, Mayuka Jayawardhana, Matthias Zwicker, Tom Goldstein
TL;DR
3D-GS scenes can contain millions of Gaussians, creating storage and memory demands, while heuristic pruning can be less effective at high compression. PUP uses a spatial sensitivity score and multiple prune-refine rounds on pretrained models; after 90% pruning, it reports a 3.56× average rendering-speed increase and better image quality metrics than existing techniques.
Problem
3D-GS scenes can contain millions of Gaussians, creating high storage and memory requirements that limit their viability on resource-limited devices.
Source: §1 · p. 2Method
PUP scores Gaussian sensitivity to training-view reconstruction error using a Fisher approximation, then prunes and fine-tunes pretrained models over multiple rounds.
Source: §1 · p. 2, §4 · p. 3, §4.4 · p. 4Results
Takeaways & Limitations
PUP is a post-hoc pruning technique applicable to pretrained 3D-GS models without changing their training pipeline.
Source: §1 · p. 2The approach assumes the scene is converged because the Fisher approximation requires a small L1 residual to be accurate.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Recent advances in novel view synthesis have enabled real-time rendering speeds with high reconstruction accuracy. 3D Gaussian Splatting (3D-GS), a foundational point-based parametric 3D scene representation, models scenes as large sets of 3D Gaussians. However, complex scenes can consist of millions of Gaussians, resulting in high storage and memory requirements that limit the viability of 3D-GS on devices with limited resources. Current techniques for compressing these pretrained models by pruning Gaussians rely on combining heuristics to determine which Gaussians to remove. At high compression ratios, these pruned scenes suffer from heavy degradation of visual fidelity and loss of foreground details. In this paper, we propose a principled sensitivity pruning score that preserves visual fidelity and foreground details at significantly higher compression ratios than existing approaches. It is computed as a second-order approximation of the reconstruction error on the training views with respect to the spatial parameters of each Gaussian. Additionally, we propose a multi-round prune-refine pipeline that can be applied to any pretrained 3D-GS model without changing its training pipeline. After pruning 90% of Gaussians, a substantially higher percentage than previous methods, our PUP 3D-GS pipeline increases average rendering speed by 3.56$\times$ while retaining more salient foreground information and achieving higher image quality metrics than existing techniques on scenes from the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets.
PUP 3D-GS (Ours)
Figure 1 shows PUP pruning the Deep Blending playroom scene while preserving fine detail and accelerating rendering. It also compares PUP with LightGaussian.
PUP 3D-GS (Ours)
- 4.15× speed-up: PUP reduces the playroom scene from 2.65M to 0.265M Gaussians while raising rendering speed from 76.65 FPS to 318.06 FPS.The figure reports this result for the Deep Blending playroom scene.Source: Figure 1
- LightGaussian runs at 261.27 FPS and loses substantially more fine detail than PUP in the playroom comparison.The comparison is at 0.265M Gaussians for each method.Source: Figure 1
1. Introduction
3D-GS offers real-time rendering with image quality comparable to NeRF, but its many Gaussians create storage and memory demands. PUP introduces a sensitivity-based, multi-round post-hoc pruning pipeline for pretrained models.
1. Introduction
- 3D-GS offers real-time rendering and comparable image quality to NeRF, but scenes with millions of Gaussians impose high storage and memory requirements.NeRF inference can take several seconds per image.Source: §1 · pp. 1–2
- PUP scores Gaussian sensitivity to reconstruction error using a computationally feasible Hessian-derived estimate, then prunes and fine-tunes the remaining Gaussians.The approach removes 80% of base-scene Gaussians while preserving image quality metrics.Source: §1 · p. 2
- PUP’s multi-round pipeline removes 90% of Gaussians and surpasses LightGaussian in rendering speed and image quality; it can be applied to pretrained models without changing their training pipeline.The paper also reports higher-fidelity reconstructions from multi-round prune-refine than an equivalent single round.Source: §1 · p. 2
2. Related work
Related work covers NeRF and 3D-GS for novel-view synthesis, uncertainty estimation, and approaches to reducing Gaussian-splat storage. PUP differs in its direct use of Fisher information for spatial pruning and its post-hoc focus.
2. Related work
- NeRF-based approaches offer visual fidelity but slow training and inference, while 3D-GS provides faster inference with comparable visual fidelity.The cited comparison places 3D-GS against state-of-the-art NeRF-based approaches.Source: §2 · p. 2
2.1. Uncertainty Estimation
- Prior NeRF uncertainty methods model transient changes, occlusion, or sparse views using ensembles or variational inference, while BayesRays uses Fisher information with a hypothetical perturbation field.PUP instead computes Fisher information directly from gradients for 3D Gaussian splats.Source: §2.1 · p. 2
2.2. Pruning Gaussian Splat Models
- FisherRF estimates uncertainty from color parameters using a diagonal Fisher approximation for active-view selection, whereas PUP uses spatial mean and scale parameters for pruning.Other compression methods use quantization, neighboring-Gaussian relationships, or pruning heuristics; some modify training or do not reduce primitive count.Source: §2.1 · p. 2, §2.2 · p. 2
3. Background: 3D Gaussian Splatting
3D-GS represents a scene with independently parameterized Gaussians optimized against posed training images. Rendering projects and blends the Gaussians, while training minimizes a loss combining L1 residual and SSIM terms.
3. Background: 3D Gaussian Splatting
- Each Gaussian has position, scale, rotation, base color, spherical harmonics, and opacity parameters; training images come with camera poses.A sparse Structure from Motion point cloud initializes the Gaussians, and densification adds Gaussians where small-scale geometry is insufficiently reconstructed.Source: §3 · p. 3
- A Gaussian’s spatial shape is represented by its mean position and a covariance matrix derived from its rotation and scaling matrices.The scaling and rotation parameters are converted into those matrices during view synthesis.Source: §3 · p. 3
- For each camera pose, a differentiable rasterizer projects visible Gaussians, and pixel colors blend their view-dependent colors and projected opacity contributions.The blend accounts for the ordered Gaussians overlapping each pixel.Source: §3 · p. 3
- Training optimizes a loss that combines L1 residual error with an SSIM loss between the rendered and ground-truth images.The loss is optimized by stochastic gradient descent.Source: §3 · p. 3
4. Method
PUP 3D-GS treats uncertainty as how far Gaussian parameters can change without affecting training-view reconstruction loss, and uses a tractable Hessian approximation to score sensitivity. It computes the score over image patches and applies it in repeated prune-refine rounds.
4. Method
- Uncertainty measures how much a Gaussian’s parameters can be perturbed without changing reconstruction loss over the input views, reflecting ambiguity in underconstrained scene reconstruction.Limited camera observations can leave multiple Gaussian locations and scales consistent with an input view.Source: §4 · p. 3
4.1. Fisher Information Matrix
- The method differentiates training-view L2 reconstruction error twice, then uses the converged-model residual condition to reduce the Hessian to a gradient-product Fisher approximation.Because the approximation uses gradients of reconstructed images, it depends on input poses rather than input images.Source: §4.1 · p. 3
4.2. Sensitivity Pruning Score
- Each Gaussian’s block-diagonal Hessian yields an independent sensitivity score, computed as a log determinant and restricted in the final score to spatial mean and scale.The log determinant represents the relative impact of the Gaussian’s parameters on reconstruction error.Source: §4.2 · p. 4
4.3. Patch-wise Uncertainty
- Image-patch Fisher approximations are summed across views to estimate the scene-level Hessian; experiments use 4 × 4 patches because patch scores correlate with pixel-wise scores.The patches are computed by rendering at lower resolution and evaluating the Fisher approximation on the resulting pixels.Source: §4.3 · p. 4
4.4. Multi-Round Pipeline
- Repeated prune-refine can recover small reconstruction residuals and empirically outperform an equivalent single round of pruning and fine-tuning.Each round prunes and fine-tunes without further Gaussian densification; Figure 3 marks 80% then 50% pruning as the 90% compression setting.Source: §4.4 · p. 4, Figure 3 · p. 5
5. Experiments
PUP 3D-GS is evaluated on scenes from three datasets using matched pretrained models and a prune-refine pipeline. At 90% pruning, it compares favorably with LightGaussian across nearly every reported metric, and quantization further reduces model size.
5.1. Datasets
- Evaluation uses all nine Mip-NeRF 360 scenes, two Tanks & Temples scenes, and two Deep Blending scenes, with dataset-provided COLMAP camera poses.Source: §5.1 · p. 5
5.2. Implementation Details
- The method is implemented in the original 3D-GS codebase, adapts LightGaussian’s pruning/refining framework, computes pruning scores across training views in seconds, and measures FPS on an Nvidia RTXA4000 GPU.Source: §5.2 · p. 5
5.3.1 Analysis of PUP 3D-GS Pipeline
- After pruning 80% of Gaussians, fine-tuning recovers image-quality metrics with a slight speed decrease; a second prune-refine round removes 50% of those remaining and recovers most quality degradation while more than tripling rendering speed over 3D-GS.Table 1 reports mean PSNR, SSIM, LPIPS, FPS, and point-cloud size on Mip-NeRF 360, with operations applied cumulatively by row.Source: §5.3.1 · p. 5, Table 1 · p. 5
5.3.2 Comparison with LightGaussian
- Figure 4 compares two-round prune-refine results with LightGaussian on bonsai and room from Mip-NeRF 360 and train from Tanks & Temples.Source: Figure 4 · p. 6
- At 90% pruning with identical final sizes, PUP 3D-GS increases 3D-GS rendering speed by 3.56× on average and outperforms LightGaussian across nearly every metric, except Tanks & Temples PSNR by 0.36 points.Both methods prune 80% of Gaussians, then 50% of the remainder; Figure 3 reports consistent wins across metrics and pruning-percentage permutations.Source: Table 2 · p. 6, §5.3.2 · p. 6
5.3.3 Vectree Quantization
- After 90% pruning, Vectree-quantized PUP 3D-GS outperforms LightGaussian in image quality, rendering speed, and size; final models average 51.70×, 52.56×, and 51.06× smaller than unpruned 3D-GS across the three datasets.The size-reduction figures correspond respectively to Mip-NeRF 360, Tanks & Temples, and Deep Blending.Source: §5.3.3 · p. 7, Table 3 · p. 7
6. Ablations
The ablations favor two-round pruning, an 80% then 50% pruning schedule, and a sensitivity score based on spatial mean and scale parameters. Including rotations or using RGB parameters lowers image quality, while rotation inclusion also raises computational demands.
6.1. Multi-Round Pruning
- Two-round pruning outperforms pruning 90% in one round across all metrics with the same 10,000 fine-tuning iterations; three rounds are substantially worse at lower pruning percentages and only slightly better at higher ones.The authors choose the simpler two-round approach because both configurations outperform LightGaussian by a much larger margin at 90%.Source: Table 4 · p. 7, §6.1 · p. 7
6.2. Per-Round Pruning Percentages
- Pruning 80% then 50% of the remaining Gaussians optimizes image quality and rendering speed at exactly 90% total pruning, and outperforms LightGaussian across all metrics and percentage permutations.The authors target 90% pruning because preserving visual fidelity at extreme compression ratios is challenging.Source: §6.2 · p. 7
6.3. Spatial vs. Color Parameters
- The mean-and-scale spatial sensitivity score outperforms versions including rotations or using RGB color parameters; adding rotations creates 10 × 10 matrices 2.78× larger and exceeds the RTXA4000’s memory capacity.The method excludes rotations because, when invariances are present, they do not change 3D geometry; image quality also decreases with rotation or RGB parameters.Source: §6.3 · pp. 7–8, Table 5 · p. 8
7. Limitations
PUP 3D-GS relies on a converged scene for its Fisher approximation, which requires a small L1 residual, and its Fisher-matrix memory cost scales with the number of Gaussians. The authors report that prune-refine preserves a small residual after removing 80% of the Gaussians and that the memory cost is not prohibitive on 16GB Nvidia RTX A4000 or larger GPUs.
7. Limitations
- The Fisher approximation assumes a converged scene because it requires a small L1 residual to be accurate.The authors report that prune-refine preserves a small residual after removing 80% of the Gaussians, allowing a second pruning round.Source: §7 · p. 8
- Computing the Fisher matrices requires memory proportional to N × 36, where N is the total number of Gaussians.The authors report this requirement is not prohibitive on 16GB Nvidia RTX A4000 or larger GPUs.Source: §7 · p. 8
8. Conclusion
PUP 3D-GS is a post-hoc pruning pipeline for pretrained 3D-GS models that uses a sensitivity score and multi-round prune-refine. After removing 90% of Gaussians, it increases average rendering speed by 3.56× while retaining more salient foreground information and achieving higher image quality metrics than existing techniques on three datasets.
8. Conclusion
- After pruning 90% of Gaussians, PUP 3D-GS increases average rendering speed by 3.56× and achieves higher image quality metrics than existing techniques across three datasets.The pipeline retains more salient foreground information on Mip-NeRF 360, Tanks & Temples, and Deep Blending scenes.Source: §8 · p. 8
- PUP 3D-GS is a post-hoc pipeline for pruning pretrained 3D-GS models without changing their training pipelines.It uses a sensitivity score derived from a computationally tractable Fisher approximation of the reconstruction-error Hessian over training views.Source: §8 · p. 8
- The pipeline prunes Gaussians and fine-tunes the remaining model over multiple rounds.Its sensitivity score assesses each Gaussian’s sensitivity to reconstruction error over the training views.Source: §8 · p. 8
A.1. Fisher Information Derivation: A Bayesian Interpretation of PUP 3D-GS
The appendix interprets PUP 3D-GS pruning scores through a Bayesian view, then reports ablations, scene evaluations, comparisons, and limitations. Across the visual comparisons, PUP 3D-GS retains foreground detail while acknowledging background degradation at extreme pruning ratios.
A.1. Fisher Information Derivation: A Bayesian Interpretation of PUP 3D-GS
- Under a uniform prior, the Fisher matrix is the posterior Hessian, and each Gaussian’s block log determinant ranks its entropy in the posterior approximation.The derivation formulates the reconstruction objective as a log-likelihood with Gaussian error, then applies a Laplace approximation at the converged scene parameters.Source: §A.1 · p. 11
A.2. Ablation on Patch Size
- The 4 × 4 patches used in the experiments yield slightly better image quality metrics, while 2 × 2 and 8 × 8 patches produce similar results.Table 6 reports mean PSNR, SSIM, LPIPS, FPS, and point cloud size for Mip-NeRF 360.Source: §A.2 · p. 11, Table 6 · p. 11
A.3. Ablation on Per-Round Pruning Percentages
- Across Mip-NeRF 360 scenes, pruning 80% then 50% optimizes image quality and rendering speed at exactly 90% total pruning; our method outperforms LightGaussian across all metrics and per-round pruning permutations.Source: §A.3 · p. 11
A.4. Scene Evaluations
- Scene evaluations report PSNR, SSIM, LPIPS, and FPS after removing exactly 90% of Gaussians; FPS was measured on an Nvidia RTX A4000.The EAGLES bicycle model, already 2.51× smaller than vanilla 3D-GS, becomes 25.14× smaller with PUP 3D-GS while achieving better image quality and rendering speed than LightGaussian.Source: §A.4 · p. 11, §A.5 · p. 12, Table 11 · p. 12
A.6. Comparison with Other Pruning Methods
- At 58.7% pruning on Mip-NeRF 360 bonsai, PUP 3D-GS raises PSNR from 32.25 to 32.55 after one prune-refine round, compared with Compact-3DGS’s increase from 29.87 to 29.91.The paper also reports a 10× reduction in Gaussian count, compared with 2.28× for Compact-3DGS and 6.94× for Mini-Splatting.Source: §A.6 · p. 12
A.7. Potential Directions for Future Research
- The authors identify training-time pruning and selecting subsets of pixels, rays, or views as future directions, while pruning anchor-based methods remains an open problem.They note that direct Gaussian pruning does not work for anchor-based approaches with fixed Gaussian counts per anchor, and pruning anchors is ill-advised because anchors occur where geometry is insufficient.Source: §A.7 · p. 13
A.8. Additional Scene Visualizations
- Visual comparisons show higher fidelity and more retained foreground detail, including individual leaves and legible text; residual comparisons report less L1 error than LightGaussian.Figures 11 and 12 compare residuals against ground truth and base 3D-GS renderings, respectively.Source: §A.8 · p. 13, §A.10 · p. 13, Figure 9 · p. 14
A.9. Background Degradation
- At 90% pruning, some background degradation is observed and lossless compression is not expected; PUP 3D-GS prioritizes foreground detail, unlike LightGaussian’s tendency to favor background preservation.The authors suggest background-score reweighting with masking as a possible way to control this trade-off.Source: §A.9 · p. 13, Figure 10 · p. 14