LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS

Zhiwen Fan; Kevin Wang; Kairun Wen; Zehao Zhu; Dejia Xu; Zhangyang Wang

Source-linked AI summary

LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS

Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, Zhangyang Wang

arXiv:2311.17245v6Nov 28, 2023cs.CV

TL;DR

3D-GS can expand SfM points into millions of Gaussians, creating gigabyte-level storage demands that hinder scalability and efficiency. LightGaussian combines significance-based pruning and recovery, SH distillation with pseudo-views, and vector quantization to compact these representations. It achieves over 15× data reduction and over 200 FPS while minimally affecting rendering quality, and its pruning generalizes to other Gaussian-based representations.

Problem
3D-GS can require gigabyte-level storage because densification expands SfM points into millions of Gaussians, creating storage and rendering-efficiency challenges.
Method
LightGaussian reduces Gaussian count through pruning and recovery, compacts spherical harmonics through distillation and pseudo-views, and lowers attribute bit-width through vector quantization.
Results
Over 15× data reduction and over 200 FPS are achieved while minimally impacting rendering quality.
Takeaways & Limitations
Gaussian pruning generalizes to other Gaussian-based representations, including Scaffold-GS, where pruning 80% of neural Gaussians increases rendering speed from 152 to 178 FPS.
Takeaways & Limitations
Applying 3D reconstruction in public spaces or with drone footage may infringe on personal privacy.

Abstract

from arXiv · show

Recent advances in real-time neural rendering using point-based techniques have enabled broader adoption of 3D representations. However, foundational approaches like 3D Gaussian Splatting impose substantial storage overhead, as Structure-from-Motion (SfM) points can grow to millions, often requiring gigabyte-level disk space for a single unbounded scene. This growth presents scalability challenges and hinders splatting efficiency. To address this, we introduce LightGaussian, a method for transforming 3D Gaussians into a more compact format. Inspired by Network Pruning, LightGaussian identifies Gaussians with minimal global significance on scene reconstruction, and applies a pruning and recovery process to reduce redundancy while preserving visual quality. Knowledge distillation and pseudo-view augmentation then transfer spherical harmonic coefficients to a lower degree, yielding compact representations. Gaussian Vector Quantization, based on each Gaussian's global significance, further lowers bitwidth with minimal accuracy loss. LightGaussian achieves an average 15x compression rate while boosting FPS from 144 to 237 within the 3D-GS framework, enabling efficient complex scene representation on the Mip-NeRF 360 and Tank & Temple datasets. The proposed Gaussian pruning approach is also adaptable to other 3D representations (e.g., Scaffold-GS), demonstrating strong generalization capabilities.

1 Introduction

3D Gaussian Splatting enables real-time, photo-realistic rendering but can require gigabyte-level storage because densification produces excessive Gaussians. LightGaussian targets this redundancy by reducing Gaussian count and feature dimension while preserving visual quality.

3D-GS supports real-time, photo-realistic rendering for complex and large-scale scenes but may require over 1GB of storage.A typical unbounded 360-degree scene, Bicycle, requires 1.4GB.
LightGaussian addresses storage and rendering-speed issues by targeting redundancy in both Gaussian count (N) and feature dimension (F).
Gaussian Pruning and Recovery removes Gaussians with minimal impact on visual quality and then recovers the representation for smooth adaptation.
SH Distillation compacts higher-degree spherical harmonic coefficients using pseudo-view augmentation, while Vector Quantization reduces precision for less significant Gaussian attributes.Quantization-aware fine-tuning is used to maintain quality.
782MB to 45MB compression is reported with an SSIM decrease of only 0.007 on Mip-NeRF 360, while rendering exceeds 200 FPS.The pruning approach also generalizes to formats such as Scaffold-GS.

2 Related Works

Related work covers efficient neural and point-based scene representations, pruning and quantization, and knowledge distillation. These approaches motivate LightGaussian’s compacting pipeline for Gaussian-based novel view synthesis.

NeRF methods achieve high-quality view synthesis but continue to face slow inference, especially for large-scale scenes.
Point-based representations support applications including shape reconstruction, molecular modeling, cloud rendering, and real-time novel view synthesis.
3D-GS stores extensive attributes for each Gaussian, creating gigabyte-level storage demands and motivating region-based, codebook, view-direction, and binary-mask compression methods.
Pruning reduces model complexity by removing nonsignificant parameters, with unstructured and structured approaches operating at weight or neuron/channel levels.
Vector quantization represents data with discrete codebook entries and assigns each vector to a closest match using mean square error.
Knowledge distillation trains a smaller student model by transferring knowledge from a larger teacher model and has been applied to neural scene representations.

3 Methods

LightGaussian compresses 3D Gaussian Splatting by reducing redundant Gaussian count and feature dimensions while preserving rendering quality. Its pipeline combines significance-guided pruning and recovery, SH distillation with pseudo views, and selective quantization.

3 Methods: LightGaussian targets both Gaussian count (N) and feature dimension (F) to transform dense 3D-GS models into compact representations.The framework applies Gaussian Prune and Recovery, SH Distillation, and Vector Quantization in sequence.
3.1 Background: 3D Gaussian Splatting: 3D-GS represents scenes with explicit Gaussians initialized from SfM points and densified into millions of elements, creating substantial storage demands.Each Gaussian stores position, spherical-harmonic color, opacity, rotation, and scale attributes.
3.3 Distilling into Compact SHs: SH Distillation transfers full-degree spherical harmonics to lower-degree coefficients using teacher–student pixel-intensity supervision and pseudo camera positions sampled around training views.Reducing SH degree lowers feature storage, while pseudo-view augmentation supplies synthesized viewpoints for distillation.
3.2 Gaussian Pruning & Recovery: Gaussian Pruning and Recovery ranks Gaussians by global significance, removes lower-ranked elements, and jointly fine-tunes survivors without additional densification.Gaussian co-adaptation uses photometric loss on the original training views for 5,000 iterations.
3.2 Gaussian Pruning & Recovery: Global significance scores estimate each Gaussian’s contribution across training-view pixels by combining ray intersection, opacity, transmittance, and Gaussian volume.The method normalizes volume using the 90% largest sorted Gaussian volumes and clips the result to avoid excessive influence from floating background Gaussians.
3.4 Gaussian Attributes Vector Quantization: Vector Quantization is applied selectively to SH coefficients because quantizing all Gaussian attributes, including position, rotation, and scale, causes significant accuracy loss.The method focuses quantization on less significant SH elements while retaining essential attributes at higher precision.

4 Experiments

Experiments evaluate LightGaussian on large-scale view-synthesis datasets, ablate its compression components, and test generalization to Scaffold-GS. The method substantially reduces storage while preserving visual quality and improving rendering speed.

Experimental Settings: Experiments compare LightGaussian with Plenoxel, Mip-NeRF360, 3D-GS, Instant-NGP, and VQ-DVGO on Mip-NeRF360 and Tanks and Temples.Mip-NeRF360 contains nine real-world large-scale scenes, while Tanks and Temples supplies additional unbounded scenes.
Baseline Comparison: 0.06 FPS is achieved by compact MLP-based NeRF methods on Mip-NeRF360, while Plenoxel requires 2.1GB for one large-scale scene.These results illustrate the speed and storage constraints motivating compact Gaussian representations.
Quantitative Results: 15× storage reduction cuts Mip-NeRF360 storage from 782MB to 45MB, while rendering exceeds 200 FPS.On Tanks and Temples, storage falls from 380MB to 22MB and rendering speed nearly doubles relative to 3D-GS.
Qualitative Results: LightGaussian retains visual quality comparable to 3D-GS, including challenging scenes with thin structures.The comparison focuses on intricate details and background regions.
Generalization: Pruning 80% of Scaffold-GS neural Gaussians increases rendering speed from 152 to 178 FPS while preserving visual fidelity.This demonstrates that Gaussian Pruning can optimize another Gaussian-based representation.
Ablation Studies: Ablations show that pruning and recovery, SH distillation, pseudo-view augmentation, and significance-guided VQ jointly improve efficiency while limiting quality loss.Reducing SH to 2 degrees lowers SSIM from 0.927 to 0.926, whereas 1 degree lowers it to 0.923.

5 Conclusion, Limitations, and Broad Impact

LightGaussian converts heavy point-based representations into compact formats for efficient novel-view synthesis. The paper reports substantial data reduction and faster rendering, while identifying privacy concerns for some reconstruction applications.

Conclusion: LightGaussian transforms heavy point-based representations into compact formats for efficient novel-view synthesis.The framework identifies and prunes the least significant Gaussians generated through densification.
Conclusion: Over 15× data reduction and rendering above 200 FPS are reported with minimal impact on rendering quality.The conclusion presents these as the framework’s principal efficiency outcomes.
Limitations and Future Work: Zero-shot compression across various 3D-GS-based frameworks remains a future research direction.
Broad Impact: 3D reconstruction may infringe personal privacy when applied in public spaces or with drone footage.This concern is stated as a broad-impact ethical boundary rather than a reported technical failure.

Appendix

The appendix details LightGaussian’s pipeline, significance-score computation, quantization fine-tuning, and ablation visualizations. It specifies the representation dimensions and explains how post-quantization optimization restores quality.

Pipeline: The initial 3D-GS representation stores XYZ, 3-degree SH coefficients, opacity, rotation, and scale for each Gaussian.The listed attribute dimensions are 3, 48, 1, 4, and 3, respectively.
Pipeline: The pipeline computes global significance, prunes the least significant Gaussians, fine-tunes the remainder, and reduces SH degree from 3 to 2.The algorithm then continues with compact SH training and vector quantization.
Significance Scoring: Global significance is computed from Gaussian-ray hit counts weighted by opacity and volume.The criterion is evaluated over rays rendered from sampled training cameras.
Vector Quantization: VQ lowers SSIM from 0.923 to 0.915, while joint optimization improves it back from 0.915 to 0.923.The post-finetuning step jointly adjusts the codebook and other Gaussian attributes.
Ablation Visualization: The ablation visualization compares rendered RGB images and residual maps against ground truth across experiment configurations.The final model, Exp #9, is reported as close to 3D-GS, with co-adaptation and SH distillation mitigating information loss.

6 More Experiment Results

LightGaussian is evaluated on synthetic, indoor, and outdoor scenes, showing substantial compression with preserved or comparable rendering quality and improved efficiency. Module analyses and visual comparisons support the effectiveness of pruning, SH distillation, and vector quantization.

Evaluation setup: The experiments include Mip-NeRF360, Tank & Temple, and the synthetic Blender dataset, with scene-wise evaluation and detailed visualizations.Table 7 reports per-scene results on Synthetic-NeRF, while Tables 8–10 report PSNR, SSIM, and LPIPS comparisons on real-world scenes.
Synthetic-NeRF 360° results: 52.38MB to 7.89MB average storage and 310 to 411 FPS on the Synthetic-NeRF 360° dataset, with only a slight rendering-quality decrease.The evaluation uses eight synthetic objects with controlled camera poses and applies global-significance calculation, SH reduction, and vector quantization.
Qualitative comparisons: LightGaussian achieves comparable rendering quality to 3D-GS and outperforms VQ-DVGO in additional Mip-NeRF360 visual comparisons.Residual maps compare rendered images with ground truth images for 3D-GS, LightGaussian, and VQ-DVGO.
Real-world quantitative results: 782MB to 45MB average model size on Mip-NeRF360 and Tank & Temple, while metrics remain comparable to 3D-GS across all scenes.LightGaussian also reports better rendering quality than Plenoxel, INGP, mip-NeRF360, and VQ-DVGO.
Module analysis: Gaussian Pruning & Recovery removes redundant Gaussians with negligible quality degradation while preserving rendering accuracy.The analysis attributes this effectiveness to global significance based on the splatting principle.
Module analysis: Vector quantization applied to the least important Gaussians further reduces model size in the post-processing analysis.The module analysis reports this result for experiment #7.