Source-linked AI summary
Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis
Simon Niedermayr, Josef Stumpfegger, Rüdiger Westermann
TL;DR
3D Gaussian splat representations create storage and rendering challenges for network streaming and low-power devices. Sensitivity-aware clustering, quantization-aware fine-tuning, entropy encoding, and hardware rasterization address these constraints, achieving up to 31× compression and up to a 4× rendering-speed increase with indiscernible quality loss.
Problem
3D Gaussian splat representations create storage and rendering challenges for network streaming and low-power devices.
Method
Sensitivity-aware clustering, quantization-aware fine-tuning, entropy encoding, and GPU rasterization compress and render Gaussian scene parameters.
Results
Up to 31× compression and up to a 4× rendering-speed increase are achieved with indiscernible image-quality loss.
Takeaways & Limitations
The compressed representation can be streamed over networks and rendered on low-power devices for mobile VR/AR applications and games.
Takeaways & Limitations
The method cannot aggressively compress Gaussian positions without significant rendering error.
Abstract
from arXiv · showhide
Recently, high-fidelity scene reconstruction with an optimized 3D Gaussian splat representation has been introduced for novel view synthesis from sparse image sets. Making such representations suitable for applications like network streaming and rendering on low-power devices requires significantly reduced memory consumption as well as improved rendering efficiency. We propose a compressed 3D Gaussian splat representation that utilizes sensitivity-aware vector clustering with quantization-aware training to compress directional colors and Gaussian parameters. The learned codebooks have low bitrates and achieve a compression rate of up to $31\times$ on real-world scenes with only minimal degradation of visual quality. We demonstrate that the compressed splat representation can be efficiently rendered with hardware rasterization on lightweight GPUs at up to $4\times$ higher framerates than reported via an optimized GPU compute pipeline. Extensive experiments across multiple datasets demonstrate the robustness and rendering speed of the proposed approach.
1. Introduction
3D Gaussian splatting offers fast rendering but creates substantial storage and integration challenges for low-end devices. The proposed method compresses scene parameters and uses GPU rasterization to reduce memory demands while improving rendering performance.
- 31× compression is achieved with indiscernible image-quality loss, while rendering speed is greatly improved over [13].Figure 1 reports measurements at 1080p on NVIDIA RTX 3070M and RTX A5000 GPUs.
- 3D Gaussian scenes usually contain millions of Gaussians and require up to several gigabytes, limiting use on devices with restricted video memory.The specialized compute pipeline also cannot be seamlessly integrated with hardware rasterization used for polygonal models.
- The method targets storage and rendering by compressing reconstructed scene parameters and rendering the compressed representation through GPU rasterization.The approach focuses on redundant spherical-harmonic coefficients and multivariate Gaussian parameters, which occupy most of the storage.
- Sensitivity-aware vector quantization encodes color and Gaussian parameters into compact codebooks.A sensitivity measure is derived from each parameter’s contribution to the training images.
- Quantization-aware fine-tuning recovers information lost during clustering, followed by entropy and run-length encoding of spatially ordered Gaussians.The pipeline linearizes Gaussians to exploit spatial coherence before entropy compression.
- Benchmark experiments validate reduced memory consumption, improved rendering performance, network streaming suitability, and contributions from individual pipeline steps.An ablation study evaluates the contribution of each step.
2. Related Work
Prior work improves novel-view rendering through explicit scene representations and compresses grid-based radiance fields, but point-based radiance-field compression remains underexplored. This paper positions its approach as the first to target compact, interactive rendering of such representations on low-end devices.
- NeRF models scenes with neural volumetric radiance fields rendered by volume rendering, but explicit grids and point structures have been introduced to improve efficiency.These alternatives include voxel grids, octrees, hash grids, and point-based representations.
- 3D Gaussian Splatting provides state-of-the-art reconstruction through optimized Gaussian kernels and efficient differentiable rendering.The kernels are adaptively refined and optimized against training images.
- Grid-based NeRF compression uses tensor decomposition, frequency transforms, voxel pruning, and vector quantization to reduce storage costs.These methods address the considerable storage required by full spatial grids.
- Inference-focused methods reduce memory for grid-based representations on low-end devices with limited video memory.The cited approaches target deployment constraints rather than point-based radiance fields.
- The proposed approach is presented as the first to compress point-based radiance fields for high-quality novel-view synthesis at interactive frame rates on low-end devices.This distinguishes it from prior compression work focused primarily on grid-based scene representations.
- Quantization-aware training simulates low-precision weights during training to reduce quantization errors during inference.Prior neural-scene and voxel-based NeRF work reports effective quantization with negligible rendering-quality loss.
3. Differentiable Gaussian Splatting
Differentiable Gaussian splatting represents scenes with optimized 3D Gaussian kernels whose projected footprints are blended to form pixels. Gaussian positions, shapes, opacity, and view-dependent colors are optimized to match training images.
- Each 3D Gaussian is centered at x and described by covariance Σ, opacity α, rotation q, scaling s, and spherical-harmonic color coefficients.The covariance is parameterized through rotation and scaling matrices.
- A 3D Gaussian projects to a 2D Gaussian whose covariance depends on the view transformation and the Jacobian of the projective approximation.The projection provides each Gaussian’s 2D color and opacity footprint.
- A pixel color is computed by blending the contributing 2D Gaussians in sorted order.Each contribution uses a view-dependent color and opacity modulated by exponential falloff from the projected center.
- The Gaussian positions, rotations, scales, opacities, and spherical-harmonic coefficients are optimized so rendered images match the training images.This optimization is performed through differentiable rendering.
4. Sensitivity-Aware Scene Compression
The method compresses 3D Gaussian appearance and shape parameters with sensitivity-aware codebooks, quantization-aware fine-tuning, and entropy encoding while preserving sensitive parameters.
- Sensitivity analysis: Sensitivity varies across Gaussian parameters, so the method measures each parameter’s image-quality impact before clustering.A large gradient magnitude indicates high sensitivity, and sensitivities can be computed with one backward pass per training image.
- Sensitivity-aware vector clustering: Sensitivity-aware k-Means constructs compact codebooks by assigning vectors to centroids using weighted distances based on parameter sensitivity.Centroids are iteratively updated from their assigned vectors.
- Color compression: Less than 5% of SH coefficients are highly sensitive, so those vectors are excluded from clustering while the remaining coefficients are compressed.This selective treatment is intended to keep clustering-induced rendering error low.
- Gaussian shape compression: Gaussian shape compression normalizes each scaling vector, clusters normalized covariance matrices, and stores the scalar scale factor separately.Highly sensitive Gaussian shapes are also excluded from clustering and added to the codebook.
- Quantization-aware fine-tuning: Quantization-aware fine-tuning recovers information lost during clustering while reducing scene parameters to lower-bit representations.The method optimizes scene and codebook parameters on the training images and simulates Min-Max quantization during the forward pass.
- Entropy encoding: Entropy and run-length encoding exploit spatial coherence after Gaussians are ordered along a Morton Z-order curve.The compressed representation stores codebooks for SH coefficients and shape parameters with codebook indices.
5. Novel View Rendering
The renderer uses a compute pre-pass to cull and prepare projected Gaussians, then draws sorted splats with GPU hardware rasterization.
- Preprocess: The renderer discards Gaussians whose projected 99% confidence intervals do not intersect the view frustum.For retained Gaussians, it computes direction-dependent color and stores projection data in an atomic append buffer.
- Rendering: GPU rasterization renders sorted Gaussians as planar quads whose vertices are computed from projected 2D covariance information.Each splat covers the projected Gaussian’s 99% confidence interval.
6. Experiments
Experiments across multiple datasets evaluate compression quality, runtime, rendering speed, and individual pipeline components. The method achieves large memory savings and faster rendering with small image-quality losses, while positional compression remains a limitation.
- Results: 31× maximum and 26× average compression on real-world scenes incur only 0.23 average PSNR loss.A 0.5 PSNR difference is considered indistinguishable to the human eye.
- Rendering Times: 4× faster rendering is achieved through reduced memory bandwidth requirements and hardware rasterization.About a 2× gain comes from reduced bandwidth, with additional speedup from hardware rasterization.
- Loss Contribution: SH coefficient compression causes the largest loss increase but also provides the highest memory reduction.Shape quantization reduces memory by about 60%, quantization-aware fine-tuning by about 50%, and entropy plus run-length encoding adds another 50% saving.
- Codebook Sizes: Codebook size has little effect on average reconstruction error, while larger codebooks reduce maximum error with minimal memory overhead.The analysis averages errors over test images and reports deviations from the maximum error.
- Sensitivity Thresholds: Sensitivity thresholds control which SH vectors and shape parameters are clustered and expose image-quality sensitivity to SH coefficient quantization.The threshold ablation reports average error and maximum deviation from the baseline.
- Limitations: Attempts to compress 3D Gaussian positions further introduced significant rendering error.Experiments quantized positions to a lattice and embedded positional constraints into training without achieving further compression.
7. Conclusion
The paper introduces a compressed 3D Gaussian pipeline that reduces representation size and accelerates rendering while maintaining nearly indistinguishable image quality. The resulting data can support network streaming and rendering on low-power devices, including mobile VR/AR applications and games.
- Conclusion: The pipeline compresses 3D Gaussian color and shape parameters, achieving up to 31× compression and up to 4× faster rendering.These headline results are stated in the conclusion.
- Applications: Compressed data can be streamed over networks and rendered on low-power devices for mobile VR/AR applications and games.The conclusion identifies these as suitable application settings.
- Future Work: Future work targets reducing training-time memory and compressing positional information end-to-end.The paper also proposes investigating advanced compression and rendering options for optimized representations.
A. Detailed Scene Analysis
The detailed analysis reports the evaluation protocol, memory allocation across scene parameters, and the organization of the compressed representation. Random test views are compared against uncompressed and compressed renderings across several datasets.
- Evaluation: PSNR, SSIM, LPIPS, memory consumption, and compression ratio are reported for every scene.Results are provided for Mip-Nerf360, Deep Blending, Tanks&Temples, and synthetic scenes in separate tables.
- Evaluation: Random test views are compared with ground-truth images and renderings from uncompressed and compressed scene representations.The protocol covers all scenes used in the paper.
- Memory Analysis: Gaussian center coordinates and codebook indices generally consume the most memory, while color-codebook size varies substantially by scene.These memory requirements are illustrated in Fig. 7.
D. Timing Statistics
The timing statistics identify fine-tuning as the dominant compression-stage cost, while compressed scenes accelerate renderer preprocessing. The section also reports pruning behavior and covariance-normalization results relevant to the compressed representation.
- Compression pipeline: 70% of the total compression-pipeline time is spent in fine-tuning.Table 6 reports average and maximum stage times, with entropy and run-length encoding grouped into Encoding.
- Novel view renderer: 5× acceleration is achieved in the renderer preprocessing stage with compressed scene representations.Table 7 reports timings in milliseconds for renderer stages on an NVIDIA RTX A5000 using Mip-NeRF360 scenes.
- Sensitivity calculation and pruning: Up to 15% of Gaussians are eliminated because they have zero sensitivity in the color parameters and no impact on training images.Higher pruning thresholds remove more Gaussians with minimal PSNR loss but can remove fine details such as small leaves.
- Covariance matrix clustering: Covariance-matrix clustering produces centroids with normalized scaling vectors.The derivation establishes that the centroid covariance matrix is itself normalized.