Source-linked AI summary
Compact 3D Gaussian Representation for Radiance Field
Joo Chan Lee, Daniel Rho, Xiangyu Sun, Jong Hwan Ko, Eunbyung Park
TL;DR
NeRFs are hindered by the computational cost of volumetric rendering, while 3DGS trades that bottleneck for substantial memory and storage demands from numerous Gaussian primitives and attributes. The paper addresses both issues with learnable masking, grid-based color representation, and codebook-based geometry compression. With quantization and entropy coding, it reports over 25× reduced storage, faster rendering, and maintained scene quality compared with 3DGS.
Problem
NeRFs suffer from volumetric-rendering bottlenecks, while 3DGS requires many Gaussians and attributes, creating substantial memory and storage demands.
Method
The framework reduces redundant Gaussians with learnable volume masking and compresses color and geometry using a grid-based neural field and learned codebooks.
Results
Over 25× reduced storage is achieved with quantization and entropy coding while maintaining scene quality and enhancing rendering speed compared with 3DGS.
Takeaways & Limitations
The framework provides compact 3D scene representation with high-quality reconstruction, fast training, and real-time rendering.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRFs) have demonstrated remarkable potential in capturing complex 3D scenes with high fidelity. However, one persistent challenge that hinders the widespread adoption of NeRFs is the computational bottleneck due to the volumetric rendering. On the other hand, 3D Gaussian splatting (3DGS) has recently emerged as an alternative representation that leverages a 3D Gaussisan-based representation and adopts the rasterization pipeline to render the images rather than volumetric rendering, achieving very fast rendering speed and promising image quality. However, a significant drawback arises as 3DGS entails a substantial number of 3D Gaussians to maintain the high fidelity of the rendered images, which requires a large amount of memory and storage. To address this critical issue, we place a specific emphasis on two key objectives: reducing the number of Gaussian points without sacrificing performance and compressing the Gaussian attributes, such as view-dependent color and covariance. To this end, we propose a learnable mask strategy that significantly reduces the number of Gaussians while preserving high performance. In addition, we propose a compact but effective representation of view-dependent color by employing a grid-based neural field rather than relying on spherical harmonics. Finally, we learn codebooks to compactly represent the geometric attributes of Gaussian by vector quantization. With model compression techniques such as quantization and entropy coding, we consistently show over 25$\times$ reduced storage and enhanced rendering speed, while maintaining the quality of the scene representation, compared to 3DGS. Our work provides a comprehensive framework for 3D scene representation, achieving high performance, fast training, compactness, and real-time rendering. Our project page is available at https://maincold2.github.io/c3dgs/.
1. Introduction
The paper presents a compact 3D Gaussian representation that reduces redundant Gaussians and compresses their attributes while preserving reconstruction quality and fast rendering. It combines learnable volume masking, grid-based color representation, and codebook-based geometry compression, achieving substantial storage savings compared with 3DGS.
- Motivation: 3DGS renders quickly through rasterization but requires many Gaussians and substantial attribute storage to preserve high-fidelity images.Its attributes include view-dependent color and covariance, creating a significant memory and storage burden.
- Gaussian reduction: The framework uses volume-based masking to remove non-essential Gaussians while maintaining high performance during training.The masking strategy targets redundant points produced by densification through cloning and splitting.
- Attribute compression: View-dependent colors are extracted from a compact grid-based neural field instead of being stored separately for every Gaussian.The initial implementation uses a hash-based grid selected for compactness and fast processing.
- Attribute compression: Gaussian geometry is represented with learned codebooks, storing a codebook index for each Gaussian rather than duplicating similar scale and rotation patterns.The small codebooks add limited spatial and computational overhead during training.
- Results: Over 25× storage compression is achieved with quantization and entropy coding while maintaining scene quality and improving rendering speed compared with 3DGS.Across datasets, the framework first shows about 15× reduced storage; on Deep Blending, storage efficiency exceeds 28× and rendering speed increases nearly 40%.
2. Related Work
Prior neural and point-based rendering methods address quality, speed, or memory limitations but retain important rendering and storage constraints. 3DGS improves rendering speed through rasterization, yet its many Gaussian primitives and attributes create a substantial memory footprint.
- Neural Radiance Fields: NeRF represents scenes with neural fields and volumetric rendering, but dense ray sampling creates a computational bottleneck for real-time use.This limitation is especially relevant on handheld devices and low-end GPUs.
- Neural Radiance Fields: Grid-based methods improve training speed but can remain slow at inference and require large amounts of memory.Later work reduces memory through factorization, hash grids, quantization, and pruning, while volumetric sampling still limits real-time rendering of large scenes.
- Point-based Rendering: Point-based neural rendering uses rasterization-based splatting and differentiable rasterization to mitigate artifacts associated with simple point clouds.These methods represent points with neural features and often process them through convolutional networks.
- Point-based Rendering: 3DGS can exceed 1GB for a realistic scene because high-quality rendering requires many Gaussians with covariance matrices and high-degree spherical harmonics.The paper targets this parameter-intensive requirement while preserving rendering quality, training speed, and real-time rendering.
3. Method
The method compresses 3D Gaussian scenes by masking unessential Gaussians, replacing per-Gaussian color with a grid-based neural field, and vector-quantizing geometric attributes. It trains these components within a rendering-based framework while reducing computation and storage.
- Each Gaussian represents position, opacity, scale, rotation, and spherical-harmonics view-dependent color for rendering.
- 3.1. Gaussian Volume Mask: Learnable volume-and-transparency masks remove unessential Gaussians using binary masking and a straight-through estimator.Masking uses Gaussian volume and transparency together, which is more effective than using either aspect alone.
- 3.1. Gaussian Volume Mask: Masked Gaussians are eliminated at every densification throughout training, reducing unessential primitives and GPU-memory use.Unlike original 3DGS, masking continues through the entire training process, and removed Gaussians require no stored mask parameter afterward.
- 3.2. Geometry Codebook: Residual vector quantization represents Gaussian scale and rotation with cascaded codebooks, storing selected codebook indices instead of full geometric attributes.The method applies R-VQ across multiple stages and uses the final reconstructed vectors for rendering.
- 3.3. View-Dependent Color: A hash-grid neural field followed by a tiny MLP represents view-dependent color from Gaussian positions and viewing directions.The approach contracts unbounded positions before querying hash grids and uses 0-degree spherical-harmonics components as color inputs.
- 3.4. Training: The model renders from compact Gaussian attributes and is trained end-to-end with rendering, masking, and geometry-codebook losses.R-VQ codebooks are trained with K-means initialization only during the final 1K training iterations to limit computation.
4. Experiment
Experiments evaluate the compact representation on real-world and synthetic datasets against 3DGS, showing substantially lower storage and faster rendering while preserving reconstruction quality.
- Datasets and Setup: The evaluation covers Mip-NeRF 360, Tanks&Temples, Deep Blending, and NeRF-Synthetic, with 30K training iterations and an Ours+PP post-processing variant.Post-processing applies 8-bit quantization, pruning, and Huffman encoding to selected attributes.
- Evaluation Protocol: Tables 1–3 report qualitative results across the real-world and synthetic evaluations, with baseline values drawn from original reports or matched re-evaluations.The 3DGS* baseline is re-evaluated under the authors’ training configuration where specified.
- Real-world Scenes: On real-world scenes, the method reduces storage and increases rendering speed while achieving reconstruction comparable to 3DGS.On Deep Blending, it achieves almost 40% faster rendering and over 15× compactness than 3DGS.
- Synthetic Scenes: On NeRF-Synthetic, the method achieves over 10× compression and 50% faster rendering while maintaining high-quality reconstruction.Training duration is slightly higher than for 3DGS in this setting.
GT 3DGS Ours
The ablations examine masking, view-dependent color, geometry codebooks, and post-processing, showing that each contributes to compactness or efficiency while preserving rendering quality.
- Learnable Volume Masking: Learnable volume masking reduces Gaussian count while retaining or slightly increasing visual quality.On Playroom, masking yields a 140% increase in storage efficiency and a 65% increase in rendering speed.
- Compact View-dependent Color: The neural-field color representation provides more than threefold storage efficiency over directly storing high-degree SH.It uses a slightly reduced number of Gaussians but requires slightly more training and rendering time; with masking, rendering is faster or comparable to 3DGS.
- Geometry Codebook: The geometry codebook reduces storage requirements by approximately 30% while maintaining reconstruction quality, training time, and rendering speed.Visualizations show mostly identical Gaussian geometry after residual vector quantization, with only subtle distortions in a few primitives.
- Post-processing: Simple post-processing effectively reduces the relatively large storage cost of the end-to-end color representation.Table 5 compares per-attribute storage before and after quantization, pruning, and entropy coding.
5. Conclusion
The paper presents a compact 3D Gaussian representation that reduces storage and increases rendering speed while retaining high-quality reconstruction compared with 3DGS.
- 5. Conclusion: The framework combines volume-based masking, a neural field for view-dependent color, and learnable codebooks for compact Gaussian attributes.Its experiments report more than tenfold lower storage, faster rendering, and high-quality reconstruction relative to 3DGS.
6. Implementation Detail
Implementation retains 3DGS hyper-parameters while specifying the vector-quantization, hash-grid, and MLP configurations used for compact representation.
- 6. Implementation Detail: The implementation uses codebook size C=64 and R-VQ stage count L=6, with hash-grid color features across 16 resolutions and a 2-layer 64-channel MLP.Positions and opacities use half-tensor storage, and real and synthetic scenes receive different maximum hash-map sizes and learning hyper-parameters.
7. Fast inference pipeline
The inference pipeline reduces rendering time by lowering the number of Gaussians and precomputing grid features during testing.
- The masking strategy reduces the number of Gaussians, lowering both training and rendering times.
- Precomputing grid features during testing further minimizes inference computation.
8. Additional ablation study
Additional ablations examine how compactness affects quality and show that the proposals have different robustness profiles across scenes and rate settings.
- 8. Additional ablation study: The study varies masking, hashmap size, and R-VQ stages across scenes achieving over 10× compression.
- 8. Additional ablation study: The geometry codebook performs poorly when codebook diversity is extremely limited.
- 8. Additional ablation study: The neural-field color representation remains robust in low-rate conditions across various scenes.
- 8. Additional ablation study: Storage can be halved with minimal performance loss, making the default configuration a well-rounded choice across scenes.
- 8. Additional ablation study: Masking based on both Gaussian volume and transparency achieves the best results.
- 8. Additional ablation study: Neural fields efficiently represent continuous signals, while vector quantization suits repetitive components such as shared geometry patterns.
9. Inference memory
The method is evaluated for inference memory and per-scene novel-view synthesis across multiple datasets, with consistently reduced GPU memory requirements.
- 9. Inference memory: The method consistently reduces GPU memory requirements across diverse scenes.
- 9. Inference memory: Per-scene novel-view synthesis results are reported on Mip-NeRF 360, Tanks&Temples, Deep Blending, and NeRF-synthetic datasets.