PaperlayerView exact revision on arXiv

Source-linked AI summary

Compact 3D Scene Representation via Self-Organizing Gaussian Grids

Wieland Morgenstern, Florian Barthel, Anna Hilsmann, Peter Eisert

arXiv:2312.13299v2cs.CV

TL;DR

3DGS offers fast, high-quality rendering but has high storage requirements that limit deployment on constrained devices. The paper reorganizes Gaussian parameters into a smooth 2D grid, uses parallel sorting and training-time smoothness regularization, and achieves 17x to 42x smaller files without sacrificing rendering quality. The resulting representation preserves 3DGS structure for established renderers.

  • Problem

    3DGS combines fast rendering and high quality but stores millions of high-dimensional Gaussians in an unstructured list, creating high storage requirements for practical deployment.

  • Method

    The method exploits equivalent Gaussian-parameter permutations, uses Parallel Linear Assignment Sorting to arrange attributes on a smooth 2D grid, and applies a smoothness loss during training.

  • Results

    17x to 42x reduction in storage size is achieved without sacrificing visual quality, with the decompressed scenes retaining the 3DGS structure.

  • Takeaways & Limitations

    The representation targets compact storage and high-quality, high-speed novel-view synthesis for small devices and fast web applications.

  • Takeaways & Limitations

    Direct comparison with DRACO is not possible because available implementations do not support encoding the additional Gaussian attributes, and post-training compression cannot exploit training-time representation adaptation.

Abstract

from arXiv · show

3D Gaussian Splatting has recently emerged as a highly promising technique for modeling of static 3D scenes. In contrast to Neural Radiance Fields, it utilizes efficient rasterization allowing for very fast rendering at high-quality. However, the storage size is significantly higher, which hinders practical deployment, e.g. on resource constrained devices. In this paper, we introduce a compact scene representation organizing the parameters of 3D Gaussian Splatting (3DGS) into a 2D grid with local homogeneity, ensuring a drastic reduction in storage requirements without compromising visual quality during rendering. Central to our idea is the explicit exploitation of perceptual redundancies present in natural scenes. In essence, the inherent nature of a scene allows for numerous permutations of Gaussian parameters to equivalently represent it. To this end, we propose a novel highly parallel algorithm that regularly arranges the high-dimensional Gaussian parameters into a 2D grid while preserving their neighborhood structure. During training, we further enforce local smoothness between the sorted parameters in the grid. The uncompressed Gaussians use the same structure as 3DGS, ensuring a seamless integration with established renderers. Our method achieves a reduction factor of 17x to 42x in size for complex scenes with no increase in training time, marking a substantial leap forward in the domain of 3D scene distribution and consumption. Additional information can be found on our project page: https://fraunhoferhhi.github.io/Self-Organizing-Gaussians/

1 Introduction

3DGS provides fast, high-quality rendering but stores millions of high-dimensional Gaussians, creating a storage challenge for constrained devices and fast web applications. The proposed method reorganizes Gaussian features into a smooth 2D grid and compresses them while preserving visual quality.

  • Motivation: 3DGS enables very fast rendering at high quality, but its unstructured high-dimensional Gaussian representation requires substantially more storage.Storage size is especially important for real-time applications, portable devices, and fast web applications.
  • Approach: The method exploits equivalent permutations of Gaussian parameters by arranging them into a structured 2D grid with increased local smoothness.A smoothness loss guides the Gaussian arrangement toward a visually equivalent configuration that is easier to compress.
  • Results: 17x to 42x reduction in overall file size is achieved without sacrificing rendering quality.The parameters are quantized and encoded using efficient 2D image compression methods.
  • Implications: The compact representation supports high-quality, high-speed novel-view synthesis with a small storage footprint.The authors target applications on small devices with limited storage capacity and fast web applications.
  • Contributions: The contributions include a smooth 2D-grid representation, a GPU sorting algorithm, compression interfaces, and storage reduction while maintaining visual quality.The decompressed scenes retain the 3DGS structure for integration with established renderers.

2 State of the Art

Prior 3D scene representations trade rendering quality, speed, memory, or storage, while existing grid-sorting methods do not scale adequately to millions of multidimensional Gaussians. This work motivates a scalable approach that maps Gaussian attributes to a smooth 2D grid for more efficient compression.

  • NeRF-based representations: NeRFs deliver high rendering quality but are slow during training and inference.Subsequent methods improve quality or speed, often with trade-offs in memory or quality.
  • Explicit grids: Plenoxels accelerate rendering through sparse 3D grids and interpolation, but their storage grows as O(n3) and quality is lower.The cited comparison describes a speed-quality-storage trade-off for grid-based explicit representations.
  • 3D Gaussian Splatting: 3DGS combines fast rendering with high quality by representing scenes using millions of optimized 3D Gaussians, but stores them in an unstructured list with high storage requirements.Its Gaussian parameters include position, color, size, density, orientation, and spherical harmonics.
  • Grid-based sorting: Ordering ambiguity allows high-dimensional Gaussian data to be mapped onto a similarity-organized 2D grid, facilitating more efficient compression.Related methods include Self-Organizing Maps, Self-Sorting Maps, and Linear Assignment Sorting.
  • Scalability challenge: Existing grid-sorting algorithms are inadequate for millions of Gaussians, motivating continuous sorting during training without increasing training duration.The proposed work adapts and expands sorting methodologies for the scale and dimensionality of 3DGS.
  • Proposed representation: Mapping all Gaussian attributes to shared 2D positions makes neighboring values similar, enabling substantially smaller storage through compression.The representation preserves local spatial relationships while creating multiple data layers with the same layout.

3 Method

The method organizes high-dimensional 3DGS attributes into aligned, smooth 2D grids and compresses them with standard image codecs. It combines parallel iterative sorting, neighborhood regularization, and coordinate-space processing while preserving compatibility with 3DGS renderers.

  • Training scheme: The training scheme adds parallel 2D sorting and local-neighborhood optimization to the 3DGS framework.Sorting establishes smooth neighborhoods, while the neighborhood loss reinforces local smoothness during differentiable rendering.
  • Aligned attribute grids: All Gaussian attributes share aligned grid positions because each position represents one Gaussian splat object.Attribute channels therefore cannot be sorted independently without requiring additional storage to restore alignment.
  • Parallel Linear Assignment Sorting: PLAS sorts millions of multidimensional Gaussians efficiently by approximating the assignment problem with GPU-parallel block processing.It initializes random positions, filters the grid to form targets, shifts independently processed blocks, and reduces the blur radius as assignments converge.
  • Parallel Linear Assignment Sorting: Within each block, groups of four elements are reassigned by selecting the lowest-distance permutation among 24 possibilities.This blockwise operation enables parallel execution while matching Gaussian values to the smoothed grid target.
  • Smoothness regularization: Smoothness regularization penalizes deviations from a Gaussian-filtered version of each sorted attribute grid.A Huber loss is used instead of MSE because it is less sensitive to outliers and improves the compression rate.
  • Storage and coordinates: The method applies logarithmic coordinate-space contraction, stores unactivated parameters with 3DGS activations, and uses quantization plus JPEG XL compression.RGB grids use lossy JPEG XL, while other attributes use lossless JPEG XL; the codec can be replaced by other 2D coding techniques.

4 Evaluation

Across four datasets, the method substantially reduces storage while preserving rendering quality and maintaining fast training and rendering. Evaluation also shows that sorting and smoothness regularization are central to compression effectiveness, while shared sorting limits perfect organization of individual attributes.

  • Quantitative results: 17x to 42x average storage reduction is achieved versus 3DGS without sacrificing visual quality; Deep Blending reaches 41.6x.With spherical harmonics deactivated, the reduction reaches 127x while improving PSNR.
  • Quantitative results: The method achieves slightly higher PSNR and slightly lower L-PIPS than Mip-NeRF360 and VQ-TensoRF on some datasets while retaining real-time rendering and fast training.Training takes 10 to 30 minutes, compared with several hours for Mip-NeRF360.
  • Quantitative results: 515 fps versus 385 fps is measured on Truck for the proposed and vanilla 3DGS scenes, respectively, alongside 1.55M versus 2.58M Gaussians and better visual quality.The comparison uses the same GPU and default viewer settings.
  • Qualitative results: Qualitative comparisons across datasets show rendering quality very similar to vanilla 3DGS while using a fraction of its storage size.Figure 6 presents side-by-side renderings from example scenes across all four datasets.
  • Grid organization: Color values form similar-color clusters in the 2D grids, but the grids do not become more organized with additional iterations.The number of Gaussians grows from about 100k to 1.5 million by iteration 15000, making sorting increasingly difficult; one shared permutation also prevents perfect organization of isolated attributes.
  • Compression analysis: Post-training compression produces much lower PSNR than sorting and smoothing during training, showing that local smoothness is significant for the approach’s success.Compared with zip compression, the method reduces storage by 16x; versus Making Gaussian Splats smaller, it achieves higher PSNR and lower storage size.

5 Conclusion

The proposed representation compresses 3DGS scenes by arranging Gaussian parameters smoothly in a 2D grid and encoding them with standard image-coding techniques. It reduces storage substantially while preserving visual quality, with negligible training-time change; future work targets greater efficiency and dynamic 4D scenes.

  • 5 Conclusion: The method smoothly arranges initially unordered, high-dimensional Gaussian parameters in a 2D grid for standard image coding.The training scheme adds a smoothness loss while preserving the original scene accuracy.
  • 5 Conclusion: The data can be reduced to 2% of the original size at the same visual quality.
  • 5 Conclusion: Training time changes only negligibly compared with the original 3DGS approach.
  • 5 Conclusion: Future work will investigate stronger compression, improved parameter representations, training-time quantization, and extensions to 4D scenes with temporal dependencies.

A Results for SSIM

The SSIM evaluation compares the proposed method with vanilla 3DGS and prior NeRF-based methods. It reports SSIM close to vanilla 3DGS while reducing storage by 17x to 42x depending on the dataset.

  • A Results for SSIM: The SSIM results correlate with the PSNR results reported in table 1.
  • A Results for SSIM: The comparison includes SSIM results for the proposed method, default 3DGS, and prior NeRF-based methods.
  • A Results for SSIM: The proposed method achieves SSIM close to vanilla 3DGS while reducing size by 17x to 42x depending on the dataset.

B Parameter Selection

The paper introduces a brief overview of 3DGS training parameters and identifies the parameters changed for its new training algorithm.

  • B Parameter Selection: 3DGS training involves many different training parameters.
  • B Parameter Selection: The parameter overview is used to highlight the parameters changed for the new training algorithm.
  • B Parameter Selection: The discussion focuses on parameter selection within the 3DGS training process.

B.1 Densification

The method changes default 3DGS densification parameters to reduce the number of Gaussians and minimize sorting time during training. These parameters control densification frequency, gradient-based splitting, opacity filtering and resetting, and size-based splitting or cloning.

  • B.1 Densification: The method reduces the number of Gaussians created during optimization to minimize sorting time during training.It makes minimal changes to the default 3DGS parameters.
  • B.1 Densification: Densification interval determines how often densification is executed, with lower values producing more frequent densification and more Gaussians.
  • B.1 Densification: Densify grad threshold controls which Gaussians are split and cloned according to their accumulated xyz-position gradient.A lower threshold generates more Gaussians during training.
  • B.1 Densification: Densify min opacity filters Gaussians below an opacity threshold, with higher values leading to fewer Gaussians.
  • B.1 Densification: Opacity reset interval sets all Gaussian opacities to 0.01 every x steps.An infinite interval denotes deactivated opacity resetting.
  • B.1 Densification: Percent dense defines the scene-scaled threshold separating small from large Gaussians during densification.Large Gaussians are split into two smaller copies, while small Gaussians are cloned identically.
  • B.1 Densification: The altered densification settings produce fewer Gaussian splats and omit the regular pruning performed every 3k steps by default 3DGS.

B.2 Smoothness Regulation

The method regularizes sorted 2D attribute grids to make Gaussian parameters smoother and more compressible. Sorting keys prioritize attributes affecting PSNR, while other attributes are smoothed over neighborhoods defined by those keys.

  • Smoothness regularization: Smoothness regularization blurs each 2D attribute grid and computes a Huber-loss error term.
  • Smoothness regularization: The regularization introduces kernel size, Gaussian-blur sigma, an overall loss multiplier, and separate multipliers for the five attribute grids.The five grids represent position, color, opacity, scaling, and rotation.
  • Sorting parameters: Sorting after densification uses separate multipliers for position, spherical harmonics, opacity, scaling, and rotation.
  • Parameter selection: Position, color, and scale are sorting keys because they strongly affect PSNR, whereas opacity and rotation are smoothed in neighborhoods instead.Neighborhoods are defined by position, color, and scale.
  • Parameter selection: The same parameter sets were used across all datasets, while exhaustive tuning was deferred until a more compact representation for spherical harmonics is available.

C Sorting performance

The sorting algorithm trades runtime against sorting quality through a relative L2 threshold and scales to large Gaussian grids. It achieves FLAS-comparable quality with substantially shorter runtime, while smoothing is important for preserving visual quality after compression.

  • Threshold sensitivity: Decreasing the relative L2 threshold improves sorting quality but increases runtime through additional iterations.Training experiments fixed the threshold at 10^-4.
  • Algorithm comparison: FLAS sorts a random 512x512x3 grid in 131s with VAD 3.53, while the proposed algorithm takes 5.7s with VAD 4.02.The shuffled grid has VAD 3607.45, and the proposed result follows 8015 reorders.
  • Scaling: With the chosen parameters, sorting usually takes less than 10 seconds during training.
  • Scaling: The largest evaluated model contains 4.37M Gaussians and requires a grid with side length 2091, yet remains sortable in well below a minute.
  • Compression quality: Applying sorting and compression only after vanilla 3DGS training drastically reduces visual quality.The result indicates that smoothing regularization during 3DGS training plays a significant role in efficient attribute compression.
Loading 2312.13299v2…