PaperlayerView exact revision on arXiv

Source-linked AI summary

Mini-Splatting: Representing Scenes with a Constrained Number of Gaussians

Guangchi Fang, Bing Wang

arXiv:2403.14166v3cs.CV

TL;DR

3DGS can require millions of inefficiently distributed Gaussians, making compact representation difficult while preserving rendering quality. Mini-Splatting reorganizes Gaussian positions through densification and simplification, and reports balanced rendering quality, resource consumption, and storage across benchmarks, with limitations in depth-based modeling of large sky areas and alpha-blending-related failures.

  • Problem

    3DGS uses many Gaussians whose inefficient spatial distribution produces overlapping and under-reconstruction, limiting rendering quality and speed while complicating compact representations.

  • Method

    Mini-Splatting reorganizes Gaussian positions using blur split and depth reinitialization for densification, plus intersection preserving and Gaussian sampling for simplification.

  • Results

    Mini-Splatting achieves a balance among rendering quality, resource consumption, and storage across benchmarks and datasets.

  • Takeaways & Limitations

    The method provides a constrained Gaussian scene representation while preserving high-quality rendering across the evaluated benchmarks.

  • Takeaways & Limitations

    Depth-based reinitialization does not accurately model large sky areas in some Tanks&Temples scans, and alpha blending contributes to depth collapse, object misalignment, and blending-boundary failures.

Abstract

from arXiv · show

In this study, we explore the challenge of efficiently representing scenes with a constrained number of Gaussians. Our analysis shifts from traditional graphics and 2D computer vision to the perspective of point clouds, highlighting the inefficient spatial distribution of Gaussian representation as a key limitation in model performance. To address this, we introduce strategies for densification including blur split and depth reinitialization, and simplification through intersection preserving and sampling. These techniques reorganize the spatial positions of the Gaussians, resulting in significant improvements across various datasets and benchmarks in terms of rendering quality, resource consumption, and storage compression. Our Mini-Splatting integrates seamlessly with the original rasterization pipeline, providing a strong baseline for future research in Gaussian-Splatting-based works. \href{https://github.com/fatPeter/mini-splatting}{Code is available}.

1 Introduction

3DGS can represent scenes with high quality but often uses millions of inefficiently distributed Gaussians. Mini-Splatting reorganizes Gaussian positions through densification and simplification to balance quality, resources, and storage.

  • Motivation: 3DGS models scenes with millions of detailed Gaussians, but clustered centers can constrain rendering quality and speed.Scenes in Mip-NeRF360 may use 1 to 6 million Gaussians.
  • Motivation: The analysis identifies overlapping and under-reconstruction as major spatial-distribution problems that complicate minimal representations preserving rendering quality.These phenomena correspond to clustering in some areas and insufficient modeling in others.
  • Approach: Mini-Splatting reorganizes Gaussian spatial positions rather than directly pruning them.The approach reframes simplification as spatial reorganization.
  • Approach: Densification uses blur split and depth reinitialization, while simplification uses intersection preserving and Gaussian sampling.The strategies leverage screen-space and world-space information to produce denser, more uniform distributions.
  • Results: Mini-Splatting achieves a balance among rendering quality, resource consumption, and storage across multiple benchmarks and datasets.The paper presents this as an outcome of integrating densification, simplification, and further processing.

2 Related Work

Related work spans 3DGS extensions, point-cloud simplification, neural-representation compression, and Gaussian pruning. Existing pruning methods often target storage or local attributes without addressing inefficient Gaussian spatial distribution.

  • 3D Gaussian Splatting: 3DGS enables real-time, high-quality novel-view rendering and has been extended to autonomous driving, human avatars, content generation, and dynamic scenes.The related work positions 3DGS as a recent advancement with broad application activity.
  • 3D Gaussian Splatting: Unlike ray-marching methods requiring dense ray-point sampling, 3DGS represents scenes with large collections of elliptical Gaussians.The representation changes the computational structure of novel-view rendering.
  • Point-Cloud Simplification: Point-cloud simplification commonly uses farthest-point, random, or grid sampling, while other methods preserve feature points or geometric structures.Learning-based methods may integrate sampling with downstream tasks during training.
  • Representation Compression: Neural-representation compression uses voxel pruning or masking, inspiring Gaussian pruning based on image gradients, opacity, or scale.These approaches adapt compression ideas from neural radiance fields to 3DGS.
  • Representation Compression: Prior Gaussian-pruning methods overlook inefficient spatial distribution, which can produce suboptimal simplification after pruning.This motivates methods that address spatial arrangement rather than only Gaussian attributes.

3 Analyzing 3DGS from a Perspective of Point Clouds

Viewing 3DGS as a point-cloud-like representation reveals overlapping and under-reconstruction in Gaussian centers. Standard pruning reduces Gaussian count but does not resolve these spatial problems without degrading PSNR.

  • Gaussian Representation: 3DGS represents scene geometry with 3D Gaussians characterized by opacity, center, and covariance, then renders them through depth ordering and alpha blending.View-dependent color is modeled with spherical-harmonic coefficients, and each Gaussian is projected to 2D.
  • Gaussian Representation: Adaptive density control splits or clones high-gradient Gaussians and prunes low-opacity or oversized Gaussians during optimization.This strategy dynamically adds and removes primitives to control representation density.
  • Point-Cloud Perspective: Gaussian centers can approximate geometric structure, with Gaussian scales corresponding to surface segments under the point-cloud interpretation.The assumption is especially relevant for foreground Gaussians with relatively small 3D scales.
  • Challenges in Gaussian Simplification: Vanilla 3DGS exhibits overlapping clusters and under-reconstructed regions where details are missing or artifacts appear.The figure visualizes projected centers and compares these patterns with rendering quality and Gaussian count.
  • Challenges in Gaussian Simplification: Pruning reduces the Gaussian count from 6.1 million to 0.6 million but lowers PSNR from 25.2 dB to 24.9 dB, while both spatial phenomena remain.Random, grid, and density-preserved sampling are also examined after pruning.
  • Challenges in Gaussian Simplification: The sampling comparisons indicate that achieving a minimal Gaussian representation while maintaining rendering quality requires more than standard pruning and sampling.The figure compares pruning, random sampling, grid sampling, and density-preserved sampling.

4 Methodology

The methodology reorganizes Gaussian positions through densification and simplification rather than direct pruning, using screen- and world-space information to improve spatial coverage while constraining Gaussian count. It addresses blur, under-reconstruction, overlapping, and depth artifacts through targeted splitting, depth-based initialization, intersection preserving, and importance-weighted sampling.

  • Overview: The method reorganizes Gaussian spatial positions instead of directly pruning them to constrain Gaussian count while preserving rendering quality.Its densification and simplification stages use both screen-space and world-space information.
  • Densification: Blur split targets Gaussians with large maximum-contribution areas, which are associated with blurry artifacts and under-reconstruction.The criterion uses S_i, the maximum contribution area, with threshold T_blur = theta_blur · H · W; theta_blur is set to 2 × 10^-4.
  • Densification: Depth reinitialization uses dense depth points to alleviate overlapping and under-reconstruction and initialize a denser, more uniform Gaussian distribution.The pipeline reprojects depth points into world space, samples roughly 3.5 million points per scene, and repeatedly reinitializes Gaussians during optimization.
  • Densification: Mid-point depth avoids alpha-blending artifacts by selecting the maximum-contribution Gaussian for each pixel and using the ray-ellipsoid intersection midpoint as its depth.Center-based depth has comparable rendering quality after optimization, while mid-point depth gives better dense point-cloud reconstruction and is used in the main pipeline.
  • Simplification: Intersection preserving removes Gaussians that do not intersect the ray while retaining smooth opacity or blending weights rather than binarizing them.This is intended to simplify the representation without compromising rendering quality through strict 0-or-1 conversion.
  • Simplification: Importance-weighted sampling incorporates scene geometry to maintain rendering quality with a sparse Gaussian set, unlike importance-only pruning that can damage local geometry.Neighboring Gaussians may share similar importance and be removed together; the experiments report stronger geometry preservation for importance-weighted sampling, with blending-weight importance performing best in pruning and sampling.

5 Implementation Variants

Mini-Splatting provides three implementation variants for resource-efficient training and rendering, following a 30K-step optimization schedule with staged densification and simplification.

  • Implementation schedule: Mini-Splatting uses a 30K-step optimization schedule, enables densification through iteration 15K, and simplifies at iterations 15K and 20K.The variants are described in the appendix and are designed for resource-efficient training and rendering.

6 Experiments

Experiments evaluate Mini-Splatting across rendering quality, resource consumption, storage compression, and ablations. The results show that reorganizing Gaussian positions can preserve or improve quality with fewer Gaussians, while depth-based strategies have a dataset-specific limitation.

  • Experimental Setup: Experiments use Mip-NeRF360, Tanks&Temples, and Deep Blending with processing matched to the official 3DGS implementation.The implementation integrates the densification and simplification algorithms into the 3DGS optimization pipeline.
  • Rendering Quality: Mini-Splatting-D surpasses 3DGS across most rendering metrics, while Mini-Splatting remains comparable to 3DGS with 7× fewer Gaussians.Mini-Splatting-D also surpasses Zip-NeRF in SSIM and LPIPS on Mip-NeRF360.
  • Limitations: Tanks&Temples shows decreased PSNR because large sky areas are not accurately modeled by the depth-based strategy.The paper attributes this dataset-specific behavior to sky regions in the scans.
  • Rendering Quality: At similar Gaussian counts, Mini-Splatting outperforms the compared Gaussian-Splatting baselines, while Mini-Splatting-D provides higher quality with more Gaussians.The comparison is presented through number-quality curves on Mip-NeRF360.
  • Resource Consumption: The efficient representation accelerates training and rendering and reduces peak memory usage, while Mini-Splatting* also runs on a GTX 1060 6G GPU.Mini-Splatting-D has training time and memory consumption similar to 3DGS in outdoor scenes despite using more Gaussians.
  • Densification: Densification improves rendering quality as Gaussian counts increase, with LPIPS highly correlated with the number of Gaussians.The ablation adds blur splitting and depth reinitialization incrementally to the 3DGS baseline.
  • Densification: Mid-point depth reinitialization yields comparable quality to Gaussian-center depth and better dense point-cloud reconstruction than blending depth.Blending depth can degrade quality because of noise points caused by depth artifacts.
  • Simplification: Intersection preserving and sampling reduce Gaussian counts by approximately half relative to direct pruning while maintaining comparable rendering quality.The simplification comparison begins from Mini-Splatting-D and evaluates direct pruning, intersection preserving, and sampling.

7 Conclusion

Mini-Splatting represents scenes with constrained Gaussian counts by densifying and simplifying Gaussian positions. Its variants target rendering quality, resource consumption, and storage compression.

  • Conclusion: Mini-Splatting addresses constrained Gaussian representation through a Gaussian densification and simplification algorithm.The method reorganizes Gaussian spatial positions rather than directly pruning them.
  • Conclusion: Blur splitting and depth initialization construct a dense Gaussian representation, while intersection preserving and Gaussian sampling limit its size while preserving rendering quality.These are the core densification and simplification components described in the conclusion.
  • Conclusion: The experiments report effectiveness across benchmarks in rendering quality, resource consumption, and storage compression.The conclusion summarizes results for Mini-Splatting and its variants.
  • Gaussian Visualization: Elliptical Gaussians are visualized with colors from first-order SH coefficients and normals of their corresponding ellipsoids, excluding opacity below 0.1.The visualization treats Gaussian centers as geometric structure and ellipsoids as surface segments with consistent normals.

B Gaussian Splatting with Dense Initialization

Dense initialization improves quantitative results for all evaluated methods, while Mini-Splatting-D with sparse initialization still outperforms dense-initialized 3DGS on SSIM and LPIPS. Visualizations show that Mini-Splatting variants produce more uniform Gaussian-center distributions than dense-initialized 3DGS.

  • Dense Initialization: Dense point clouds are obtained through MVS and randomly sampled to 2 million points per scene to prevent memory overflow.These dense points replace the original sparse points for initialization.
  • Visual Analysis: Dense-initialized 3DGS still exhibits overlapping around the bicycle’s top tube and spokes, whereas Mini-Splatting-D and Mini-Splatting distribute centers more uniformly.The centers are projected onto the rendered image for visual comparison.
  • Quantitative Results: Dense initialization significantly improves the quantitative results of all evaluated methods.The comparison includes 3DGS and Mini-Splatting variants with sparse and dense initialization on Mip-NeRF360.
  • Quantitative Results: Mini-Splatting-D with sparse initialization achieves superior SSIM and LPIPS scores compared with 3DGS using dense initialization.This result underscores the effectiveness of the densification algorithm in the reported comparison.

C Artifacts of Blending Depth

Alpha blending produces depth artifacts because multiple Gaussians contribute ambiguously along rays. The paper analyzes these failures and derives Gaussian depth from ray–ellipsoid intersections or maximum density.

  • Depth Collapse: Depth collapse assigns unreasonably low background depths when black objects are optimized against 3DGS’s default black background.Low accumulated opacity along the ray causes the corresponding depth to collapse.
  • Object Misalignment: Large Gaussians around objects and front-camera floaters cause depth points to misalign with Gaussian centers.These Gaussians represent reflections and noise, making alpha-blended depth estimates inaccurate.
  • Blending Boundary: Alpha blending creates smooth depth edges because all Gaussians near a boundary receive relatively low blending weights.The resulting blending boundary becomes visible after reprojection into world space.
  • Intersection-Based Depth: The ray–ellipsoid formulation solves a quadratic in t to determine whether intersection points exist and compute their midpoint depth.The discriminant identifies real intersections, while the midpoint parameter provides Gaussian depth.
  • Maximum-Density Depth: An alternative formulation maximizes Gaussian density along the input ray, yielding topt = −B/(2C), numerically equal to the intersection midpoint parameter.The intersection formulation is preferred because it additionally provides a discriminant for testing whether an intersection exists.

E Importance Metric

The paper compares importance metrics for Gaussian sampling and finds that their effectiveness depends on scene characteristics. Blending weights work well indoors, while outdoor scenes require intersection and projected-area information.

  • Metric Comparison: Blending-weight importance provides relatively acceptable pruning performance and is used to construct importance-weighted sampling.Direct pruning of 3DGS with accumulated blending weights outperforms several recent works in the reported comparison.
  • Metric Construction: The first importance metric sums blending weights wij across rays intersecting each Gaussian Gi.K denotes the total number of rays intersecting Gi.
  • Indoor Scenes: Blending-weight importance performs better indoors, where Gaussians generally have low uncertainty and similar scales.This setting is less affected by floaters and scale variation.
  • Outdoor Scenes: For outdoor scenes, the second metric accumulates weights only for Gaussians with ray intersection midpoints and includes projected area to limit preserved Gaussian scale.The metric aggregates per-image importance values for each Gaussian.
  • Metric Design: Importance metrics combine blending weight with other information differently across indoor and outdoor scans, making their design case-dependent and hand-crafted.The paper presents this metric design as an experimental appendix technique.

G Limitation and Future Work

Mini-Splatting’s depth-based reinitialization and manually controlled sampling ratio remain subject to scene-dependent limitations. The paper identifies sky regions and background quality as future-work challenges.

  • Depth Reinitialization: Depth-based Gaussian reinitialization fails in areas without a reliable depth value, such as the sky.The paper suggests multiview consistency and background removal as possible remedies.
  • Sampling Control: The sampling ratio manually controls the Gaussian count, but selecting the minimum count that preserves high-quality rendering remains unresolved.A high sampling ratio can distort background objects; image-wise uncertainty is proposed as a possible remedy.
  • Pipeline Variants: Algorithm 1 separates densification from simplification, with simplification disabled for Mini-Splatting-D.The pipeline uses BlurSplit, DepthReinit, Intersection, and sampling-related stages at specified iterations.

H Additional Quantitative Results

Additional evaluations across three datasets show that Mini-Splatting generally improves over 3DGS and can surpass Zip-NeRF on SSIM and LPIPS. Table 6 reports the quantitative comparison, while Fig. 15 illustrates known limitations.

  • Cross-Dataset Results: Mini-Splatting consistently improves over 3DGS in most scans across all three datasets.The additional evaluation reports improvements in the majority of tested scans.
  • Metric Comparison: Mini-Splatting surpasses Zip-NeRF on SSIM and LPIPS in the additional quantitative results.The comparison is reported across the three evaluated datasets.
  • Limitations: Fig. 15 identifies depth-based failure in sky regions and concentrated background-object distributions when using few Gaussians.These observations provide visual context for the method’s scope boundaries.
  • Evaluation Setup: Table 6 compares Mini-Splatting and prior methods quantitatively, with 3DGS* denoting a retrained official-implementation model.The table is the source of the reported additional evaluation.
Loading 2403.14166v3…