Source-linked AI summary
Pixel-GS: Density Control with Pixel-aware Gradient for 3D Gaussian Splatting
Zheng Zhang, Wenbo Hu, Yixing Lao, Tong He, Hengshuang Zhao
TL;DR
3DGS depends heavily on initial point-cloud quality, causing artifacts where initialization is sparse. Pixel-GS uses pixel-aware gradient weighting and camera-distance gradient scaling, and experiments report state-of-the-art rendering quality with real-time speed on Mip-NeRF 360 and Tanks & Temples.
Problem
3DGS can produce blurry and needle-like artifacts in regions with insufficient initial points because its density-growth condition does not adequately grow large Gaussians observed across multiple viewpoints.
Method
Pixel-GS weights per-view gradients by Gaussian pixel coverage and scales the gradient field by camera distance to suppress floater growth.
Results
Pixel-GS achieves state-of-the-art rendering quality while maintaining real-time rendering speed on Mip-NeRF 360 and Tanks & Temples.
Takeaways & Limitations
Pixel-GS more effectively grows points in sparsely initialized regions, reduces blurring, needle-like artifacts, and floaters, and remains robust to initialization-point quality.
Abstract
from arXiv · showhide
3D Gaussian Splatting (3DGS) has demonstrated impressive novel view synthesis results while advancing real-time rendering performance. However, it relies heavily on the quality of the initial point cloud, resulting in blurring and needle-like artifacts in areas with insufficient initializing points. This is mainly attributed to the point cloud growth condition in 3DGS that only considers the average gradient magnitude of points from observable views, thereby failing to grow for large Gaussians that are observable for many viewpoints while many of them are only covered in the boundaries. To this end, we propose a novel method, named Pixel-GS, to take into account the number of pixels covered by the Gaussian in each view during the computation of the growth condition. We regard the covered pixel numbers as the weights to dynamically average the gradients from different views, such that the growth of large Gaussians can be prompted. As a result, points within the areas with insufficient initializing points can be grown more effectively, leading to a more accurate and detailed reconstruction. In addition, we propose a simple yet effective strategy to scale the gradient field according to the distance to the camera, to suppress the growth of floaters near the camera. Extensive experiments both qualitatively and quantitatively demonstrate that our method achieves state-of-the-art rendering quality while maintaining real-time rendering speed, on the challenging Mip-NeRF 360 and Tanks & Temples datasets.
1 Introduction
3DGS can produce blurry and needle-like artifacts when sparse initialization leaves regions underrepresented. Pixel-GS changes gradient-based growth using pixel coverage and adds gradient scaling to improve reconstruction and suppress floaters.
- 1 Introduction: Sparse SfM initialization leaves some regions underrepresented, and 3DGS consequently struggles to grow points there, producing blurry or needle-like artifacts.The problem is linked to 3DGS’s adaptive density control and its dependence on initial point-cloud quality.
- 1 Introduction: 3DGS averages gradient magnitudes equally across viewpoints, although large Gaussians can cover very different numbers of pixels across those views.This mismatch can reduce the effective growth signal for large Gaussians in sparse regions.
- 1 Introduction: Pixel-GS weights per-view gradients by the number of pixels covered by each Gaussian, amplifying growth for large Gaussians while leaving small-Gaussian growth largely unchanged.The method is designed to grow points in areas containing large Gaussians and insufficient initialization points.
- 1 Introduction: Pixel-GS also introduces gradient scaling based on camera distance to suppress floater artifacts near the camera.This strategy is presented alongside the pixel-aware growth condition as a contribution.
2 Related Work
Related work spans implicit neural radiance fields, point-based radiance-field representations, and methods for mitigating floater artifacts. 3DGS provides fast rasterized rendering but remains connected to point-cloud continuity and initialization challenges.
- 2 Related Work: NeRF-based methods fit radiance fields with implicit functions and use volumetric rendering, but process sampled points through neural networks during rendering.The related-work discussion contrasts these approaches with point-based rasterization.
- 2 Related Work: Point-based radiance fields represent scenes with points and commonly render them by GPU rasterization, while differentiable point methods address holes and outliers through local-domain modeling.Point-NeRF is cited as using pruning and growth to repair such artifacts.
- 2 Related Work: 3DGS uses sparse SfM initialization, three-dimensional Gaussian distributions, and spherical harmonics while substantially accelerating rendering through rasterization.Its representation fits each point’s influence area and color features.
- 2 Related Work: Floater artifacts commonly appear near cameras and become more severe with sparse input views, motivating depth priors, editing methods, and distortion-based regularization.These approaches address the artifact through different scene-representation or rendering constraints.
3 Method
Pixel-GS modifies 3DGS density control by weighting viewpoint gradients according to covered pixels, then scales the gradient field by camera distance to suppress floaters.
- 3.1 Preliminaries: Pixel-aware Gradient counts a Gaussian’s participating pixels using projection, influence-radius, opacity, and accumulated-transmittance conditions for each viewpoint.The pixel contribution decreases exponentially with distance from the projected Gaussian center.
- 3.1 Preliminaries: The original 3DGS growth rule averages NDC-coordinate gradient magnitudes across participating viewpoints and splits or clones a Gaussian when the average exceeds τpos.The rule is evaluated during adaptive density control, with Gaussian i participating across M_i viewpoints.
- 3.2 Pixel-aware Gradient: Sparse initial SfM regions create large Gaussians that participate across many viewpoints but obtain larger gradients mainly when their projected centers receive sufficient pixel coverage.Edge-only views contribute smaller NDC-coordinate gradients, helping explain why the original averaging can fail to grow these regions effectively.
- 3.2 Pixel-aware Gradient: Pixel-aware Gradient weights each viewpoint’s NDC-coordinate gradient by the number of participating pixels when deciding whether a Gaussian splits or clones.Large Gaussians therefore receive stronger contributions from views with greater pixel coverage, while smaller Gaussians change little under the modified averaging.
- 3.3 Scaled Gradient Field: Because pixel-based optimization can amplify near-camera gradients, Pixel-GS scales the NDC gradient field according to camera distance to suppress floater growth.The method identifies near-camera floaters as occupying large screen areas and having significant NDC-coordinate gradients.
4 Experiments
Experiments evaluate Pixel-GS on 30 scenes from Mip-NeRF 360 and Tanks & Temples using reconstruction quality, efficiency, qualitative comparisons, ablations, and initialization sparsity tests. Pixel-GS consistently outperforms competing methods and remains robust when initial points are sparse, while its components and growth strategy reveal dataset- and efficiency-related trade-offs.
- 4.2 Main Results: Pixel-GS consistently outperforms competing methods on Mip-NeRF 360 and Tanks & Temples, especially on LPIPS, while maintaining real-time rendering speed.The evaluation compares against NeRF methods, original 3DGS, and retrained 3DGS* using PSNR, SSIM, and LPIPS.
- 4.4 Analysis: The pixel-aware gradient and scaled gradient field both improve reconstruction quality on Mip-NeRF 360, but pixel-aware gradients reduce quality on Tanks & Temples.The component contributions are evaluated quantitatively and qualitatively across both datasets.
- 4.4 Analysis: Qualitative comparisons examine three Mip-NeRF 360 scenes and four Tanks & Temples scenes, with highlighted regions marking distinct quality differences.The comparison uses Pixel-GS and retrained 3DGS* across Bicycle, Flowers, Treehill, Barn, Caterpillar, Playground, and Train.
- 4.4 Analysis: Lowering 3DGS’s point-growth threshold increases memory use and reduces rendering speed, yet still produces lower reconstruction quality than Pixel-GS.The comparison uses a threshold reduction from 2e-4 to 1.28e-4 to make optimized point counts comparable.
- 4.4 Analysis: Pixel-GS remains more robust than 3DGS as increasing proportions of Mip-NeRF 360 initialization points are dropped.The dropping rate denotes the percentage of original SfM points removed before Gaussian initialization.
5 Conclusion
Pixel-GS addresses 3DGS artifacts caused by insufficient initialization points by weighting gradients according to covered pixels and scaling gradients by camera distance. The method improves detail and artifact suppression while preserving real-time rendering and robustness to initialization quality.
- 5 Conclusion: Pixel-GS weights per-view gradients by Gaussian-covered pixels to grow large Gaussians in under-initialized regions while avoiding unnecessary growth elsewhere.The method targets blurring and needle-like artifacts associated with insufficient initial points.
- 5 Conclusion: Scaling the gradient field by camera distance suppresses floaters near cameras while Pixel-GS achieves state-of-the-art rendering quality and real-time speed.The method uses slightly more memory because added points are concentrated in under-initialized regions.
A Additional Results
Pixel-GS improves reconstruction in under-sampled regions while using fewer points and shows consistent scene-level gains, especially on LPIPS. Per-scene tables and Figure 6 provide quantitative and qualitative evidence across the evaluated datasets and ablations.
- Additional Results: LPIPS improves consistently across the vast majority of scene-level scenarios, with stronger perceptual gains than those reflected by PSNR and SSIM.The authors identify LPIPS as more reflective of human visual perception than PSNR and SSIM.
- Additional Results: The per-scene quantitative results are broken down for Mip-NeRF 360, Tanks & Temples, and the corresponding ablation studies.Tables 5–10 report scene-level metrics for the main evaluations and ablations.
- Additional Results: Pixel-GS achieves better reconstruction with fewer points and distributes newly grown points more uniformly than reduced-threshold 3DGS in sparse regions.Figure 6 contrasts rendering quality, memory consumption, and point-cloud distributions for the methods.
- Additional Results: Lowering 3DGS’s growth threshold still tends to add points in already dense areas, whereas Pixel-GS enhances modeling where initial points are insufficient.The comparison indicates that threshold reduction alone does not target sparse regions as effectively.