Source-linked AI summary
GaussianPro: 3D Gaussian Splatting with Progressive Propagation
Kai Cheng, Xiaoxiao Long, Kaizhi Yang, Yao Yao, Wei Yin, Yuexin Ma, Wenping Wang, Xuejin Chen
TL;DR
3DGS struggles when SfM provides insufficient initialization in texture-less regions of large scenes. GaussianPro guides densification through progressive propagation using reconstructed geometry, patch matching, and planar constraints, achieving superior rendering results on Waymo and MipNeRF360, including a 1.15 dB PSNR improvement over 3DGS on Waymo.
Problem
3DGS depends on SfM initialization, which fails to provide enough points in texture-less regions of large-scale scenes.
Method
GaussianPro progressively propagates geometric information from reconstructed regions and uses patch matching plus planar constraints to guide Gaussian densification.
Results
1.15 dB PSNR improvement over 3DGS is reported on the Waymo dataset, with superior rendering results also reported on MipNeRF360.
Takeaways & Limitations
GaussianPro produces compact Gaussian representations and improves rendering performance across Waymo and MipNeRF360 while remaining robust to varying numbers of training images.
Abstract
from arXiv · showhide
The advent of 3D Gaussian Splatting (3DGS) has recently brought about a revolution in the field of neural rendering, facilitating high-quality renderings at real-time speed. However, 3DGS heavily depends on the initialized point cloud produced by Structure-from-Motion (SfM) techniques. When tackling with large-scale scenes that unavoidably contain texture-less surfaces, the SfM techniques always fail to produce enough points in these surfaces and cannot provide good initialization for 3DGS. As a result, 3DGS suffers from difficult optimization and low-quality renderings. In this paper, inspired by classical multi-view stereo (MVS) techniques, we propose GaussianPro, a novel method that applies a progressive propagation strategy to guide the densification of the 3D Gaussians. Compared to the simple split and clone strategies used in 3DGS, our method leverages the priors of the existing reconstructed geometries of the scene and patch matching techniques to produce new Gaussians with accurate positions and orientations. Experiments on both large-scale and small-scale scenes validate the effectiveness of our method, where our method significantly surpasses 3DGS on the Waymo dataset, exhibiting an improvement of 1.15dB in terms of PSNR.
1. Introduction
GaussianPro addresses 3DGS’s dependence on incomplete SfM initialization in texture-less regions by guiding Gaussian densification with progressive geometric propagation. It combines reconstructed geometry, patch matching, and planar constraints to improve scene coverage and rendering quality.
- Problem: 3DGS relies on SfM point clouds for Gaussian initialization, but SfM leaves texture-less regions sparsely or incorrectly initialized.This makes reliable densification difficult in those regions.
- Method: GaussianPro progressively propagates geometric information to create more compact and accurate Gaussians, particularly across low-texture surfaces.The strategy uses existing reconstructed geometries as priors and applies patch matching to estimate positions and orientations.
- Method: The method additionally uses a planar loss to provide further constraints during Gaussian optimization.This supplements propagation with a geometric constraint on Gaussian shapes.
- Results: GaussianPro achieves new state-of-the-art rendering performance on Waymo and MipNeRF360 and remains robust to varying numbers of input images.The reported gains cover both large-scale and small-scale datasets.
2. Related Work
Prior work spans neural and explicit scene representations for novel-view rendering, while 3DGS improves speed but remains vulnerable to geometric inaccuracies. GaussianPro addresses this gap by explicitly constraining Gaussian growth toward real surfaces.
- Multi-view Stereo: MVS reconstructs 3D models from posed images by establishing pixel correspondences and optimizing 3D structure.Traditional methods use hand-crafted image features, while learning-based MVS methods extend this reconstruction paradigm.
- Neural Radiance Fields: NeRF produces high-quality novel views with a continuous density field but relies on computationally heavy ray marching.Subsequent work improves sampling, scene parameterization, or geometric regularization.
- 3D Gaussian Splatting: 3DGS uses splatting-based rasterization to avoid ray marching and achieve real-time rendering, but can produce artifacts under changed sampling conditions.Reported remedies include low-pass filtering and other modifications to 3DGS.
- Geometric Limitations: 3DGS may grow Gaussians away from real surfaces, creating redundancy and reducing rendering quality for new viewpoints.GaussianPro therefore incorporates a planar prior to constrain growth near scene surfaces.
3. Preliminaries
3DGS represents scenes with anisotropic Gaussians and renders pixels by depth-sorted alpha blending. Its densification procedure splits or clones Gaussians according to covariance size when reconstruction gradients indicate insufficient representation.
- Scene Representation: 3DGS models a scene as anisotropic 3D Gaussians rendered through splatting-based rasterization.Each Gaussian is represented using a mean vector and covariance matrix.
- Scene Representation: The covariance matrix is parameterized as Σ = RSST RT, using an orthogonal rotation matrix and diagonal scale matrix.This parameterization preserves the covariance matrix’s positive semi-definite property during optimization.
- Rendering: Each pixel color is computed by alpha-blending depth-ordered projected Gaussians that overlap the pixel.The blend uses each Gaussian’s projected opacity and learnable color, and the attributes are optimized through differentiable rendering.
- Densification: 3DGS densifies regions with large rendering gradients by splitting Gaussians with large covariance and cloning those with small covariance.This split-versus-clone decision is based on whether the current Gaussian sufficiently represents its corresponding 3D region.
4. Method
GaussianPro combines 3D Gaussians with view-dependent depth and normal maps, progressively propagating geometric information to guide densification and planar optimization. Patch matching, geometric filtering, and plane constraints produce more accurate Gaussian placement and geometry, especially where existing models are inadequate.
- Progressive Gaussian Propagation: Rendered Gaussian depth and normal maps provide the starting geometry for progressive propagation and subsequent Gaussian generation.The method renders these maps by accumulating Gaussian positions and orientations through alpha blending.
- Progressive Gaussian Propagation: Patch matching selects propagated plane candidates by homography-warping pixels to neighboring views and choosing the candidate with highest color consistency.The selected candidate updates the current pixel’s depth and normal, and propagation is iterated to transmit geometry across larger regions.
- Progressive Gaussian Propagation: Geometric filtering removes unreliable propagated depths and normals, while regions with sufficiently large rendered-versus-filtered depth differences are selected for new Gaussians.Selected pixels are projected into 3D using filtered depth and normal information.
- Plane Constraint Optimization: 3DGS can produce novel-view artifacts when Gaussian shapes deviate from surface geometry, whereas the proposed constraint encourages Gaussians to follow the real surface.The method targets the geometry mismatch that is especially problematic in large-scale scenes with limited views.
- Plane Constraint Optimization: A planar constraint aligns rendered and propagated normals and regularizes Gaussian scale so the shortest axis represents the surface normal and the Gaussian becomes planar.The constraint combines normal consistency with a scale regularization term that drives the minimum Gaussian scale toward zero.
5. Experiment
Experiments on Waymo and MipNeRF360 evaluate GaussianPro against 3DGS and other methods, including ablations, sparse-view robustness, and efficiency analyses. GaussianPro improves rendering quality while maintaining compact representations, with especially strong gains on Waymo.
- Quantitative and Qualitative Results: GaussianPro significantly outperforms competing methods on Waymo across all reported evaluation metrics.Textureless street regions make SfM initialization difficult, while propagation complements missing geometry and planar constraints improve plane modeling.
- Quantitative and Qualitative Results: 1.15 dB PSNR improvement over 3DGS is reported on the Waymo dataset.Visual results also show sharper details and better renderings in both rich-texture and textureless regions.
- Quantitative and Qualitative Results: GaussianPro achieves comparable results with a slight improvement over 3DGS on MipNeRF360.The dataset’s small, richly textured scenes usually provide high-quality SfM initialization, while weak-texture indoor surfaces still benefit from the method.
- Effectiveness of the Propagation Strategy and Planar Constraint: The propagation strategy significantly improves the baseline, and the planar constraint further enhances rendering by modeling plane normals accurately.The ablation study attributes the propagation gain to refined geometry, especially where initial 3DGS geometry contains significant errors.
- Robustness against Sparse Training Images: GaussianPro consistently outperforms 3DGS when trained with 30%, 50%, 70%, or 100% of the training images.This robustness evaluation uses randomly selected subsets from a MipNeRF360 room scene.
- Efficiency Analysis: GaussianPro improves rendering quality with only a slight increase in training time and maintains comparable real-time rendering performance.Compared with direct MVS initialization, GaussianPro avoids approximately 4 times longer training, excessive Gaussian counts, and noticeably slower rendering.
6. Conclusion
GaussianPro guides Gaussian densification according to scene surface structure and adds plane constraints during optimization. It delivers superior rendering results to 3DGS on Waymo and MipNeRF360 while maintaining compact representations and robustness to training-image variation.
- Conclusion: GaussianPro guides Gaussian densification according to scene surface structure and adds plane constraints to model planar surfaces.The method does not specially model dynamic objects and can present artifacts in those regions.
- Conclusion: GaussianPro achieves superior rendering results to 3DGS on Waymo and MipNeRF360 while maintaining compact Gaussian representations.The method shows significant improvements in structured scenes and remains robust to variations in the number of training images.