Source-linked AI summary
Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images
Jaeyoung Chung, Jeongtaek Oh, Kyoung Mu Lee
TL;DR
Few-shot 3D Gaussian Splatting can overfit because limited images provide insufficient global geometry, leading to incorrect convergence and floating artifacts. The paper aligns monocular dense depth with sparse COLMAP points and uses it, together with smoothness and early stopping, to regularize optimization. On NeRF-LLFF, the method outperforms 3DGS in few-shot settings and produces plausible geometry, though it depends on monocular depth and COLMAP quality.
Problem
3D Gaussian Splatting tends to overfit with few images because local splats lack sufficient global geometric cues, producing incorrect geometry or floating artifacts.
Method
The method aligns dense relative depth from a pre-trained monocular estimator with sparse COLMAP points and uses depth guidance, smoothness, and early stopping during Gaussian Splatting optimization.
Results
The method outperforms 3D Gaussian Splatting in few-shot NeRF-LLFF experiments and creates plausible geometry.
Takeaways & Limitations
Adjusted estimated depth can provide effective geometric regularization for Gaussian Splatting when only a small number of images is available.
Takeaways & Limitations
Performance depends on the monocular depth model’s learned domain and on COLMAP, which may fail on textureless or challenging surfaces.
Abstract
from arXiv · showhide
In this paper, we present a method to optimize Gaussian splatting with a limited number of images while avoiding overfitting. Representing a 3D scene by combining numerous Gaussian splats has yielded outstanding visual quality. However, it tends to overfit the training views when only a small number of images are available. To address this issue, we introduce a dense depth map as a geometry guide to mitigate overfitting. We obtained the depth map using a pre-trained monocular depth estimation model and aligning the scale and offset using sparse COLMAP feature points. The adjusted depth aids in the color-based optimization of 3D Gaussian splatting, mitigating floating artifacts, and ensuring adherence to geometric constraints. We verify the proposed method on the NeRF-LLFF dataset with varying numbers of few images. Our approach demonstrates robust geometry compared to the original method that relies solely on images. Project page: robot0321.github.io/DepthRegGS
1. Introduction
Few-shot 3D Gaussian Splatting can overfit because independent splats lack global geometric cues, producing incorrect geometry and floating artifacts. The proposed method uses depth guidance, smoothness regularization, and early stopping to improve geometric optimization from limited images.
- Motivation: 3D Gaussian Splatting is prone to overfitting with few images because independent splats lack global geometric supervision.Insufficient geometric information can cause convergence to local optima, optimization failure, or floating artifacts.
- Motivation: Depth provides direct geometric information, but practical depth sensors require additional equipment.This motivates estimating dense depth from images instead of relying on dedicated RGB-aligned depth hardware.
- Proposed approach: The method adjusts monocular relative depth using sparse COLMAP points to resolve scale ambiguity and obtain a dense geometric guide.Sparse COLMAP points alone cannot regularize all Gaussian splats in few-shot settings, while monocular depth requires scale alignment across views.
- Proposed approach: Depth-guided Gaussian Splatting uses a pre-trained monocular depth model and smoothness constraint to optimize scenes from few RGB images.The adjusted depth assists color-based optimization, reduces floating artifacts, and encourages geometrically plausible solutions.
- Optimization strategy: The method introduces early stopping when the depth-guided loss begins to rise and evaluates each strategy through ablation studies.A smoothness term is reported to direct optimization toward correct geometry, with comprehensive experiments showing enhanced performance.
2. Related Work
Related work spans classical multi-view reconstruction, neural radiance fields, and explicit representations such as 3D Gaussian Splatting. Few-shot reconstruction remains difficult because limited images yield sparse geometric evidence and can produce floating artifacts.
- Multi-view reconstruction: SfM and MVS reconstruct 3D structures from multiple images, while COLMAP estimates camera poses and sparse 3D keypoints.These methods provide geometric foundations for image-based reconstruction.
- Novel view synthesis: NeRF represents scenes with an MLP and volume rendering, while later work pursues faster rendering through explicit representations.Examples include sparse voxels, feature point clouds, tensors, and polygons.
- Few-shot reconstruction: Few-shot reconstruction is challenging because reliable 3D points and camera poses generally improve as more images are used.Both COLMAP and NeRF typically rely on many multi-view images for high-quality scenes.
- Few-shot reconstruction: 3D Gaussian Splatting can generate floating artifacts with few images because its splats operate locally and sparse COLMAP points cannot guide every splat.This motivates adding denser global geometry information.
3. Method
The method uses a scale-adjusted monocular depth prior, aligned with sparse COLMAP points, to guide Gaussian splatting toward plausible geometry from few images. It combines rendered-depth supervision, smoothness regularization, and few-shot-specific optimization changes.
- 3.1. Preparing Dense Depth Prior: A pretrained monocular depth model supplies dense depth, whose scale and offset are fitted to sparse COLMAP depth to resolve scale ambiguity.The adjusted depth is then used as a geometry guide for Gaussian-splatting optimization.
- 3.2. Depth Rendering through Rasterization: The rasterization pipeline renders Gaussian-splat depth by alpha-weighting splat depths, enabling rapid depth rendering with minimal computational load.The rendered depth is guided toward the adjusted dense prior using an L1 distance.
- 3.3. Unsupervised Smoothness Constraint: An edge-aware smoothness constraint regularizes adjacent depths while excluding image boundaries with significant depth differences.A Canny edge mask prevents smoothing across likely geometric discontinuities.
- 3.3. Unsupervised Smoothness Constraint: The final objective combines the original color and D-SSIM losses with depth and smoothness losses weighted by λ_depth and λ_smooth.This preserves the original appearance terms while adding geometric regularization.
- Few-Shot Optimization: For few-shot optimization, the method limits spherical-harmonic degree to 1, uses moving-averaged depth loss, and removes periodic opacity resets.These modifications address convergence and overfitting behavior observed under limited image information.
4. Experiment
Experiments on NeRF-LLFF evaluate the method with 2–5 randomly selected training views, comparing 3DGS, the proposed method, and oracle depth-guided reconstruction. Depth guidance improves geometric plausibility and reconstruction quality in few-shot settings, while ablations identify essential components and initialization trade-offs.
- Experiment settings: Experiments optimize NeRF-LLFF scenes using k-shot settings with k=2,3,4,5 randomly selected training images and a shared test set.The dataset contains 8 forward-facing scenes, and each experiment uses ten random selections.
- Qualitative comparison: Qualitative comparisons in 2-view and 5-view settings show that our method establishes more plausible geometry than 3DGS.The comparison includes input views, 3DGS, the proposed method, and ground truth in cropped patches.
- Quantitative comparison: The proposed method typically outperforms 3DGS, particularly when the number of input images is limited.Across all methods and scenes, reducing the number of images lowers visual quality.
- Oracle depth: Accurate pseudo-GT depth produces detailed and reliable reconstructions with limited images, highlighting the value of strong geometric cues.Fine details are visible in both RGB and depth outputs.
- Ablations: Ablations show that absolute depth adjustment is necessary, smoothness adds geometric cues, and early stopping prevents performance degradation from overfitting.Unadjusted monocular depth can cause training failure, while depth loss monitors divergence from the geometry guide.
- Initialization: Unprojecting dense depth into initialization points performs worse than sparse COLMAP initialization because the numerous points are not effectively merged or pruned.Using all COLMAP points can improve outcomes through favorable initial points unavailable with k images.
5. Limitation and Future Work
The approach is constrained by its reliance on monocular depth estimation and COLMAP, with performance varying across domains and failing where COLMAP cannot recover reliable points.
- Performance depends heavily on the monocular depth estimation model and the domain of its learned data.
- Fitting estimated depth to COLMAP points makes the method unable to handle textureless plains or challenging surfaces where COLMAP fails.
- Future work includes optimizing scenes using interdependent estimated depths rather than COLMAP points.
- The authors also propose geometry regularization across datasets, including regions such as the sky where depth estimation is difficult.
6. Conclusion
The paper introduces a depth-regularized model for learning 3D Gaussian splats from few images. It combines monocular depth estimation with SfM-based scale adjustment and reports improved few-shot reconstruction with plausible geometry.
- The paper introduces a model for learning 3D Gaussian splatting from a small number of images.
- Depth regularizes Gaussian splats using monocular depth estimation whose scale is adjusted based on SfM points.
- The method outperforms 3D Gaussian splatting in a few-shot setting and creates plausible geometry.
- Additional experiments show that improved depth and initialization points significantly enhance Gaussian-splatting-based 3D reconstruction.