PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseGS: Sparse View Synthesis using 3D Gaussian Splatting

Haolin Xiong, Sairisheek Muttukuru, Hanyuan Xiao, Rishi Upadhyay, Pradyumna Chari, Yajie Zhao, Achuta Kadambi

arXiv:2312.00206v4cs.CVcs.LGeess.IV

TL;DR

Sparse-view 3DGS can produce floaters and background collapse because sparse observations leave scene geometry ambiguous. SparseGS combines depth rendering and priors, unseen-view regularization, and floater pruning, and reports high-quality reconstruction across unbounded and forward-facing settings with very few input images.

  • Problem

    Sparse input views leave 3DGS vulnerable to projection ambiguity and reconstruction artifacts such as floaters and background collapse.

  • Method

    SparseGS combines softmax-scaling and mode-selection depth rendering, Unseen Viewpoint Regularization, depth warping, diffusion guidance, and explicit floater pruning.

  • Results

    SparseGS reports state-of-the-art sparse-view novel view synthesis performance on forward-facing and 360-degree unbounded scenes.

  • Takeaways & Limitations

    SparseGS supports coherent reconstruction of 360-degree unbounded scenes using as few as 12 input images.

Abstract

from arXiv · show

3D Gaussian Splatting (3DGS) has recently enabled real-time rendering of unbounded 3D scenes for novel view synthesis. However, this technique requires dense training views to accurately reconstruct 3D geometry. A limited number of input views will significantly degrade reconstruction quality, resulting in artifacts such as "floaters" and "background collapse" at unseen viewpoints. In this work, we introduce SparseGS, an efficient training pipeline designed to address the limitations of 3DGS in scenarios with sparse training views. SparseGS incorporates depth priors, novel depth rendering techniques, and a pruning heuristic to mitigate floater artifacts, alongside an Unseen Viewpoint Regularization module to alleviate background collapses. Our extensive evaluations on the Mip-NeRF360, LLFF, and DTU datasets demonstrate that SparseGS achieves high-quality reconstruction in both unbounded and forward-facing scenarios, with as few as 12 and 3 input images, respectively, while maintaining fast training and real-time rendering capabilities.

1. Introduction

SparseGS targets sparse-view 3DGS artifacts by combining depth-based supervision, unseen-view regularization, and explicit floater pruning. The framework is designed for coherent reconstruction of unbounded scenes from as few as 12 input images.

  • Motivation: Sparse input views make 3DGS vulnerable to projection ambiguity, floaters, and background collapse, especially when scene coverage is limited.Floaters arise from misplaced Gaussians, while background collapse results from Gaussians placed at incorrect depths.
  • Method: SparseGS introduces softmax-scaling and mode-selection depth rendering to address floater artifacts in limited-input 3DGS.These techniques go beyond conventional alpha-blending depth rendering.
  • Method: SparseGS uses an explicit adaptive pruning operator to remove undesirable floating Gaussians from the reconstructed scene.The procedure exploits the manipulability of explicit 3DGS representations.
  • Scope and outcome: SparseGS reports state-of-the-art sparse-view synthesis performance on both forward-facing datasets and 360-degree unbounded scenes.The paper identifies 360-degree unbounded scenes as a setting that many existing few-shot techniques struggle to handle.
  • Method: Unseen Viewpoint Regularization mitigates overfitting by regularizing 3DGS from viewpoints different from the input views.The module is intended to reduce background collapse in sparse-input settings.

2. Related Work

Research on neural scene representations balances rendering fidelity with efficiency, while sparse-view synthesis remains difficult because limited observations weaken reconstruction quality. SparseGS builds on 3DGS by combining explicit point-based rendering with depth, diffusion, and pruning components.

  • Radiance Fields: Neural Radiance Fields provide high-quality rendering and anisotropic view interpolation but require long training times and can become blurry with sparse inputs.These limitations motivate more efficient and sparse-view-capable representations.
  • 3D Gaussian Splatting: 3D Gaussian Splatting reduces training cost through a compact explicit representation and supports real-time high-resolution rendering.Its splats are parameterized by position, rotation, scale, opacity, and view-dependent color coefficients.
  • Few-Shot Novel View Synthesis: Sparse-view novel view synthesis methods include multi-plane image approaches and NeRF-based methods that constrain view variation or use related priors.These methods address the need to synthesize views from few input images.
  • SparseGS: SparseGS combines depth priors, diffusion constraints, and floater pruning to improve few-shot novel view synthesis.Its training pipeline aligns rendered softmax depth with dpt, generates novel views, applies Score Distillation Sampling, and periodically prunes floaters.

3. Methods

SparseGS combines depth-aware supervision, unseen-view regularization, and pruning to improve sparse-view 3D Gaussian reconstruction. Its depth rendering and image-based regularization target floaters, overfitting, background collapse, and high-frequency artifacts.

  • Overview: SparseGS uses depth correlation loss, Unseen Viewpoint Regularization, and floater pruning to improve view consistency and depth accuracy.UVR combines Score Distillation Sampling for distant viewpoints with depth warping for nearby viewpoints.
  • Depth Rendering: Alpha-blending depth can adjust transmittance of misplaced Gaussians instead of correcting their positions, producing geometrically incorrect reconstructions.The problem arises because all Gaussians along a ray influence the rendered depth.
  • Depth Rendering: Mode-selection chooses the highest-contributing Gaussian, while softmax-scaling emphasizes high-weight Gaussians and still propagates gradients to off-mode Gaussians.Softmax-scaling addresses the limitation of mode selection when closer Gaussians are translucent.
  • Depth Correlation Loss: Patch-based Pearson depth correlation aligns Gaussian-rendered depth with monocular pseudo-ground truth despite inconsistent scale and shift.A global Pearson loss is also applied to prevent discontinuities at patch boundaries.
  • Unseen Viewpoint Regularization: SDS regularizes renders from distant viewpoints, reducing high-frequency geometry and texture artifacts while preserving scene structure.Depth warping reprojects training images to nearby viewpoints, and stable warped regions depend on reasonable convergence of the Pearson depth loss.
  • Floater Pruning: Because softmax depth is only a soft constraint, SparseGS prunes Gaussians at false modes after training to remove remaining floaters.False-mode pruning addresses cases where mode-selected and alpha-blended depths do not align.

4. Experiments

Experiments evaluate SparseGS on unbounded and forward-facing datasets under sparse-view settings, including quantitative comparisons, ablations, and qualitative analyses of reconstruction quality.

  • Experimental Settings: Mip-NeRF360 experiments use seven 360° scenes with 12 or 24 training views, while LLFF and DTU evaluate forward-facing scenes.Mip-NeRF360 testing uses every eighth image; inputs are downscaled to one quarter resolution.
  • Comparison: SparseGS significantly outperforms previous NeRF-based methods and concurrent FSGS and DNGaussian methods on Mip-NeRF360 in both 12-view and 24-view settings.Evaluation reports PSNR, LPIPS, and SSIM, including tests with limited COLMAP and sparser SfM initializations.
  • Comparison: SparseGS remains competitive against previous and concurrent state-of-the-art methods on both LLFF and DTU forward-facing datasets.Metric discrepancies can arise because PSNR penalizes sharp artifacts while LPIPS emphasizes perceptual similarity.
  • Ablation Studies: Softmax-scaling depth improves PSNR by 1.37dB compared to 3DGS and significantly enhances rendered depth-map quality.The ablation is conducted on Mip-NeRF360 under the 12-view setting.
  • Ablation Studies: Unseen Viewpoint Regularization improves PSNR by an average 0.33dB, while floater pruning further boosts PSNR by 0.28dB.The modules reduce high-frequency artifacts and remove misplaced low-opacity Gaussians, respectively.
  • Qualitative Evaluation: Qualitative evaluations show sharper high-frequency geometry and cleaner scene structure, while competing methods exhibit background failures or foreground over-pruning.SparseGS preserves sharp subject-background boundaries and produces less noisy reconstructions.

5. Conclusion

SparseGS targets sparse-view 3D reconstruction by combining depth-based controls, unseen-view regularization, and floater removal. The method produces cleaner, more coherent scenes under challenging sparse-input conditions.

  • Conclusion: SparseGS uses softmax-scaling and mode-selection depth rendering to reduce misplaced Gaussians and combines SDS and depth warping to reduce geometry collapse and texture noise.A floater-pruning process identifies and removes low-opacity floaters while preserving fine details.
  • Conclusion: Under the challenging 12-view Mip-NeRF360 setting, SparseGS outperforms state-of-the-art methods with cleaner and more coherent scene outputs.The conclusion emphasizes sparse-view reconstruction quality rather than a single reported metric.

Supplementary Contents

The supplementary material provides derivations, view-sampling details, and algorithmic documentation for the method's depth, regularization, and pruning components.

  • Supplementary Contents: Section F derives the softmax-scaling depth gradients, Section G details UVR view sampling, and Section H describes Advance Floater Pruning.These sections cover mathematical, sampling, and algorithmic supplementary material.

F. Derivation of the Softmax Depth Gradient

The supplementary derivation develops gradients for softmax depth rendering in the rasterizer and uses accumulators to keep computation efficient.

  • Derivation Setup: The derivation considers Gaussians ordered from nearest to farthest along a camera ray and defines transmittance-weighted contributions for a single pixel.The setup introduces α1 as the closest Gaussian and αN as the farthest.
  • Gradient Derivation: Separate derivative expressions are provided for each Gaussian's camera depth and alpha values.These derivatives support backpropagation through the softmax-depth rasterizer.
  • Efficient Backpropagation: Forward-pass denominator sums are transferred to the backward pass, while an accumulator strategy avoids an O(n^2) computational blowup.The implementation follows the accumulator approach used in prior work.

G. View Sampling for UVR

UVR samples both distant and nearby novel viewpoints: elliptical-cylinder views support SDS, while small rotations of training cameras support depth warping.

  • Elliptic Cylinder Sampled Views: SDS samples cameras on an ellipse fitted to the training-view positions, with random z-coordinates within the input views’ 10th–90th percentile bounds.Sampled cameras look toward the scene center and align their up vectors with the average training-view up vector.
  • Warped Views: Depth warping rotates each training camera around the average up-vector by a predefined angle to create nearby pseudo-training viewpoints.These warped views retain substantial overlap with the original training images.
  • Warped Views: The Rodrigues rotation formula generates four warped viewpoints per training camera at -3°, -1.5°, 1.5°, and 3°.The rotation uses the estimated scene-center up-axis.

H. Advance Floater Pruning Algorithm

The floater-pruning algorithm compares mode-selection and alpha-blending depths, uses scene-wide unimodality statistics to set adaptive thresholds, and removes Gaussians identified by the resulting masks.

  • Depth-Based Masking: Floater masks are based on the relative difference between mode-selection depth and alpha-blending depth for each training view.Large discrepancies indicate Gaussians incorrectly placed too close to the camera.
  • Adaptive Thresholding: Bimodal depth-difference histograms indicate floaters, so a dip test measures unimodality and the scores are averaged across training views.The mean score determines the percentile threshold for each view.
  • Adaptive Thresholding: The adaptive threshold uses an exponential curve with manually estimated parameters a = 0.97 and b = −7.5.These parameters were estimated from multiple scenes, datasets, and real-world captures.
  • Pruning Procedure: For each training camera, the algorithm renders both depth types, computes the dip statistic, applies the percentile threshold, masks high-difference pixels, and removes mapped Gaussians.The scene-wide dip score is averaged before per-view thresholds are computed.

I. More Qualitative Comparisons with Few-shot Methods [40, 49]

At 12 input views on Mip-NeRF360, SparseGS preserves sharp details and fills uncovered regions more accurately than SparseNeRF, while FSGS retains foreground detail but can produce floaters and incoherent backgrounds.

  • Comparison with SparseNeRF: With 12 input views, SparseNeRF often over-smooths regions lacking training-view coverage, whereas SparseGS preserves sharp details and fills missing regions more accurately.The comparison is qualitative on the Mip-NeRF360 dataset.
  • Comparison with FSGS: FSGS maintains foreground structure and details but can lack background coherence and produce floaters.This comparison is also reported for 12-view Mip-NeRF360 experiments.
Loading 2312.00206v4…