PaperlayerView exact revision on arXiv

Source-linked AI summary

SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting

Sara Sabour, Lily Goli, George Kopanas, Mark Matthews, Dmitry Lagun, Leonidas Guibas, Alec Jacobson, David J. Fleet, Andrea Tagliasacchi

arXiv:2406.20055v2cs.CVcs.LG

TL;DR

Real-world captures often violate the static-scene and consistent-imaging assumptions used by 3DGS, complicating reconstruction. SpotLessSplats uses semantic features with robust optimization to detect transient distractors; the paper reports state-of-the-art reconstruction on casual captures and similar quality with fewer splats.

  • Problem

    Common reconstruction assumptions of static scenes and consistent imaging are violated by real-world moving objects and lighting variations, which can degrade reconstruction.

    Source: §1 · p. 1

  • Method

    SpotLessSplats detects distractors without supervision using semantic features and robust optimization, with clustering or a lightweight MLP for feature-space outlier grouping.

    Source: §1 · p. 2, §4 · p. 4, §6 · p. 9

  • Results

    SpotLessSplats is reported to achieve state-of-the-art robust reconstruction and outperform existing methods on casual captures.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Utilization-based pruning can preserve rendering quality while reducing the number of Gaussians by 2× to 4.5× on clean MipNeRF360 datasets.

    Source: §5.2 · p. 8

    Low-resolution diffusion features can miss thin structures, and aggressive pruning can remove scene parts rarely viewed in training data.

    Source: §6 · p. 9

Abstract

from arXiv · show

3D Gaussian Splatting (3DGS) is a promising technique for 3D reconstruction, offering efficient training and rendering speeds, making it suitable for real-time applications.However, current methods require highly controlled environments (no moving people or wind-blown elements, and consistent lighting) to meet the inter-view consistency assumption of 3DGS. This makes reconstruction of real-world captures problematic. We present SpotLessSplats, an approach that leverages pre-trained and general-purpose features coupled with robust optimization to effectively ignore transient distractors. Our method achieves state-of-the-art reconstruction quality both visually and quantitatively, on casual captures. Additional results available at: https://spotlesssplats.github.io

1 Introduction

Real-world captures violate the static-scene and consistent-imaging assumptions common in NeRF and 3DGS, while residual-based robustness can be ill-suited to 3DGS training. SpotLessSplats uses learned semantic features and adaptive outlier detection, reporting stronger reconstruction on casual-capture benchmarks.

1 Introduction

  • Real-world moving objects, lighting variations, and other photometric inconsistencies violate assumptions of simultaneous, well-posed, noise-free capture and can degrade reconstruction.These conditions limit the applicability of methods that rely on controlled captures.Source: §1 · p. 1
  • Residual-based outlier methods have been adapted from NeRF to 3DGS, but 3DGS densification introduces residual variance that can compromise transient detection.The paper contrasts Phototourism's longer-term appearance changes with the shorter transients common in casual captures.Source: §1 · pp. 1–2
  • SpotLessSplats uses text-to-image feature space and adaptive outlier detection, with clustering or an unsupervised MLP, to identify structured distractors in 3DGS training images.The paper also proposes sparsification that yields similar reconstruction quality with two to four times fewer splats.Source: §1 · p. 2
  • The method is reported to consistently outperform competing methods in reconstruction accuracy on challenging casual-capture benchmarks.The contributions describe this as state-of-the-art robust reconstruction and a substantial margin over existing methods.Source: §1 · p. 2

2 Related work

Prior NeRF and 3DGS approaches address robustness through appearance modeling or photometric residual masks, but RGB-based masks can confuse distractors with similarly colored backgrounds. SpotLessSplats instead uses semantic features for casual captures with transient occlusions and limited appearance change.

2 Related work

  • 3DGS offers faster rendering than NeRF while maintaining high quality, and its rasterized Gaussians are alpha-blended for image formation.The representation uses GPU-friendly rasterization to render an entire image in one forward pass.Source: §2 · p. 2
  • NeRF-W models uncertainty masks and global appearance, while later NeRF and 3DGS methods use learned appearance representations or predicted masks to handle in-the-wild captures.These approaches include learned global or local appearance embeddings and 2D masks that downweight high-error pixels.Source: §2 · pp. 2–3
  • RobustNeRF and NeRF-W-style masks use RGB residuals and can misclassify transients whose colors resemble the background, while requiring careful hyperparameter tuning.The paper illustrates the color-confusion case with an adapted RobustNeRF mask that labels similar-color balloon pixels as inliers.Source: §2 · p. 2, Figure 2 · p. 2
  • SLS targets casual captures with longer-duration transients and minimal appearance changes, unlike Phototourism captures with changes across weeks and times of day.NeRF On-the-go uses semantic features with an MLP but also supervises masks using structural rendering error, which can lead to over- or under-masking.Source: §2 · pp. 2–3

3 Background

3DGS represents a scene with anisotropic Gaussian splats and renders them through screen-space rasterization and alpha blending. For casual captures, robust optimization predicts image masks and uses a masked L1 loss, though directly transferring robust NeRF masking to 3DGS is ineffective without adaptations.

3 Background

  • 3DGS represents a scene as anisotropic Gaussian splats with means, covariances, opacities, and view-dependent colors, then rasterizes and alpha-blends them for rendering.The projected covariance is approximated using the projection Jacobian, and splats are sorted along the view direction for alpha blending.Source: §3 · p. 3
  • SpotLessSplats is shown reconstructing NeRF On-the-go scenes with different transient-occlusion levels while avoiding transient leakage and under-reconstruction.The figure caption describes qualitative and quantitative results on the dataset.Source: Figure 3 · p. 3

3.1 Robust optimization of 3DGS

  • The framework assumes casually captured images may include transient effects and does not assume a particular transient object class, appearance, or shape.Distractors are image portions that break the assumptions of a perfectly static and 3D-consistent world.Source: §3.1 · p. 3
  • Robust optimization predicts unsupervised inlier/outlier masks for each training image and optimizes 3DGS with a masked L1 loss based on rendered-image residuals.The referenced RobustNeRF mask thresholds photometric errors and uses a spatial dilation filter, but directly applying such ideas to 3DGS does not remove outliers effectively without adaptations.Source: §3.1 · p. 4

4 Method

SpotLessSplats detects distractors through semantic feature similarities and adapts robust optimization to 3DGS training. Its method combines spatial or spatio-temporal clustering with changes to masking, residual tracking, pruning, and appearance modeling.

4 Method

  • SLS identifies distractors through semantic feature similarities rather than relying solely on photometric errors.The method seeks feature subspaces likely to cause large photometric errors and removes those pixels from optimization.Source: §4 · p. 4

4.1.1 Spatial clustering.

  • Spatial clustering over-segments feature maps into 100 connected regions, greedily merging regions to reduce inter-cluster feature variance.The resulting cluster labels are propagated to pixels and used as the training mask.Source: §4.1.1 · p. 4

4.1.2 Spatio-temporal clustering.

  • The spatio-temporal alternative trains an MLP to predict pixel inlier probabilities from features, using residual-derived labels where status is confident and semantic similarity otherwise.The classifier is co-trained with 3DGS; lower and upper residual labels provide weak supervision.Source: §4.1.2 · pp. 4–5, Figure 4 · p. 5

4.2 Adapting 3DGS to robust optimization

  • Direct robust masking can overfit a premature 3DGS model, skew inlier estimates across image-based batches, or interact with densification.The method addresses these issues with scheduled sampling, multi-batch residual tracking, and utilization-based pruning.Source: §4.2 · p. 5, §4.2.1 · p. 5, §4.2.2 · p. 5

4.2.1 Warm up with scheduled sampling.

  • Scheduled sampling gradually applies masks because early residuals and the not-yet-converged classifier can produce random masks.Sparse gradient sampling in uncertain areas supports outlier classification.Source: §4.2.1 · p. 5

4.2.2 Trimmed estimators in image-based training.

  • A discounted histogram of residual magnitudes across batches estimates the generalized median despite image-to-image variation in outlier proportions.Residuals are binned with width 10^-3, and the generalized median is extracted as a tau quantile.Source: §4.2.2 · p. 5

4.2.3 A friendly alternative to “opacity reset”.

  • Utilization-based pruning replaces opacity reset, avoiding its interference with residual tracking while compressing scenes and retaining comparable reconstruction quality in outlier-free scenes.Utilization is based on rendered-color gradients with respect to projected splat positions; pruning uses a threshold of 10^-8.Source: §4.2.3 · p. 5
  • SLS outperforms baseline methods on 3DGS and NeRF on RobustNeRF datasets, preventing over- or under-masking.The figure also includes quantitative and qualitative evaluations.Source: Figure 5 · p. 6

5 Results

SpotLessSplats outperforms robust NeRF baselines on the evaluated distractor-filled captures and improves substantially over vanilla 3DGS. Its pruning and design adaptations reduce Gaussian counts while preserving cleaner reconstructions.

5.1 Distractor-free 3D reconstruction

  • On RobustNeRF, SLS-mlp outperforms all robust NeRF-based baselines; on NeRF On-the-go, the authors’ method has superior quantitative results to SOTA robust NeRF methods and significantly improves over vanilla 3DGS.On RobustNeRF, results are closer to ideal clean-model performance, particularly on Yoda and Android. Figure 6 shows SLS is free of high-occlusion lingering-distractor leaks modeled as noisy floaters in baselines.Source: §5.1 · pp. 7–8, Figure 6 · p. 7

5 Results

  • Evaluation covers casual captures in RobustNeRF and NeRF On-the-go, with clean test sets; comparisons include vanilla 3DGS and robust NeRF methods.RobustNeRF includes distractor-filled and distractor-free training splits, while NeRF On-the-go has low-, medium-, and high-occlusion scenes.Source: §5 · pp. 6–7

5.1 Distractor-free 3D reconstruction

  • NeRF-HuGS over-masks static content across all four RobustNeRF scenes or under-masks transients in challenging sparsely sampled views.The comparison also reports vanilla 3DGS modeling distractors as floaters, view-dependent effects, or both.Source: §5.1 · pp. 7–8

5.2 Effect of utilization-based pruning

  • UBP reduces Gaussian counts 4×–6× with at least 2× less training time and 3× less inference time; on MipNeRF360 it reduces counts 2×–4.5× while preserving quality.UBP can slightly degrade quantitative measurements on cluttered scenes, although final renderings are cleaner with fewer floaters.Source: Figure 7 · p. 7, §5.2 · p. 8

5.3 Ablation study

  • UBP removes leaked distractors, while GLO, warm-up scheduling, and Bernoulli sampling improve results; disabling UBP may raise metrics but leaks transients.The ablation reports that replacing UBP with the original 3DGS opacity reset is ineffective.Source: Figure 10 · p. 9, §5.3 · p. 9
  • An MLP handles subtle transients better than a similarly sized CNN, regularizer weight has little effect, and agglomerative-clustering gains diminish at 100 clusters.The comparison uses MLP and CNN variants of roughly 30K parameters; the reported cluster plateau is for C≥100.Source: Figure 9 · p. 8, §5.3 · p. 9

6 Conclusion

SpotLessSplats uses semantic features for unsupervised distractor masking and adds gradient-based pruning to reduce splat counts. Its feature and pruning choices have limitations for nearby same-class instances, thin structures, and rarely viewed regions.

6 Conclusion

  • SLS uses semantic features for unsupervised distractor masking through fast spatial clustering or a lightweight MLP trained alongside 3DGS.SLS-agg assigns inlier-outlier labels to clusters; SLS-mlp provides higher-precision grouping while being marginally slower.Source: §6 · p. 9
  • Gradient-based pruning reduces splat counts while retaining vanilla 3DGS reconstruction quality, and is compatible with distractor suppression.Source: §6 · p. 9
  • Nearby distractors and non-distractors from the same semantic class may not be distinguished, and low-resolution features can miss thin structures.Upsampling features for clustering can also produce imprecise edges.Source: §6 · p. 9
  • Aggressive uncertainty thresholds in pruning can remove scene parts that are rarely viewed in training data.Source: §6 · p. 9

A Distinguishing between semantically similar instances

The method can distinguish same-class distractor and non-distractor instances when they are separated in image space, but close proximity can lead to over-masking.

A Distinguishing between semantically similar instances

B Warm-up scheduler

The warm-up uses a staircase exponential scheduler whose parameters control decay speed and step length; the steepest-decay scenes use a different β1 value.

B Warm-up scheduler

C Additional results on the Crab dataset

Crab (2) adds test viewpoints beyond those used for training, and SLS-MLP performs close to an ideal model trained on clean data while outperforming vanilla 3DGS.

C Additional results on the Crab dataset

D Additional Results on NeRF On-the-go dataset

Qualitative results on six additional NeRF On-the-go scenes show SpotLessSplats handling glass, color-matched distractors, and some distractor shadows. The results also indicate that shadow robustness is incomplete.

D Additional Results on NeRF On-the-go dataset

Related papers

Loading 2406.20055v2…