PaperlayerView exact revision on arXiv

Source-linked AI summary

NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild

Weining Ren, Zihan Zhu, Boyang Sun, Jiaqi Chen, Marc Pollefeys, Songyou Peng

arXiv:2405.18715v2cs.CV

TL;DR

NeRFs assume static scenes, while real-world captures can contain distractors and existing methods struggle in complex or high-occlusion settings. NeRF On-the-go predicts uncertainty from DINOv2 features and incorporates it into NeRF training; it handles varied distractor levels and trains up to one order of magnitude faster than RobustNeRF.

  • Problem

    NeRFs assume static scenes, but real-world captures contain distractors, and existing methods struggle in complex in-the-wild scenes or with sparse image inputs.

    Source: §1 · pp. 1–2

  • Method

    NeRF On-the-go predicts per-ray uncertainty from DINOv2 features and uses structural similarity loss and decoupled training to guide distractor removal.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    NeRF On-the-go handles varied distractor levels across scenes and trains up to one order of magnitude faster than RobustNeRF.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The plug-and-play module is designed for rapid NeRF training from casually captured images across indoor and large-scale street scenes.

    Source: §1 · p. 2

    The method struggles to predict correct uncertainty in regions with strong view-dependent effects, such as reflective windows and metals; under ℓ2 loss, similar-colored distractors can receive low uncertainty.

    Source: §3.2 · p. 4, §5 · p. 8

Abstract

from arXiv · show

Neural Radiance Fields (NeRFs) have shown remarkable success in synthesizing photorealistic views from multi-view images of static scenes, but face challenges in dynamic, real-world environments with distractors like moving objects, shadows, and lighting changes. Existing methods manage controlled environments and low occlusion ratios but fall short in render quality, especially under high occlusion scenarios. In this paper, we introduce NeRF On-the-go, a simple yet effective approach that enables the robust synthesis of novel views in complex, in-the-wild scenes from only casually captured image sequences. Delving into uncertainty, our method not only efficiently eliminates distractors, even when they are predominant in captures, but also achieves a notably faster convergence speed. Through comprehensive experiments on various scenes, our method demonstrates a significant improvement over state-of-the-art techniques. This advancement opens new avenues for NeRF in diverse and dynamic real-world applications.

1. Introduction

NeRFs can render photorealistic, geometrically consistent views, but their static-scene assumption and the burden of removing distractors limit practical use in dynamic real-world captures. NeRF On-the-go predicts uncertainty from DINOv2 features and uses it in NeRF training to remove distractors, including in high-occlusion scenes.

1. Introduction

  • Vanilla NeRF assumes a static scene, while removing distractors through per-pixel annotation is labor-intensive, especially for long captures of large scenes.Training also requires RGB images with camera poses and manual camera-setting adjustments.Source: §1 · p. 1
  • Prior methods leave a gap: RobustNeRF performs poorly in complex in-the-wild scenes and even scenes without distractors, motivating the question of handling any distractor ratio.Other cited methods use segmentation for specific moving-object classes, which can fail on undefined classes.Source: §1 · pp. 1–2
  • NeRF On-the-go handles varied distractor levels across scenes and trains up to one order of magnitude faster than RobustNeRF.The paper describes its approach as a plug-and-play module for casually captured images.Source: §1 · p. 2
  • The method predicts per-sample pixel uncertainty from DINOv2 features, uses structural similarity loss to distinguish distractors from static background, and decouples uncertainty and NeRF training.The paper says this design particularly enhances distractor elimination in high-occlusion scenes.Source: §1 · p. 2

2. Related Work

Related work uses uncertainty in scene reconstruction and addresses dynamic content through segmentation, motion, geometry, or scene decomposition. NeRF On-the-go instead uses DINOv2 features as a prior for distractor removal in casually captured image sequences.

2. Related Work

  • Scene-reconstruction uncertainty may be epistemic, arising from data limitations such as restricted viewpoints, or aleatoric, arising from noise, appearance changes, and distractors.Prior work applies uncertainty to tasks including NeRF exploration, floater removal, active learning, and indoor reconstruction.Source: §2 · p. 2
  • NeRF-W [27] handles transient objects and variable illumination with transient and appearance embeddings; Ha-NeRF [5] hallucinates NeRFs from tourism images, while Neural Scene Chronology [25] reconstructs temporal variation.Kim et al. [21] use DINO-based uncertainty for pose-free NeRF-W [27], whereas this paper targets distractor removal in casually captured sequences.Source: §2 · p. 2
  • Dynamic-scene SLAM and SfM use robust kernels [8] [32], RANSAC, or added segmentation, flow, and geometric information, but handcrafted features struggle at high occlusion.The passage identifies external modules for predefined classes [62] [63] [64], optical or scene flow [45] [47] [61] [66], and geometry-based approaches [3] [63].Source: §2 · p. 2
  • Video-focused dynamic NeRF methods can underperform on photo collections, while RobustNeRF targets static reconstruction from dynamic scenes and struggles with complex, varied occlusions.EmerNeRF [58] uses DINOv2 features for scene decomposition; NeRF On-the-go uses them as prior knowledge for distractor removal.Source: §2 · pp. 2–3

3. Method

NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features, uses a modified SSIM-based loss to distinguish distractors, and decouples uncertainty and NeRF optimization. Dilated patch sampling increases context per iteration; the paper reports faster training and better distractor removal.

3. Method

  • The method’s pipeline combines DINOv2-based uncertainty prediction, uncertainty learning for distractor removal, decoupled optimization, and ray sampling.The paper presents these components as the stages of its method.Source: §3 · p. 3

3.1. Uncertainty Prediction with DINOv2 Features

  • DINOv2 features feed a shallow MLP that predicts per-ray uncertainty, with feature-similar rays regularized to encourage consistent predictions.The neighbor set is formed from rays in the same minibatch whose feature vectors exceed a cosine-similarity threshold.Source: §3.1 · p. 3

3.2. Uncertainty for Distractor Removal in NeRF

  • Because ℓ2 RGB error can assign low uncertainty when distractors resemble the background, the method uses SSIM’s luminance, contrast, and structure cues to distinguish them.The paper reports that SSIM can identify distractors where RGB error alone is challenging.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Optimization

  • The uncertainty objective is applied to the prediction MLP, while NeRF training uses a separate loss with gradient flow detached between the two models.The authors state this decoupling is intended to make uncertainty robust to different distractors.Source: §3.2 · p. 4, §3.3 · pp. 4–5

3.4. Dilated Patch Sampling

  • Dilated patch sampling accelerates training and improves distractor removal compared with other sampling strategies.The method samples rays from an enlarged, dilated patch to increase contextual information in each training iteration.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate NeRF On-the-go on casually captured scenes with varied distractor levels, test its components, and examine convergence, static-scene performance, and large obstructions.

4. Experiments

  • The On-the-go evaluation uses 12 casually captured indoor and outdoor sequences, with distractor ratios from 5% to over 30%.Six sequences representing different occlusion rates are selected for quantitative evaluation.Source: §4 · p. 5

4.1. Evaluation

  • Across scenes with varied distractor ratios, NeRF On-the-go outperforms baselines on the On-the-go dataset.The evaluation includes confined indoor scenes and large outdoor scenes; Table 1 compares novel-view-synthesis metrics across methods.Source: §4 · p. 5, §4.1 · p. 5, Figure 6 · p. 6

4.2. Ablation Study

  • On the highly occluded Patio-High scene, dilation rates from 1 to 4 improve convergence and rendering quality, while higher rates can collapse uncertainty optimization.The authors attribute the collapse to patches losing semantic information and becoming more like random samples.Source: §4.2 · p. 7, Table 3 · p. 8
  • The loss ablations find that SSIM distinguishes distractors from static elements better than ℓ2, joint uncertainty-and-NeRF training drops performance, and omitting Lreg harms some views.These ablations are conducted on Patio-High.Source: §4.2 · p. 7, Table 4 · p. 8

4.3. Analysis

  • At 25K iterations, NeRF On-the-go already captures fine details that RobustNeRF shows at 250K iterations.The comparison attributes faster convergence to the uncertainty-prediction pipeline and dilated patch sampling.Source: §4.3 · pp. 7–8, Figure 7 · p. 8
  • On a static Mip-NeRF 360 scene, the method performs on par with the SOTA method [1] and better than RobustNeRF; it also models large obstructions using predicted uncertainty.RobustNeRF misses parts of the bicycle, while Figure 9 presents input frames, uncertainty maps, and renderings for large obstructions.Source: Figure 8 · p. 8, §4.3 · p. 8, Figure 9 · p. 8

5. Conclusions

NeRF On-the-go removes distractors in dynamic real-world scenes with varied distractor levels. The method can mispredict uncertainty on strongly view-dependent regions, including reflective windows and metals.

5. Conclusions

  • NeRF On-the-go enables effective and efficient distractor removal in dynamic real-world scenes with varied distractor levels.The conclusion frames this as a step toward practical, in-the-wild NeRF applications.Source: §5 · p. 8
  • Uncertainty predictions can be incorrect in regions with strong view-dependent effects, such as reflective windows and metals.The authors suggest that integrating additional prior knowledge into optimization could help.Source: §5 · p. 8

Supplementary Material

The supplementary document adds method details, experimental results, ablation studies, performance analysis, and a supplementary video with visual comparisons.

Supplementary Material

  • The supplement provides additional method details, experiments, ablations, performance analysis, and a video of visual comparisons.Source: Supplementary Material · p. 12

A.1. Dataset Details

The On-the-go dataset contains 12 casually captured sequences from indoor and outdoor scenes, with varied distractors and occlusion ratios. The supplement also describes the synthetic and RobustNeRF datasets used in evaluation.

A.1. Dataset Details

  • On-the-go uses iPhone 12, Samsung Galaxy S22, and DJI Mini 3 Pro captures, fixing exposure, white balance, and ISO per sequence; its dynamic objects span diverse occlusion ratios from 5% to 30%.Source: §A.1 · p. 12
  • The synthetic dataset has five Kubric-generated sequences, but the Chair scene is excluded because its training and test images have misaligned coordinate systems.Source: §A.1 · p. 12
  • The RobustNeRF dataset includes unintentional changes such as tablecloth movement in Android and curtain movement in Statue, which may adversely affect SAM-based methods.The passage says RobustNeRF and NeRF On-the-go can naturally accommodate these changes.Source: §A.1 · p. 12

A.2. Implementation Details of NeRF On-the-go

The section records the training setup and how baseline comparisons were prepared.

A.2. Implementation Details of NeRF On-the-go

  • Each scene was trained for 250,000 iterations, typically taking 12 hours.The reported setup used a batch size of 16,384.Source: §A.2 · p. 12
  • The method was already reported to outperform RobustNeRF after one hour of training, compared with RobustNeRF’s 12-hour run.Images were downsampled by 8×, except Arc de Triomphe and Patio, which used 4× downsampling.Source: §A.2 · p. 12

A.3. Baseline Details

  • The Mip-NeRF 360 + SAM baseline used clicked distractors or pre-identified categories, then YOLOv8 boxes and SAM masks.The Spot robot was selected by its largest yellow bounding box because YOLOv8 lacks a robot class; masking errors included missed objects and partial segments.Source: §A.3 · p. 12

B.1. Evaluation

The evaluation reports results across synthetic, benchmark, and captured in-the-wild scenes, with comparisons against several baselines.

Kubric Dataset

  • On Kubric, performance aligns with RobustNeRF, which the authors attribute to saturation on this simple dataset.The evaluation is included for completeness.Source: §B.1 · pp. 12–13

On-the-go Dataset

  • Even with known dynamic-object categories, Mip-NeRF 360 + SAM produces incorrect detections, overlooks objects, or segments them incompletely.The cited causes include partial observation, reflections, and ambiguous classifications.Source: §B.1 · p. 13

Comparison on RobustNeRF Dataset

  • On BabyYoda, the method beats the authors’ RobustNeRF run but does not reach the performance reported in the original RobustNeRF paper.The sequence is supplemental because distractors vary across images, unlike the paper’s setting.Source: §B.1 · p. 13

On-the-go Dataset

  • On the On-the-go dataset, the method consistently outperforms all baselines across varied environments.NeRF-W loses detail, RobustNeRF can miss thin structures, and Mip-NeRF 360 + SAM is affected by imperfect segmentation.Source: §B.1 · p. 13

B.2. Ablation Study

The ablations examine loss components, patch dilation, and feature extraction, finding that each design choice affects distractor removal or reconstruction quality.

Loss Ablation

  • Each loss component contributes to performance in both low- and high-occlusion scenes, while omitting the uncertainty loss causes a marked decline.The decline is attributed primarily to the SSIM formulation being tailored to uncertainty rather than scene representation.Source: §B.2 · p. 13, Table C · p. 13

Dilated Patch Ablation

Feature Extraction Module

  • DINOv1 and DINOv2 produce negligible differences, while ResNet-50 is less effective at removing distractors.The authors attribute the difference to ResNet’s emphasis on color and DINO’s emphasis on instance information.Source: §B.2 · p. 15

B.3. Analysis

The analysis reports faster convergence and examines the modified SSIM formulation, parameter tuning, and failure conditions. It finds that the method remains limited by strong view-dependent effects and sparse training views.

Our SSIM Formulation

  • The modified SSIM formulation removes distractors in Patio-High, while conventional SSIM fails to do so.The formulation is analyzed as placing greater emphasis on similarity differences between distractors and static backgrounds under the stated patch-similarity assumptions.Source: §B.3 · pp. 15–16

Parameter-tuning Free

  • The method needs no explicit outlier-ratio assignment, while RobustNeRF requires experiments with different ratios to find its best performance.The comparison is reported for Patio-High.Source: §B.3 · p. 16, Figure D · p. 16

Fast Convergence

  • Nearly one magnitude faster convergence is reported, with better performance after convergence across Tree and Patio-High occlusion conditions.The comparison uses SSIM metrics.Source: §B.3 · p. 16

Failure Case

  • Strong view-dependent effects remain difficult, and performance degrades significantly when training views are sparse.The sparse-view limitation is inherited from Mip-NeRF 360.Source: §B.3 · p. 16
Loading 2405.18715v2…