Source-linked AI summary
RobustNeRF: Ignoring Distractors with Robust Losses
Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David J. Fleet, Andrea Tagliasacchi
TL;DR
Distractors in NeRF captures can be hard to distinguish from view-dependent effects and can degrade reconstructions. RobustNeRF models them as outliers during optimization using a trimmed least-squares approach with spatially smoothed outlier labels. It reports qualitative and quantitative improvements over recent methods across synthetic, benchmark, and robot-captured datasets.
Problem
Method
RobustNeRF treats distractors as optimization outliers using iteratively re-weighted least squares, a trimmed least-squares variant, and spatial smoothing of inlier/outlier labels.
Source: §3.3 · p. 5, §5 · p. 8Results
Takeaways & Limitations
The RobustNeRF loss can be incorporated within other NeRF models, and the method requires minimal-to-no hyperparameter tuning.
Source: §1 · p. 2, §5 · p. 8Robust optimization does not reliably separate high-frequency details from outliers, and strongly robust kernels can slow fine-detail learning and reduce visual fidelity.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Neural radiance fields (NeRF) excel at synthesizing new views given multi-view, calibrated images of a static scene. When scenes include distractors, which are not persistent during image capture (moving objects, lighting variations, shadows), artifacts appear as view-dependent effects or 'floaters'. To cope with distractors, we advocate a form of robust estimation for NeRF training, modeling distractors in training data as outliers of an optimization problem. Our method successfully removes outliers from a scene and improves upon our baselines, on synthetic and real-world scenes. Our technique is simple to incorporate in modern NeRF frameworks, with few hyper-parameters. It does not assume a priori knowledge of the types of distractors, and is instead focused on the optimization problem rather than pre-processing or modeling transient objects. More results on our page https://robustnerf.github.io.
1. Introduction
NeRF reconstructs scenes from calibrated images but assumes photometric consistency, an assumption distractors can violate and make difficult to distinguish from view-dependent effects. The paper treats distractors as optimization outliers and reports a simpler, more effective approach evaluated on synthetic, public, and robot-captured scenes.
1. Introduction
- NeRF stores a 3D representation in neural-network weights and uses view-dependent radiance with volumetric rendering to synthesize photorealistic views.Understanding static 3D structure from images has applications in AR/VR, robotics, and photogrammetry.Source: §1 · p. 1
- When photometric consistency is violated, NeRF can reconstruct inconsistent content as floaters; the proposed technique automatically ignores distractors without explicit supervision.Figure 1 illustrates the inconsistency and clean reconstruction claim.Source: Figure 1 · p. 1
- NeRF training relies on photometric consistency: images from the same viewpoint should be identical up to noise, with camera settings held fixed unless a method accounts for variation.Training generally uses many accurately calibrated images, with calibration often recovered through structure-from-motion.Source: §1 · p. 1
- Distractors are non-persistent scene elements, from operator-cast shadows to passing pets or children, and can significantly degrade reconstruction when ignored.Manual removal requires pixel-by-pixel labeling, while distractor types are not known in advance.Source: §1 · p. 1
- Because repeated captures from the same viewpoint are difficult, it is hard to distinguish distractors from the view-dependent effects that give NeRF realistic appearance.The paper frames this distinction as a central challenge in handling distractors.Source: §1 · p. 1
- The method analyzes NeRF distractor handling through robust estimation and reports simpler implementation, minimal-to-no hyperparameter tuning, and state-of-the-art performance.Evaluations include synthetic rendered data, public datasets, and new natural and synthetic scenes, including robot-acquired scenes.Source: §1 · p. 2
2. Related Work
This section reviews NeRF representations and training, extensions to speed, scene coverage, and other image artifacts, and methods for representing dynamic scenes. It also describes prior approaches to removing non-static elements and the ambiguity by which distractors can appear as view-dependent effects.
Neural Radiance Fields
- A NeRF is a continuous volumetric scene representation stored in neural-network parameters, mapping position and view direction to view-dependent RGB color and view-independent density.The field is trained from images paired with camera calibration; pixels become rays, and rendered colors are fitted with an L2 photometric reconstruction loss.Source: §2 · p. 2
Recent progress on NeRF models
- NeRF research has advanced training and inference speed, 360-degree scene modeling, reduced image requirements, image-artifact handling, camera refinement, surface prediction, and additional applications.The review notes training in minutes and real-time mobile rendering in cited work.Source: §2 · p. 2
Modeling non-static scenes
- For unstructured dynamic scenes, approaches include time inputs, per-frame latent codes, and time-conditioned deformation fields, with careful regularization or additional training signals needed for acceptable results.Direct time inputs can produce cloudy geometry and lack fine detail.Source: §2 · p. 2
- Prior distractor-removal approaches include segmenting likely distractor pixels, but segmentation misses secondary effects such as shadows; earlier outlier modeling can leave cloudy geometry.These limitations are described for the approaches cited in the passage.Source: §2 · p. 2
- Figure 2 illustrates that distractors in the first and third camera views can make scene portions non-photo-consistent, which may be encoded as view-dependent effects even with ground-truth geometry.The illustration uses a static object observed by three cameras.Source: Figure 2 · p. 3
3. Method
Nonpersistent observations can be absorbed as view-dependent radiance, while outliers and fine detail can look similar during training. RobustNeRF addresses this with iteratively reweighted trimmed least squares and spatially coherent binary weights.
3.1. Sensitivity to outliers
- View-dependent radiance can memorize transient objects, while least-squares losses remain sensitive to outliers and heavy-tailed noise.Under natural, non-Lambertian conditions, both reflectance effects and outliers can be explained by view-dependent radiance.Source: §3.1 · p. 3
3.2. Robustness to outliers
- An oracle segmentation mask can exclude outlier pixels from the empirical loss, but requires a detector capable of identifying arbitrary distractors.Examples include segmenting people, sky, or non-face pixels.Source: §3.2 · p. 3
- Generic robust kernels can suppress outliers but also lose fine texture, or preserve detail while leaving outlier artifacts.Large residuals can represent either distractors or high-frequency details not yet learned; the authors report this tradeoff across estimators and settings.Source: §3.2 · pp. 3–4, Figure 4 · p. 4
3.3. Robustness via Trimmed Least Squares
- RobustNeRF marks residuals below the median as inliers, then smooths those labels spatially to preserve fine-grained details during optimization.The weight mask uses a 3×3 box kernel with a 0.5 threshold.Source: §3.3 · p. 5
- To reduce early misclassification of localized texture, the method labels 8×8 patches using weight behavior in their 16×16 neighborhoods.The neighborhood-average weight threshold is 0.6.Source: §3.3 · p. 5
- The inlier and outlier labels change with residuals during training and settle toward masks resembling oracle labels.The weight visualizations show this progression without explicit supervision.Source: §3.3 · p. 5, Figure 6 · p. 5
4. Experiments
The experiments compare RobustNeRF with mip-NeRF 360 loss variants and D2NeRF across natural and synthetic scenes, including controlled distractor settings. RobustNeRF improves reconstruction on distractor-containing captures, while ablations and sensitivity tests characterize its trade-offs.
4.3. Evaluation
- The evaluation compares RobustNeRF with L2, L1, and Charbonnier mip-NeRF 360 variants and D2NeRF, using PSNR, SSIM, and LPIPS on held-out distraction-free frames where available.The last row of the natural-scene table uses distractor-free mip-NeRF 360 training as an upper bound; RobustNeRF nearly matches that accuracy when trained on clean frames.Source: §4.1 · p. 6, §4 · p. 6, §4.3 · p. 6
4.1. Baselines
- The baselines include mip-NeRF 360 trained with L2, L1, or Charbonnier losses and D2NeRF, which reconstructs distractors rather than discarding them.NeRF-W is omitted because its performance falls short of D2NeRF.Source: §4.1 · p. 6
4.2. Datasets – Figure 7
- The study combines existing D2NeRF scenes with new natural and synthetic datasets, including unordered captures, varied distractors, and controlled synthetic difficulty levels.The natural scenes include seven captures across street, apartment, and robotics-lab settings; the synthetic scenes vary object count, size, and motion.Source: §4.2 · p. 6
4.3. Evaluation
- On natural scenes evaluated on distractor-free frames, RobustNeRF generally outperforms mip-NeRF 360 variants by 1.3–4.7 dB PSNR; on D2NeRF synthetic scenes, it matches or exceeds D2NeRF by up to 12 dB depending on unique outlier-object count.The natural-scene models were trained on frames with distractors. The mip-NeRF 360 advantage is smaller in Apartment than Robotics Lab scenes.Source: §4.3 · p. 6, Figure 9 · p. 8
- Blind trimming removes distractors but loses high-frequency texture; smoothing recovers fine detail at longer training times, while spatial windows bring training time on par with mip-NeRF 360.Patch-based evaluation adds detail and reaches PSNR 30 almost 4× faster; reducing patch size offsets gains from smoothing and patching, while trimming-threshold changes have little effect.Source: §4.3 · pp. 7–8, Figure 10 · p. 8
- As cluttered training images increase, mip-NeRF 360 drops from 33 to 25 dB, while RobustNeRF stays above 31 dB; on clean data, RobustNeRF mildly underperforms in quality and training time.The proposed explanation for the clean-data trade-off is that trimmed estimation discards some pixels even when they are not distractors.Source: Figure 11 · p. 8, §4.3 · p. 8
5. Conclusions
The paper frames distractor-robust NeRF training as robust estimation and reports strong performance across synthetic, benchmark, and newly captured scenes. Its loss can be used beyond mip-NeRF 360, while statistical inefficiency limits performance on clean data.
5. Conclusions
- RobustNeRF formulates training as iteratively re-weighted least squares with a trimmed-LS variant and a smoothness bias on the outlier process.The authors report qualitative and quantitative gains over recent methods on synthetic, benchmark, and robot-captured datasets.Source: §5 · p. 8
- The robust loss can be incorporated into NeRF models beyond mip-NeRF 360, although the experiments explore it in the mip-NeRF 360 setting.The reported comparisons include state-of-the-art methods on several dataset types.Source: §5 · p. 8
- On clean data, statistical inefficiency can produce poorer reconstructions and often longer training times.The authors identify very small distractors as a future challenge that may require adapting the spatial support used for inlier/outlier decisions.Source: §5 · p. 8
6.1. Dataset Description
The datasets combine natural captures and synthetic scenes to evaluate static-scene reconstruction with distractors. Natural captures include paired cluttered and clean images, while synthetic scenes vary distractor occupancy and difficulty.
6.1. Dataset Description
- The dataset collection includes natural and synthetic scenes, with captures both with and without distractors to support reconstruction of static scene content.Natural scenes include apartment and robotics-lab settings, while synthetic scenes provide controlled distractor conditions.Source: §6.1 · p. 12, §6.1.1 · p. 12, §6.1.2 · p. 13
6.1.1 Natural Scenes
- The apartment datasets differ in distractor behavior: Android uses three small wooden robots in cluttered photos, while Statue uses a balloon that often remains in the same volume across multiple photos.Android comprises 122 cluttered and 10 clean photos; Statue comprises 255 cluttered and 19 clean photos.Source: §6.1.1 · p. 12
- The apartment scenes pose additional reconstruction challenges from partially over- or underexposed, sparsely seen background objects and small unintended background motion.Statue images also exhibit exposure and color-calibration mismatch with RobustNeRF renders, while the table-cloth fold in the Apartment example moved during capture and the background was not thoroughly captured.Source: §6.1.1 · p. 12
- Each natural scene has 109 same-pose cluttered/clean photo pairs and about 200 additional clean photos from unpaired viewpoints for evaluation.Manual object placement means a capture session often takes several hours.Source: §6.1.1 · p. 12
6.1.2 Synthetic Scenes
- The Kubric datasets contain 200 cluttered training images and 100 clean evaluation images, with easy, medium, and hard scenes varying distractor occupancy.The hard scene’s six large distractors occupy roughly half of each image and average 10× the easy scene’s distractor pixels.Source: Figure 13 · p. 13, §6.1.2 · p. 13
6.2. Training Details
The experiments use matched training schedules for RobustNeRF and mip-NeRF 360, while D2NeRF follows its reference setup. Images are downsampled to accommodate D2NeRF, and D2NeRF hyperparameters are selected using a search over real scenes.
6.2. Training Details
- Natural images are downsampled by 8× to accommodate D2NeRF, and evaluation uses held-out images: 10 for Android, 19 for Statue, 194 for Crab, and 202 for BabyYoda.Unless otherwise stated, all available cluttered images are used for training.Source: §6.2 · p. 13
- RobustNeRF replaces mip-NeRF 360’s reconstruction loss in MultiNeRF and trains for 250,000 steps on random 16×16 patches, excluding pixels identified as outliers.Its models use 16 TPUv3 chips for nine hours; mip-NeRF 360 uses the same step count, batch size, learning-rate schedule, architecture, hardware, and duration, but samples 16,384 rays per minibatch.Source: §6.2 · p. 13
- D2NeRF follows the authors’ reference implementation for 100,000 iterations with batch size 1,024, using four V100 GPUs in place of A100s.The authors searched five key hyperparameters across 16 real-world scenes and used the configuration reported as most effective on 10 of 16 scenes.Source: §6.2 · p. 13
- D2NeRF input order follows file-system order, which is not guaranteed to represent a continuous camera path.The section discusses the effects of this random ordering versus a heuristically identified path; the qualitative Apartment example also notes motion and incomplete background capture, and Statue exposure and color-calibration differences.Source: §6.2 · p. 13, Figure 14 · p. 14
6.3. Experiments
Across five experimental subsections, RobustNeRF is compared with mip-NeRF 360 and D2NeRF, tested for hyperparameter sensitivity, and evaluated on scenes with view-dependent effects and diverse distractors.
6.3.1 Comparison to mip-NeRF 360
- Across the four natural scenes, RobustNeRF’s performance gap over mip-NeRF 360 is larger on robotics-lab scenes; clean-scene mip-NeRF 360 exceeds 32 dB PSNR on Crab and BabyYoda but is nearly 10 dB lower on Statue and Android.The authors cite difficult background geometry, sun-overexposure in some test images, and unintended object movement as challenges in the apartment scenes.Source: §6.3.1 · p. 14
6.3.2 Comparison to D2NeRF
- Reordering D2NeRF views produced no consistent performance improvement, while its first recommended hyperparameter configuration performed best in 10 of 16 real-world scenes.The first configuration was also optimal across the Statue and Crab tuning experiments; D2NeRF performance is strongly affected by hyperparameter choices.Source: Figure 15 · p. 14, §6.3.2 · p. 15, Figure 16 · p. 15
6.3.3 Sensitivity to Hyperparameters
- RobustNeRF’s thresholds and filter sizes work across a range of datasets when Tε exceeds the outlier-pixel proportion; larger neighborhoods provide stronger regularization.For the hard synthetic case, where 44% of pixels are occupied on average, thresholds above 50% worsen results; using less than 50% of the loss slows training, and 30% requires twice as many iterations to catch up. Overestimating Tε increases training time without affecting final accuracy.Source: §6.3.3 · p. 15, Figure 17 · p. 15, Figure 18 · p. 15
6.3.4 View-dependent effects
- RobustNeRF performs similarly to mip-NeRF 360 on non-Lambertian materials, semi-transparent objects, and soft shadows.Additional examples include scenes with mirrors, metallic and reflective objects, glass, and moving pedestrians or photographer body parts as distractors.Source: §6.3.4 · pp. 15–16, Figure 19 · p. 16
6.3.5 More Qualitative Results
- On Statue, Android, Crab, and BabyYoda, D2NeRF removes outliers only in the single-distractor Statue scene, whereas RobustNeRF handles distractors irrespective of number or diversity.Figures 21–24 show the four dataset comparisons.Source: §6.3.5 · p. 16, Figure 21 · p. 17, Figure 22 · p. 18
- Across four datasets, RobustNeRF eliminates floaters that mip-NeRF 360’s L1, L2, and Charbonnier losses leave, while D2NeRF removes outliers only with a single distractor.RobustNeRF removes outliers regardless of their number or diversity, and produces plausible, distraction-free models on D2NeRF natural scenes.Source: §6.3.5 · p. 16