PaperlayerView exact revision on arXiv

Source-linked AI summary

Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields

Lily Goli, Cody Reading, Silvia Sellán, Alec Jacobson, Andrea Tagliasacchi

arXiv:2309.03185v1cs.CV

TL;DR

Occlusions and missing views limit the epistemic knowledge NeRFs can acquire, while existing uncertainty methods may be heuristic or computationally demanding. BayesRays applies spatial perturbations and a Bayesian Laplace approximation to pre-trained NeRFs; its reported uncertainties outperform prior work on key metrics and support artifact removal matching or improving the state of the art.

  • Problem

    Occlusions and missing views limit the epistemic knowledge learned from multiview images, and existing NeRF uncertainty methods are heuristic or computationally demanding.

    Source: §1 · p. 1

  • Method

    BayesRays applies spatially parameterized radiance-field perturbations and a Bayesian Laplace approximation to produce an uncertainty field for any pre-trained NeRF without changing training.

    Source: §1 · p. 2

  • Results

    BayesRays reports statistically meaningful uncertainties that outperform prior work on key metrics and supports artifact removal matching or improving the state of the art.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The spatial uncertainty field can be rendered as an additional color channel and thresholded to remove artifacts from pre-trained NeRFs.

    Source: §1 · p. 2, §6 · p. 8

    BayesRays targets epistemic uncertainty and does not capture aleatoric uncertainty caused by noise or inconsistencies between views.

    Source: §6 · p. 8

Abstract

from arXiv · show

Neural Radiance Fields (NeRFs) have shown promise in applications like view synthesis and depth estimation, but learning from multiview images faces inherent uncertainties. Current methods to quantify them are either heuristic or computationally demanding. We introduce BayesRays, a post-hoc framework to evaluate uncertainty in any pre-trained NeRF without modifying the training process. Our method establishes a volumetric uncertainty field using spatial perturbations and a Bayesian Laplace approximation. We derive our algorithm statistically and show its superior performance in key metrics and applications. Additional results available at: https://bayesrays.github.io.

1. Introduction

NeRFs can remain epistemically uncertain because occlusions and missing views limit what multiview training reveals, while existing uncertainty methods are heuristic or computationally demanding. BayesRays uses spatial perturbations and a Bayesian Laplace approximation to quantify uncertainty in pre-trained NeRFs, with reported benefits for uncertainty metrics and artifact removal.

Introduction

  • Occlusions and missing views limit the epistemic knowledge a NeRF can acquire, while existing methods are heuristic or computationally demanding.Uncertainty estimates also matter for outlier detection and next-best-view planning in applications such as autonomous driving.Source: §1 · p. 1
  • BayesRays adapts photogrammetry’s feature-position uncertainty analogy by locating radiance-field regions that can be perturbed with minimal reconstruction-loss impact.Figure 2 presents the photogrammetry inspiration alongside epistemic uncertainty in NeRF.Source: §1 · p. 1, Figure 2 · p. 1
  • BayesRays applies spatially parameterized radiance-field perturbations and a Bayesian Laplace approximation to any pre-trained NeRF, without changing its training pipeline.It produces a volumetric uncertainty field that can be rendered as an additional color channel.Source: §1 · p. 2
  • BayesRays uncertainties are statistically meaningful, outperform prior work on key metrics, and support artifact removal matching or improving the state of the art.The framework also computes a spatial uncertainty field in little over a minute and proposes interactive thresholding for artifact removal.Source: §1 · p. 2

2. Related work

Prior work quantifies uncertainty through photogrammetry, deep-learning methods, and NeRF-specific proxies, with trade-offs in computational cost, training changes, or model assumptions. BayesRays applies a Laplace approximation post hoc to any pre-trained NeRF and also proposes uncertainty-based artifact removal.

Uncertainty in computer vision and deep learning

  • Classic photogrammetry models reconstruction uncertainty with 2D image-space Gaussians projected into 3D, while deep-learning uncertainty distinguishes data randomness from model knowledge gaps.Aleatoric uncertainty concerns randomness in the data; epistemic uncertainty concerns a model’s lack of knowledge, for example from missing data.Source: §2 · p. 2

Uncertainty in deep learning

  • Deep ensembles estimate epistemic uncertainty by comparing differently initialized networks, while variational Bayesian networks model parameter distributions but change training and incur computational cost.Post-hoc Laplace approximations instead estimate a weight posterior near an already-trained value using derivative information.Source: §2 · p. 2

Uncertainty in Neural Radiance Fields

  • NeRF-W and follow-ups quantify aleatoric uncertainty from transient objects, lighting changes, and camera specifications using deep-learning techniques and learned appearance embeddings.Source: §2 · p. 2
  • For NeRF epistemic uncertainty, ensemble and variational-inference methods can be costly or require intricate training-pipeline changes; BayesRays instead uses Laplace approximations on any pre-trained NeRF.The paper describes BayesRays as the first framework to apply Laplace approximations to NeRF uncertainty without variational optimization.Source: §2 · pp. 2–3

NeRF-specific uncertainty proxies

  • NeRF-specific proxies use ray-termination entropy or generative-model scene variance, but their assumptions depend on density regularizers or model priors.Source: §2 · p. 3

NeRF artifact removal

  • Unlike training-time regularizers, BayesRays proposes post-processing pre-trained NeRFs to remove spatial artifacts, matching or improving Nerfbusters in that application without additional training.Nerfbusters uses a diffusion model and visibility mask to guide additional training steps.Source: §2 · p. 3

3. Background

The background introduces a general framework for applying Laplace approximations to pre-trained NeRFs, then reviews NeRF rendering and the posterior approximation used to quantify uncertainty. A direct Laplace approximation over NeRF weights is impractical, motivating an architecture-independent spatial perturbation field.

3. Background

  • The background frames the method as applying Laplace approximations to quantify epistemic uncertainty in any pre-trained NeRF.Source: §3 · p. 3

3.1. Neural Radiance Fields

  • A NeRF maps 3D points to view-dependent radiance and view-independent density, then volume-renders sampled points along each camera ray.Its parameters are optimized to minimize squared distance between predicted and ground-truth pixel colors.Source: §3.1 · p. 3
  • Figure 4 proposes thresholding the computed uncertainty field to clean a learned scene, matching or surpassing state-of-the-art performance at lower computational and memory cost.Source: Figure 4 · p. 3

3.2. Neural Laplace Approximations

  • A Laplace approximation centers a Gaussian posterior at the trained parameter mode and derives its covariance from a second-order expansion of the negative log-likelihood.The Hessian supplies the second derivatives used in this approximation.Source: §3.2 · p. 4
  • Applying Laplace directly to NeRF weights is impractical because dense parameter correlations make covariance computation costly, transferring uncertainty to geometry requires repeated sampling, and the approach depends on architecture.The paper addresses these issues with an architecture-independent parametric perturbation field that supports a spatial uncertainty field without drawing distribution samples.Source: §3.2 · p. 4

4. Method

BayesRays treats NeRF uncertainty as the extent to which scene geometry can be perturbed without harming reconstruction accuracy. It reparameterizes a pretrained radiance field with spatial deformations, approximates their posterior uncertainty, and produces a spatial uncertainty field.

4. Method

  • The method accepts a pretrained radiance field as a differentiable black box and formulates a Laplace approximation on a reparameterization rather than the network weights.Its stated inputs include ground-truth camera parameters for the training images.Source: §4 · p. 4

4.1. Intuition

  • In an underdetermined reconstruction, multiple geometries can match the training pixels, so uncertainty is framed as how much the scene can be perturbed without hurting reconstruction accuracy.A Nerfacto reconstruction trained on 40 front and back laptop views can appear pixel-perfect from those views yet reveal geometric artifacts from another angle.Source: §4.1 · pp. 4–5

4.2. Modeling perturbations

  • BayesRays models perturbations as a trilinearly interpolated displacement grid and approximates a zero-centered Gaussian posterior; very low grid resolutions may underestimate uncertainty.The grid resolution M has diminishing returns above 256.Source: §4.2 · p. 5, Figure 5 · p. 5

4.3. Approximating H

  • The Hessian approximation uses first derivatives, and its computation depends on the pretrained model and training-camera parameters rather than training-image data.The final Hessian expression approximates its expectation by sampling R rays.Source: §4.2 · p. 5, §4.3 · p. 6

4.4. Spatial uncertainty

  • Spatially limited parameter effects make the Hessian sparse, allowing the method to approximate covariance from its diagonal and use local deformation variance to define the spatial uncertainty field U.The norm of the marginal-variance vector provides a scalar uncertainty value at each grid vertex; U is the algorithm’s main output.Source: §4.4 · p. 6

5. Experiments & Applications

BayesRays is evaluated for uncertainty calibration and NeRF cleanup, with ablations examining architecture, deformation-grid size, and uncertainty threshold. The method captures epistemic, but not aleatoric, uncertainty.

5.1. Uncertainty Evaluation – Figures 6

  • BayesRays uncertainty correlates with depth error significantly better than CF-NeRF and is extremely close to a costly ten-NeRF ensemble in AUSE and sparsification plots.Figure 6 compares ScanNet and Light Field results; ensemble uncertainty comes from ten differently seeded NeRFs.Source: Figure 6 · p. 7, §5.1 · p. 7

5.2. NeRF Clean Up – Figures 1 and 4

  • With fixed thresholds, BayesRays achieves similar PSNR to Nerfbusters at higher coverage; selecting the best of ten thresholds improves both PSNR and coverage.Cleanup takes 96 seconds versus 20 minutes, without Nerfbusters’ 3D diffusion model, additional training, or visibility mask.Source: §5.2 · p. 7
  • Qualitatively, BayesRays filters floaters missed by Nerfbusters, avoids sampling artifacts, and allows flexibility to avoid over-filtering.The examples are Century, Flowers, and Pikachu, respectively.Source: §5.2 · p. 7

5.3. Algorithmic ablations – Figures 3, 5 and 8

  • BayesRays produces qualitatively similar results with Mip NeRF, Instant NGP, and Nerfacto, supporting use across different NeRF architectures.The comparison uses the Lego scene with 60 left-hemisphere training views.Source: §5.3 · p. 7
  • BayesRays quantifies epistemic uncertainty only, so it does not capture aleatoric effects such as those caused by training inconsistencies.Source: Figure 7 · p. 8
  • Small deformation-grid sizes can miss surface uncertainty, while larger, more costly sizes yield diminishing returns beyond M > 256.Source: §5.3 · p. 8
  • Lowering the uncertainty threshold progressively removes floaters, and the once-computed field lets the threshold serve as real-time user control.Source: §5.3 · p. 8

6. Conclusions

BayesRays produces a spatial uncertainty field for trained NeRFs, without additional training or access to training images, and its uncertainty correlates with depth error and can be thresholded for cleanup. The method is limited to NeRFs, captures epistemic rather than aleatoric uncertainty, and has memory and approximation constraints.

6. Conclusions

  • BayesRays outputs a spatial uncertainty field meaningfully correlated with NeRF depth error and usable by thresholding for cleanup.It operates on trained NeRFs regardless of architecture, without additional training or access to original training images.Source: §6 · p. 8
  • The uniform deformation grid can consume substantial memory in regions with little geometric interest.The authors suggest hierarchical structures such as octrees as a possible improvement.Source: §6 · p. 8
  • The derivation focuses on the diagonal of H and disregards minimal inter-parametric correlations.The authors note that future applications may include them using low-rank matrix decompositions.Source: §6 · p. 8
  • The method targets epistemic uncertainty from missing or occluded data, not aleatoric uncertainty from noise or view inconsistencies.The authors suggest combining it with aleatoric-quantification frameworks to study all uncertainty sources.Source: §6 · p. 8
  • On the “garbage” scene, decreasing the uncertainty threshold progressively cleans a floater-heavy scene; the infinite-threshold image is the original.The threshold can serve as a real-time user control because the uncertainty field needs to be computed only once.Source: Figure 8 · p. 8
  • BayesRays quantifies uncertainty for NeRFs and cannot be trivially translated to other frameworks such as 3D Gaussian splatting.The authors look forward to similar deformation-based Laplace approximations for newer spatial representations.Source: §6 · p. 8

Related papers

Loading 2309.03185v1…