PaperlayerView exact revision on arXiv

Source-linked AI summary

ActiveNeRF: Learning where to See with Uncertainty Estimation

Xuran Pan, Zihang Lai, Shiji Song, Gao Huang

arXiv:2209.08546v1cs.CV

TL;DR

NeRF can generalize poorly and collapse in unobserved regions when posed training images are scarce, limiting use under constrained capture budgets. ActiveNeRF adds uncertainty estimation and active selection of informative views, improving novel-view synthesis on synthetic and realistic scenes, especially with scarce data. Its uncertainty formulation assumes sampled rendered rays are independent because each scene location is rarely sampled more than once per training batch.

  • Problem

    NeRF generalizes poorly from limited posed images and may collapse to zero density in unobserved regions, creating difficulties when training data are costly to capture.

  • Method

    ActiveNeRF models each location’s radiance color with a Gaussian distribution and selects supplementary views by estimating posterior uncertainty reduction and information gain.

  • Results

    ActiveNeRF shows superior performance on synthetic and realistic scenes, outperforming several heuristic baselines, with consistently better results under limited training samples.

  • Takeaways & Limitations

    ActiveNeRF improves novel-view synthesis with minimal additional resources by selecting the most informative samples and can serve as a plug-in module for NeRF extensions.

  • Takeaways & Limitations

    The uncertainty optimization assumes each scene location is sampled at most once per training batch and therefore treats rendered rays as independent.

Abstract

from arXiv · show

Recently, Neural Radiance Fields (NeRF) has shown promising performances on reconstructing 3D scenes and synthesizing novel views from a sparse set of 2D images. Albeit effective, the performance of NeRF is highly influenced by the quality of training samples. With limited posed images from the scene, NeRF fails to generalize well to novel views and may collapse to trivial solutions in unobserved regions. This makes NeRF impractical under resource-constrained scenarios. In this paper, we present a novel learning framework, ActiveNeRF, aiming to model a 3D scene with a constrained input budget. Specifically, we first incorporate uncertainty estimation into a NeRF model, which ensures robustness under few observations and provides an interpretation of how NeRF understands the scene. On this basis, we propose to supplement the existing training set with newly captured samples based on an active learning scheme. By evaluating the reduction of uncertainty given new inputs, we select the samples that bring the most information gain. In this way, the quality of novel view synthesis can be improved with minimal additional resources. Extensive experiments validate the performance of our model on both realistic and synthetic scenes, especially with scarcer training data. Code will be released at \url{https://github.com/LeapLabTHU/ActiveNeRF}.

1 Introduction

ActiveNeRF addresses NeRF’s poor generalization from sparse or incomplete observations by estimating uncertainty and actively selecting informative new views. The framework targets improved novel-view synthesis under constrained input budgets.

  • Motivation: NeRF often requires many posed images and can collapse in unobserved regions when scene observations are incomplete.Such failures are problematic when capturing training data is costly and full-scene perception is required.
  • Method: ActiveNeRF models radiance values as Gaussian distributions so unobserved regions retain larger uncertainty instead of collapsing to trivial solutions.The uncertainty model is incorporated into the NeRF framework to improve robustness with few observations.
  • Method: The method evaluates posterior uncertainty reduction for hypothetical inputs and selects samples with the greatest information gain.This active-learning criterion determines which new views should be captured.
  • Motivation: 10 posed images with left-side observations removed expose large errors or meaningless content for original NeRF, while ActiveNeRF still produces reasonably good synthesis.The comparison illustrates the benefit of uncertainty estimation under partial observations.
  • Results: ActiveNeRF outperforms several heuristic baselines on both synthetic and realistic scenes.The reported advantage is especially relevant when training data are scarce.

2 Related Works

Related work extends NeRF’s scene representation, rendering, and limited-input generalization, while ActiveNeRF instead addresses data efficiency through active learning and uncertainty estimation. The authors position it as the first approach to integrate active learning into NeRF optimization.

  • Novel View Synthesis: NeRF research extends the original framework to unbounded, accelerated, dynamic, and transient-scene settings.Examples include NeRF++, FastNeRF, D-NeRF, and NeRF-W.
  • Limited Input: Limited-input methods such as Pixel-NeRF, MVSNeRF, GRF, and DietNeRF improve generalization through image features, neural volumes, back-projected features, or semantic consistency.These approaches primarily modify model representations or training losses.
  • ActiveNeRF: ActiveNeRF differs by addressing NeRF limitations from the data perspective and increasing data efficiency with minimal additional resources.Its uncertainty module is described as orthogonal to existing NeRF extensions and usable as a plug-in module.
  • Uncertainty Estimation: Uncertainty estimation has been used to improve interpretability and reduce critical-fault risk, with Bayesian methods modeling uncertainty as probability distributions.Prior novel-view work includes NeRF-W and S-NeRF, which target different uncertainty settings.
  • Active Learning: ActiveNeRF is presented as the first approach to incorporate active learning into the NeRF optimization pipeline for resource-constrained applications.The approach analyzes imperfections in training data rather than only increasing model capacity.

3 Background

NeRF represents a scene with a continuous function that predicts density and view-dependent color, then renders rays by integrating contributions along sampled points. Training minimizes reconstruction error using coarse and fine networks.

  • Scene Representation: NeRF maps a 3D position and viewing direction to volume density and RGB radiance through a multilayer perceptron.Positional encodings provide the spatial and directional inputs, with an intermediate feature independent of viewing direction.
  • Volume Rendering: Volume rendering computes each ray’s color by accumulating emitted radiance weighted by transmittance and density along the ray.The ray is parameterized from the camera center through an image-plane pixel.
  • Volume Rendering: NeRF approximates the rendering integral as a weighted sum over sampled points along each ray.The discrete weights depend on sampled densities, intervals, and accumulated transmittance.
  • Optimization: The model minimizes squared differences between ground-truth pixel colors and predictions from parallel coarse and fine networks.Fine-model sampling is biased toward more relevant parts according to the coarse model.

4 NeRF with Uncertainty Estimation

NeRF with uncertainty estimation models each location’s radiance as a Gaussian distribution, using predicted variance to represent uncertainty and avoid trivial predictions in unobserved regions. A sparsity regularizer preserves sharper surfaces while the uncertainty model supports interpretation and subsequent active learning.

  • Uncertainty modeling: Predicted variance represents aleatoric uncertainty and increases in unobserved regions instead of allowing collapse to a trivial solution.This yields reasonable predictions in uncertain areas while indicating how the model understands the scene.
  • Uncertainty modeling: ActiveNeRF models each location’s emitted radiance with a Gaussian distribution parameterized by mean and variance.The mean is the model output, while an additional MLP branch predicts the variance.
  • Uncertainty modeling: Rendered ray colors also follow Gaussian distributions because independent location distributions combine linearly through volume rendering.The rendered mean and variance are computed from sampled points and their rendering weights.
  • Optimization: A regularization term forces sparser volume density after the unregularized objective spreads nonzero density across the scene and blurs object surfaces.On the reported scene, regularization improves performance by 1.1 PSNR.
  • Optimization: The final loss combines the uncertainty-branch loss for the fine model with the vanilla NeRF reconstruction loss for the coarse model.The uncertainty branch is used only in the fine model to ease optimization.

5 ActiveNeRF

ActiveNeRF uses uncertainty estimation to identify new viewpoints whose observations most reduce radiance-field uncertainty, then adds those captures to the training set. Bayesian updating and an acquisition function make sample selection information-driven, with lower-resolution evaluation offering an efficiency trade-off.

  • Motivation: ActiveNeRF targets blind spots caused by limited observations, where unobserved scene regions can be under-optimized and prevent photo-realistic rendering.The method is designed to improve the performance upper bound under a constrained input budget.
  • Prior and Posterior Distribution: The framework estimates a new input’s effect by combining the existing model’s prior distribution with a likelihood through Bayesian updating.The prior is conditioned on the existing training set, while the posterior conditions on the existing set and a new ray.
  • Acquisition Function: A newly observed location has posterior variance smaller than its prior variance, so variance reduction measures the information gained from the new ray.The posterior variance can be computed from the current model without knowing the ground-truth mean.
  • Acquisition Function: The acquisition function sums prior-minus-posterior variance reductions over sampled locations and rays in a candidate image.For an image of resolution H×W, it aggregates the reductions across N = H×W rays and Ns sampled locations per ray.
  • Acquisition Function: Candidate views are sampled from a spherical space, and the top-k views with the highest acquisition scores are selected for capture and added to the training set.This selection is intended to bring the most information gain with high efficiency.
  • Optimization and Inference: Evaluating candidate inputs at lower resolution trades quality for efficiency by using H/r×W/r rays with only 1/r^2 time consumption.The lower-resolution estimate approximates the influence of the full image.
  • Optimization and Inference: With new captured samples, Bayesian estimation replaces the prior color with a posterior radiance estimate during inference.The posterior mean becomes the Bayesian estimate of emitted radiance, while other rendering components remain unchanged.
  • Experiments: ActiveNeRF performs on par with or better than original NeRF across fixed-training-set settings, with significantly stronger results in low-shot settings.Reported metrics are PSNR and SSIM, where higher is better, and LPIPS, where lower is better.

6 Experiments

Experiments evaluate ActiveNeRF on synthetic and realistic scenes under sparse-data and active-learning settings. The results show stronger low-shot synthesis, informative sample selection, and substantial savings with Bayesian estimation.

  • Experimental Setup: The active-learning evaluation uses LLFF and NeRF benchmarks with PSNR and SSIM as image-quality metrics.LLFF contains realistic cellphone-captured scenes, while the NeRF dataset contains synthetic objects.
  • Uncertainty Estimation: ActiveNeRF consistently improves synthesis with limited training samples, while uncertainty modeling does not reduce quality when more data is available.NeRF with uncertainty performs on par or slightly better than baseline models; limited-sample results are consistently better.
  • ActiveNeRF: ActiveNeRF outperforms heuristic sampling approaches by selecting informative inputs that improve synthesis in less observed regions.The active-learning comparison uses random capture and furthest-view sampling as heuristic baselines.
  • ActiveNeRF: 2.2h vs. 2h: ActiveNeRF adds comparatively minor training cost under continuous learning.The reported comparison concerns ActiveNeRF’s additional training cost relative to the baseline.
  • ActiveNeRF: 75% of time consumption can be saved with Bayesian estimation, although its performance is inferior to continuous learning while remaining reasonable.The Bayesian-estimation model is also competitive with heuristic approaches under continuous learning.

7 Conclusion

ActiveNeRF supplements NeRF training with newly captured samples selected through uncertainty reduction and information gain. The framework improves novel-view synthesis with constrained resources and can serve as a plug-in for NeRF extensions.

  • Conclusion: ActiveNeRF selects newly captured samples by evaluating uncertainty reduction and information gain.The framework supplements the existing training set through active learning.
  • Conclusion: The framework promotes novel-view synthesis quality with minimal additional resources and can enhance various NeRF extensions as a plug-in module.The conclusion describes the approach as flexible and resource-efficient.

A. Derivation of Posterior Distribution (Eq.(17))

The derivation forms a posterior distribution by combining a Gaussian prior over the scene radiance with a Gaussian likelihood for newly observed data. The resulting posterior remains Gaussian.

  • Posterior Derivation: The prior distribution models the radiance at a sampled location as a Gaussian with mean bar c(r_2(t_k)) and variance bar beta^2(r_2(t_k)).This prior conditions on the existing dataset D_1.
  • Posterior Derivation: The new-data distribution is modeled as a Gaussian with mean alpha_k bar c(r_2(t_k)) + b(t_k) and variance bar beta^2(r_2).The observation is conditioned on the latent radiance at the sampled point.
  • Posterior Derivation: The posterior combines the likelihood and prior into a Gaussian distribution whose mean and variance depend on their respective uncertainty terms.The posterior expression is obtained by multiplying the Gaussian factors and normalizing.

B. Derivation of Posterior Distribution with Multiple New Data (Eq.(22))

With multiple new observations, the posterior combines each Gaussian likelihood with the prior at a shared scene location. The derivation expresses the result as a Gaussian weighted by observation and prior uncertainties.

  • Multiple New Data: For two new inputs meeting at x, the posterior multiplies both observation likelihoods with the prior distribution.The derivation states that more-input cases follow similarly with substitutions.
  • Multiple New Data: The prior for the shared location x is Gaussian with mean bar c(x) and variance beta_0^2.This prior conditions on the existing dataset D_1.
  • Multiple New Data: Each new observation is modeled as a Gaussian whose mean is alpha_i x + b_i and whose variance is beta_i^2.The two observations use separate parameters for their respective likelihoods.
  • Multiple New Data: The resulting posterior mean is a weighted combination of the two observations and the prior mean, with posterior variance gamma_3 beta_0^2.The weights gamma_1, gamma_2, and gamma_3 are given by the uncertainty-dependent expressions.
  • Multiple New Data: ActiveNeRF’s qualitative comparisons include low-shot performance against NeRF and heuristic sampling against NeRF+Random.These comparisons motivate the experimental evaluation of uncertainty-guided sample selection.

C. Model Architectures

ActiveNeRF largely follows NeRF’s architecture while adding uncertainty prediction to the encoded scene representation. Training uses positional encoding, separate coordinate and direction settings, and standard NeRF optimization configurations.

  • C. Model Architectures: ActiveNeRF uses NeRF’s architecture with positional encoding to represent high-frequency scene details.The encoding is applied separately to 3D coordinates and Cartesian viewing directions.
  • C. Model Architectures: The model processes encoded 3D coordinates with an eight-layer MLP and a residual connection at the fourth layer.A fully connected layer predicts volume density σ before view-dependent color prediction.
  • C. Model Architectures: Unlike original NeRF, ActiveNeRF adds a softplus layer that predicts the variance corresponding to each volume density.The latent feature is concatenated with encoded viewing directions to produce RGB color.
  • C. Model Architectures: The experiments sample 64 and 128 points for the coarse and fine models, respectively, and optimize with Adam.The learning rate decays from 5e−4 to 5e−5, using batches of 1024 rays on one RTX2080Ti GPU.

E. Additional Qualitative Results

Additional visualizations show ActiveNeRF across static and active scenarios. With active iterations, newly captured perceptions improve synthesis quality in previously unobserved regions.

  • E. Additional Qualitative Results: Additional visualization results cover both static and active scenarios in Figures 7–9.These figures provide qualitative results for the two evaluation settings.
  • E. Additional Qualitative Results: New perceptions are captured every 40K iterations, with improved synthesis quality visible in unobserved regions.Figure 9 presents qualitative results of ActiveNeRF during active iterations on realistic scenes.

F. Ablation Study

The ablation study examines the candidate evaluation module and the cost–quality trade-off of lower-resolution perception evaluation. ActiveNeRF consistently outperforms heuristic acquisition variants, remains competitive at sampling rate 5, and degrades gradually above 10.

  • F. Ablation Study: ActiveNeRF consistently outperforms the ActiveNeRF + Random and ActiveNeRF + FVS heuristic acquisition baselines.This ablation keeps the uncertainty module and replaces the acquisition function with heuristic approaches.
  • F. Ablation Study: The acquisition-function ablation uses four initial observations and four extra observations captured at 40K, 80K, 120K, and 160K iterations.Table 3 identifies this configuration as Setting I.
  • F. Ablation Study: At sampling rate 5, ActiveNeRF maintains competitive performance while reducing time consumption.This experiment evaluates the model when new perceptions are processed at lower resolution.
  • F. Ablation Study: When the sampling rate exceeds 10, novel-view synthesis quality is gradually affected.Table 4 reports the sampling-rate ablation.
Loading 2209.08546v1…