PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth-supervised NeRF: Fewer Views and Faster Training for Free

Kangle Deng, Andrew Liu, Jun-Yan Zhu, Deva Ramanan

arXiv:2107.02791v3cs.CVcs.GRcs.LG

TL;DR

NeRF can fit incorrect geometries from insufficient views and remains slow to train. DS-NeRF addresses both issues by supervising ray termination with uncertain depth from SFM and other sources. It renders better sparse-view results and trains 2-3x faster, while remaining compatible with NeRF variants; its effectiveness depends on depth quality and its uncertainty model assumes Gaussian keypoint error.

  • Problem

    NeRF can overfit few input views with inaccurate geometries and often takes ten hours to several days to train a scene.

  • Method

    DS-NeRF directly supervises NeRF density by matching ray termination distributions to SFM 3D keypoints while incorporating reprojection error as uncertainty.

  • Results

    DS-NeRF produces better results from fewer views and trains 2-3x faster than NeRF, with improvements also shown for NeRF variants and other depth sources.

  • Takeaways & Limitations

    Depth from standard SFM preprocessing provides a cheap supervisory signal that can be integrated into NeRF-based methods without additional training.

  • Takeaways & Limitations

    Performance depends on depth estimates, and the method’s Gaussian model for keypoint uncertainty may not hold for depth from other sources.

Abstract

from arXiv · show

A commonly observed failure mode of Neural Radiance Field (NeRF) is fitting incorrect geometries when given an insufficient number of input views. One potential reason is that standard volumetric rendering does not enforce the constraint that most of a scene's geometry consist of empty space and opaque surfaces. We formalize the above assumption through DS-NeRF (Depth-supervised Neural Radiance Fields), a loss for learning radiance fields that takes advantage of readily-available depth supervision. We leverage the fact that current NeRF pipelines require images with known camera poses that are typically estimated by running structure-from-motion (SFM). Crucially, SFM also produces sparse 3D points that can be used as "free" depth supervision during training: we add a loss to encourage the distribution of a ray's terminating depth matches a given 3D keypoint, incorporating depth uncertainty. DS-NeRF can render better images given fewer training views while training 2-3x faster. Further, we show that our loss is compatible with other recently proposed NeRF methods, demonstrating that depth is a cheap and easily digestible supervisory signal. And finally, we find that DS-NeRF can support other types of depth supervision such as scanned depth sensors and RGB-D reconstruction outputs.

1. Introduction

NeRF can overfit sparse input views by learning inaccurate geometries, while also requiring lengthy training. DS-NeRF adds inexpensive depth supervision from SFM outputs, improving sparse-view rendering and accelerating training.

  • Motivation: NeRF can learn inaccurate geometries from few views that render training images well but fail on novel views.The resulting depth maps and errors indicate that the density function does not recover surface geometry.
  • Method: DS-NeRF uses SFM-produced sparse 3D points and reprojection errors as uncertain depth supervision for ray termination distributions.The loss softly anchors NeRF’s implicit correspondences to sparse explicit correspondences recovered by SFM.
  • Results: 2-3x faster training is achieved while producing results with the same quality.The reported improvement applies to NeRFs and their variations.
  • Results: DS-NeRF synthesizes better sparse-view results than original NeRF and recent sparse-view NeRF models on NeRF Real and Redwood-3dscan.The depth supervision loss also works with alternative sources such as depth cameras.

2. Related Work

Prior work addresses few-view and slow NeRF training using learned priors, 3D scene information, or sampling and optimization strategies. DS-NeRF instead directly supervises NeRF density with accessible depth, including SFM points and sensor-derived measurements.

  • Few-view NeRF: Few-view NeRF methods use data-driven priors from training scenes to fill in missing information for test scenes.These approaches work when sufficient training scenes and applicable priors are available.
  • Few-view NeRF: MVSNeRF combines 3D knowledge from a plane-sweep volume with pretrained generalizable scene priors.Its approach targets the difficulty of modeling 3D geometry from few views.
  • Faster NeRF: Faster-NeRF research identifies volumetric optimization time and oversampling of empty space as major training bottlenecks.NeRF’s median uniform-density initialization contributes to inefficient sampling of empty regions.
  • Depth and NeRF: Depth-based methods have used sensors and keypoints for view synthesis, deformable surfaces, dynamic flows, and faster rendering.These works motivate treating depth as an increasingly accessible supervisory signal for implicit representations.
  • Depth and NeRF: DS-NeRF directly supervises the NeRF density function with a probabilistic depth loss and supports depth from SFM, time-of-flight, or RGB-D sensors.The paper reports meaningful improvements to NeRF and its variants from this supervision.

3. Depth-Supervised Ray Termination

This section revisits NeRF’s volumetric rendering through the ray termination distribution and derives depth supervision from SFM keypoints with uncertainty. The resulting loss trains termination distributions toward observed surface depths while combining color and depth objectives.

  • 3.1. Volumetric rendering revisited: NeRF represents a scene with density and radiance fields, then renders images by integrating radiance along camera rays.The field maps 3D points and viewing directions to differential density and RGB color; practical implementations approximate the integral with sampled Riemann sums.
  • 3.1. Volumetric rendering revisited: The ray termination distribution h(t)=T(t)σ(t) describes the likelihood that a ray terminates at distance t, and rendered color is its expectation over sampled radiances.NeRF implementations commonly treat the far bound as an opaque wall so h(t) sums to one.
  • 3.2. Deriving depth-supervision: Because scenes largely contain empty space and opaque surfaces, the ideal termination distribution for a ray with closest-surface depth D is δ(t−D).Empirical termination distributions become lower-variance and more sharply peaked as training views increase.
  • 3.2. Deriving depth-supervision: SFM pipelines provide sparse 3D keypoints and visibility information, allowing visible keypoint depths to be estimated by projection into each camera.The projected z value supplies the keypoint’s depth for the corresponding image coordinates.
  • 3.2. Deriving depth-supervision: DS-NeRF models keypoint depth uncertainty from reprojection error and minimizes KL divergence between the rendered termination distribution and the uncertain depth target.The equivalence to KL divergence against a Gaussian depth target enables probabilistic COLMAP supervision.
  • 3.2. Deriving depth-supervision: The overall objective adds weighted depth supervision to the color reconstruction loss: L = LColor + λD LDepth.λD balances color and depth supervision during NeRF training.

4. Experiments

Experiments evaluate DS-NeRF on few-view view synthesis, depth accuracy, training speed, compatibility with existing methods, and alternate depth sources. DS-NeRF improves sparse-view rendering and training efficiency, while its performance depends on depth quality and SFM success.

  • Few-input view synthesis: DS-NeRF outperforms competing methods on NeRF Real across multiple view-synthesis metrics, especially with 2 and 5 input views.MetaNeRF-DTU and pixelNeRF-DTU struggle on NeRF Real because of the domain gap to DTU, while PixelNeRF, IBRNet, and MVSNeRF benefit from depth supervision.
  • Depth supervision ablations: KL-divergence depth supervision produces fewer artifacts than NeRF or an MSE depth loss in qualitative ablations.The comparison uses models trained with 2 and 5 views on NeRF Real sequences.
  • Alternate depth supervision: On Redwood, replacing sparse COLMAP supervision with scaled sensor depth achieves a PSNR of 20.3 with 2 views.The denser depth measurements improve results beyond the sparse supervision setting.
  • Faster training: DS-NeRF reaches NeRF’s peak test PSNR using 2-3x fewer training iterations and is about 13 hours faster in the 5-view case.Per-iteration costs are nearly identical: approximately 362.4 ms for DS-NeRF versus 359.8 ms for NeRF on an RTX A5000.

B.1. Derivation of h(t) as a probability distribution

The paper establishes that h(t) is a valid continuous probability distribution over ray termination by proving non-negativity and unit integral under an opaque-intersection assumption. Finite near and far bounds are handled by treating the farthest radiance as an opaque wall.

  • h(t) represents a continuous probability distribution over ray termination when its values are non-negative and integrate to 1.The derivation treats h(t) as a contribution weight indexed by distance t.
  • Non-negative density σ(s) makes transmittance T(t) non-negative, so h(t)=T(t)σ(t) is non-negative for every t.σ(s) is typically implemented with a ReLU or softplus activation.
  • The unit-integral proof assumes every ray eventually intersects an opaque object, expressed by divergent accumulated density along each ray.The paper notes this is true for most modeled scenes because radiance is emitted by surfaces.
  • Because practical NeRFs sample only between near and far bounds, implementations often model the furthest radiance as an opaque wall to preserve the assumption.This maintains the probability-distribution interpretation despite finite sampling limits.

B.2. Depth-supervision implementation

Depth supervision projects rays through detected keypoints and renders their depths using NeRF’s sampling procedure, while RGB interpolation enables joint color-and-depth training in one ray-marching pass. The method is applied to several NeRF variants, but transferring meta-initializations across domains remains outside scope.

  • Depth supervision projects each detected keypoint into a camera-coordinate ray and renders its depth with NeRF’s standard sampling procedure.The ray direction is determined by the keypoint’s image coordinates and the camera-axis component −1.
  • A single ray-marching procedure jointly trains color and depth supervision to reduce the cost of repeated forward passes.This is motivated by ray marching being the dominant training-time cost when data fits in GPU memory.
  • RGB values are interpolated at keypoint rays because image coordinates are continuous, allowing those rays to supervise both modalities.Only a portion of training rays is allocated to keypoint supervision.
  • The COLMAP structure-from-motion output is used during training, while novel-view synthesis does not require it at test time.The setup can use limited inputs, including two views.
  • Depth supervision is incorporated into pixelNeRF finetuning, while cross-domain metaNeRF transfer is left unaddressed because ray bounds and coordinate scales differ.The experiments instead use default scaling and ray bounds for DTU and NeRF Real.

B.5. Dataset Splits

The evaluation partitions NeRF Real scenes into training and test views using every eighth image for testing, then creates few-view subsets from the remaining views. Redwood 3dscan uses 15 RGB-depth frames with fixed training and test viewpoint lists.

  • NeRF Real: NeRF Real assigns image ids 0, 8, 16, 24, and so on to testing, with training views distributed across the remaining viewpoints.Few-input subsets are created from the training views for view-synthesis evaluation.
  • Redwood 3dscan: Redwood 3dscan evaluation uses five scenes: table, plant, chair, car, and stool.These are the selected test scenes from the dataset.
  • Redwood 3dscan: Each Redwood scene contains 15 RGB-depth frames, split into training and test views.The fixed test viewpoints are [0, 3, 6, 9, 12].
  • Redwood 3dscan: Redwood training viewpoints are [11] [14] [1] [4] [7] [10] [13], truncated for smaller input subsets such as two views.The two-view setting uses viewpoints 5 and 11.

B.6. Depth Error Evaluation

Depth error is evaluated by comparing rendered depth from test cameras against reference depth derived from dense multi-view stereo. The reference is aligned to sparse SfM keypoint depth using a learned scale and shift, and dense MVS is excluded from training and testing.

  • Depth evaluation computes a reference depth for an input scene from the test camera poses.This reference supports comparison with rendered depth.
  • Dense MVS produces reference depth maps for training and test views, while COLMAP supplies sparse keypoints and camera poses.The dense maps are used to evaluate scenes from the selected test poses.
  • A scalar scale a and shift b align visible SfM keypoint depths from training views with their MVS depths.The alignment relates sparse training supervision to dense evaluation references.
  • Depth error compares rendered depth D̂ from a test camera against the aligned reference depth.The rendered depth is transformed before computing the error.
  • Dense MVS depth is used only for depth evaluation, not during training or testing by any method.This separates the evaluation reference from the methods’ depth inputs.

C. Additional Experiments

Additional experiments test whether learned priors help under realistic cross-scene evaluation and compare DS-NeRF with MPI-based rendering. Meta-learning baselines struggle to leverage training-scene priors, while DS-NeRF produces stronger qualitative results from two views.

  • Cross-validation: MetaNeRF and pixelNeRF struggle to leverage priors learned from NeRF Real training scenes during four-fold cross-validation.The evaluation uses six training scenes and two test scenes per split.
  • Cross-validation: Training on NeRF Real does not improve metaNeRF or pixelNeRF over training on DTU.The authors attribute this possibly to the larger DTU training set: 88 scenes versus 6 scenes per NeRF Real fold.
  • MPI comparison: DS-NeRF qualitatively produces better novel views than LLFF and NeRF when inferred from 2 input views.The comparison is shown on NeRF Real; the text reports that LLFF struggles to match DS-NeRF’s visual quality.
  • Depth supervision: COLMAP keypoint visualizations show the available depth supervision for 2, 5, and 10 input views.These visualizations document how sparse keypoint supervision changes with the number of input views.

C.2. MPI-based Experiments

The MPI-based LLFF baseline is evaluated against DS-NeRF, while keypoint ablations test how sparse depth supervision affects performance. DS-NeRF remains stronger than the NeRF baseline even when most detected keypoints are removed.

  • MPI-based baseline: LLFF is included as an MPI-based baseline on NeRF Real, with qualitative comparisons in Figure 8 and quantitative results reported in Table 1.The supplied experiment description directs readers to the main paper for the numerical comparison.
  • Keypoint supervision: DS-NeRF’s PSNR rises from 21.5 to 22.6 as keypoint supervision increases from 20% to 100%, compared with 18.2 for NeRF.With 50% of keypoints, DS-NeRF reaches 22.2 PSNR; the input is fixed to 5 views.
  • Keypoint supervision: Dropping keypoints weakens DS-NeRF’s performance but it still outperforms the NeRF baseline.The ablation uniformly removes detected keypoints while keeping the input at 5 views.
  • Keypoint supervision: COLMAP provides an average of 1615, 2172, and 2621 keypoints per training view for 2, 5, and 10 input views, respectively.Figure 9 provides qualitative examples of the corresponding keypoint supervision.
Loading 2107.02791v3…