PaperlayerView exact revision on arXiv

Source-linked AI summary

SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis

Guangcong Wang, Zhaoxi Chen, Chen Change Loy, Ziwei Liu

arXiv:2303.16196v2cs.CV

TL;DR

Few-shot NeRF suffers from limited 3D information, while accurate multi-view depth supervision is difficult and expensive to obtain. SparseNeRF instead distills local depth ranking and spatial continuity from inaccurate model- or sensor-derived depth observations. It achieves state-of-the-art performance across LLFF, DTU, and NVS-RGBD, though it does not generalize to occluded views unobserved during training.

  • Problem

    Few-shot NeRF reconstruction is under-constrained, while accurate depth maps for supervision are difficult and expensive to capture.

  • Method

    SparseNeRF uses local relative-depth ranking and spatial continuity constraints to distill priors from pre-trained depth models or coarse consumer-sensor maps.

  • Results

    SparseNeRF achieves new state-of-the-art few-shot novel view synthesis performance on LLFF, DTU, and NVS-RGBD.

  • Takeaways & Limitations

    Inaccurate real-world depth observations can provide useful priors for few-shot NeRF novel view synthesis without increasing inference-time running time.

  • Takeaways & Limitations

    SparseNeRF cannot be generalized to occluded views that are unobserved in the training views.

Abstract

from arXiv · show

Neural Radiance Field (NeRF) significantly degrades when only a limited number of views are available. To complement the lack of 3D information, depth-based models, such as DSNeRF and MonoSDF, explicitly assume the availability of accurate depth maps of multiple views. They linearly scale the accurate depth maps as supervision to guide the predicted depth of few-shot NeRFs. However, accurate depth maps are difficult and expensive to capture due to wide-range depth distances in the wild. In this work, we present a new Sparse-view NeRF (SparseNeRF) framework that exploits depth priors from real-world inaccurate observations. The inaccurate depth observations are either from pre-trained depth models or coarse depth maps of consumer-level depth sensors. Since coarse depth maps are not strictly scaled to the ground-truth depth maps, we propose a simple yet effective constraint, a local depth ranking method, on NeRFs such that the expected depth ranking of the NeRF is consistent with that of the coarse depth maps in local patches. To preserve the spatial continuity of the estimated depth of NeRF, we further propose a spatial continuity constraint to encourage the consistency of the expected depth continuity of NeRF with coarse depth maps. Surprisingly, with simple depth ranking constraints, SparseNeRF outperforms all state-of-the-art few-shot NeRF methods (including depth-based models) on standard LLFF and DTU datasets. Moreover, we collect a new dataset NVS-RGBD that contains real-world depth maps from Azure Kinect, ZED 2, and iPhone 13 Pro. Extensive experiments on NVS-RGBD dataset also validate the superiority and generalizability of SparseNeRF. Code and dataset are available at https://sparsenerf.github.io/.

1. Introduction

SparseNeRF addresses the under-constrained few-shot NeRF problem by distilling relative depth and spatial continuity from inaccurate, readily available depth observations. It achieves state-of-the-art few-shot novel view synthesis while avoiding additional inference-time cost.

  • Motivation: Few-shot NeRF reconstruction is under-constrained, particularly in textureless regions, and existing geometry constraints cannot guarantee complete 3D reconstruction.Prior approaches also rely on semantics or pre-training on similar scenes.
  • Motivation: SparseNeRF distills local depth-ranking priors from pre-trained depth models or inaccurate consumer-level sensor maps instead of requiring accurate depth supervision.These observations are easier to obtain in real-world scenes, but deriving useful depth cues from them is non-trivial.
  • Method: Directly scaling coarse depth maps to NeRF supervision imposes inconsistent geometry because inaccurate observations are not strictly aligned with NeRF’s expected depth.SparseNeRF therefore uses relative depth comparisons rather than absolute depth constraints.
  • Method: Local depth ranking makes NeRF depth ordering consistent with coarse depth, while spatial continuity distillation preserves coherent scene geometry.The method combines these relaxed constraints with accurate sparse-geometry constraints from the input views.
  • Practical implications: SparseNeRF adds no inference-time running cost because depth priors are used only during training and the method can be integrated as a plug-and-play module.The framework is designed for compatibility with various NeRFs.
  • Results: SparseNeRF improves few-shot novel view synthesis over state-of-the-art methods, including depth-based NeRFs, on LLFF, DTU, and NVS-RGBD.NVS-RGBD contains coarse maps from Azure Kinect, ZED 2, and iPhone 13 Pro.

2. Related Work

Related work on novel view synthesis includes methods for scaling NeRFs, relaxing camera-pose requirements, handling dynamic scenes, and learning from sparse views. Sparse-view methods commonly use geometry or semantic constraints, while this paper focuses on reducing dense-capture requirements.

  • Neural Radiance Fields: Block-NeRF, CityNeRF, and Mega-NeRF scale standard NeRFs to city-scale or urban-scale scenes.These methods address scene scale rather than few-view novel view synthesis.
  • Neural Radiance Fields: NeRF−−, GNeRF, and BARF synthesize novel views without requiring perfect camera poses.Their contribution is relaxing camera-pose requirements.
  • Neural Radiance Fields: Other NeRF extensions address anti-aliasing, sparse 3D grids with spherical harmonics, and dynamic scenes.These methods do not focus on generating novel views from only a few views.
  • Few-shot Novel View Synthesis: This paper studies sparse-view NeRF to reduce dense scene-capture requirements in real-world applications.The focus distinguishes it from NeRF methods targeting scale, camera poses, or dynamics.
  • Few-shot Novel View Synthesis: Few-shot novel view synthesis methods exploit continuity constraints on geometry or object semantics, including RegNeRF and InfoNeRF.InfoNeRF additionally uses ray entropy minimization and information-gain reduction regularization.

3. Our Approach

SparseNeRF addresses few-shot NeRF reconstruction by distilling relative depth and spatial continuity priors from inaccurate depth observations instead of imposing absolute-depth supervision.

  • Framework Overview: SparseNeRF uses Mip-NeRF with color reconstruction and depth-prior distillation from pre-trained models or consumer-level sensors.The framework combines a NeRF stream with local depth ranking and spatial continuity regularization.
  • Problem Formulation: Coarse depth maps are difficult to use as absolute supervision because sensor inaccuracies and model errors make direct scaling inconsistent with NeRF geometry.These observations arise from imperfect models, coarse annotations, dataset bias, and wide-range sensor depth distances.
  • Local Depth Ranking Distillation: Local depth ranking transfers nearer-versus-farther relationships within image patches from coarse depth maps to NeRF-predicted ray depths.The ranking loss samples pixel pairs and penalizes inconsistent ordering with a margin m.
  • Spatial Continuity Distillation: Spatial continuity distillation constrains neighboring NeRF depths to remain continuous when corresponding coarse-depth pixels are continuous.Nearest neighbors are selected by depth within a small region, such as a 6 × 6 patch, with margin m′ allowing small differences.
  • Full Objective Loss: The full objective combines NeRF reconstruction, ranking regularization, and continuity regularization with weights λ and γ.The objective is L = L_nerf + λR_rank + γR_conti; the reported settings include λ = 0.2 and γ = 0.02.

4. NVS-RGBD Dataset

NVS-RGBD is a real-world dataset of coarse depth maps captured with consumer-level sensors across indoor and outdoor object-centric scenes.

  • Dataset Collection: NVS-RGBD contains real-world depth maps from Azure Kinect, ZED 2, and iPhone 13 Pro consumer-level sensors.The collection includes 8 scenes for Azure Kinect, 8 for ZED 2, and 4 for iPhone 13 Pro.
  • Dataset Collection: The dataset includes indoor scenes for Kinect and ZED 2, and both indoor and outdoor scenes for iPhone 13 Pro.All scenes are object-centric, with camera poses computed via COLMAP.
  • Sensor Artifacts: The sensors exhibit distinct artifacts: Kinect has edge noise, while ZED 2 produces smooth but unstable and incorrect depth under time jittering.These artifacts motivate evaluating methods on realistic consumer-sensor depth rather than idealized measurements.

5. Experiments

Experiments on LLFF, DTU, and NVS-RGBD evaluate SparseNeRF with three views, including comparisons, ablations, depth-model variants, and user studies. Across these evaluations, SparseNeRF achieves strong visual and quantitative performance, while depth ranking and continuity distillation improve reconstruction.

  • Experimental Setup: Experiments use LLFF, DTU, and NVS-RGBD, with three training views and per-scene optimization without pre-training on other scenes for NVS-RGBD.Training uses 90k iterations per scene, taking about two hours; uncertain Kinect and ZED 2 regions are masked.
  • Comparisons on LLFF: SparseNeRF achieves the best performance in PSNR, SSIM, and LPIPS on LLFF compared with the listed state-of-the-art methods.The comparison includes pre-trained, fine-tuned, geometry-and-semantic, and depth-knowledge-distillation groups.
  • Comparisons on DTU: SparseNeRF achieves the best performance on DTU without pre-training on other scenes and produces better visual results than previous state-of-the-art methods.Pre-trained baselines benefit from similar training and test scene distributions, while DSNeRF uses sparse COLMAP points because noisy NVS-RGBD depths are difficult to apply.
  • Comparisons on NVS-RGBD: SparseNeRF achieves better performance than RegNeRF, DSNeRF, and MonoSDF on NVS-RGBD and demonstrates qualitative superiority in Figure 7.The dataset uses consumer-level depth sensors, whose noisy maps are not directly suited to DSNeRF's depth supervision.
  • Ablation Studies: Removing local depth ranking distillation degrades PSNR, SSIM, LPIPS, and depth error, while continuity distillation further improves detailed scene coherence.The ablation separates the effects of ranking and continuity distillation in quantitative and visual evaluations.
  • Further Analyses: Depth ranking outperforms depth scaling on LLFF because linear scaling is too strong for inaccurate single-view depth, whereas ranking relaxes supervision to relative comparisons.DPT Hybrid and DPT Large perform comparably and better than MiDaS small; 87.5% of users prefer the proposed continuity term over RegNeRF's.
  • Further Analyses: SparseNeRF also shows competitive Chamfer distance results with MonoSDF and UNISURF, especially for complex scenes.FreeNeRF is identified as a potential complement to SparseNeRF.

6. Conclusion

The paper presents SparseNeRF for sparse-view novel view synthesis using coarse depth priors. Local depth ranking and spatial continuity regularization improve few-shot NeRF performance, but the method does not generalize to occluded views unobserved during training.

  • Conclusion: SparseNeRF distills local depth ranking and spatial continuity priors from coarse depth maps to address under-constrained few-shot NeRF reconstruction.The priors support sparse-view novel view synthesis and coherent scene geometry.
  • Conclusion: SparseNeRF achieves state-of-the-art performance on three datasets with coarse depth priors.The conclusion attributes the framework's performance to depth ranking and continuity regularization.
  • Limitation: SparseNeRF cannot be generalized to occluded views that are unobserved in the training views.This is the stated limitation of the method.

A1. Few-shot NeRF on iPhone

An iPhone 13 Pro RGB-D setup evaluates few-shot novel view synthesis with three views. SparseNeRF significantly improves reconstructed geometry relative to RegNeRF.

  • iPhone 13 Pro Evaluation: SparseNeRF significantly improves geometry reconstruction over RegNeRF using three aligned RGB and depth views from an iPhone 13 Pro.RegNeRF does not recover correct geometry in this application.

A2. Few-shot NeRF with Simple User-click Depth Annotations

User-click depth annotations provide sparse local depth rankings for refining coarse depth priors. Adding these annotations significantly improves few-shot NeRF performance, with 30–40 depth pairs per view often sufficient.

  • User-click Annotations: Users specify near-far depth pairs in local regions to correct incorrect rankings from pre-trained depth models or sensors.Green points mark nearer samples and red points mark farther samples in the described interface.
  • User-click Annotations: Adding user-click depth annotations significantly improves few-shot NeRF performance, and 30–40 depth pairs per view lead to significant improvement.Only partial regions may need annotation for refinement in real-world scenes.

B. Detailed Descriptions of the new Dataset NVS-RGBD

NVS-RGBD provides real-world RGBD data from consumer-level sensors, while illustrating both sensor-specific depth artifacts and SparseNeRF’s visual comparisons.

  • Azure Kinect depth maps contain substantial edge noise and masked regions, while ZED 2 maps can be smooth but unstable or incorrect.
  • NVS-RGBD contains RGBD images captured by Azure Kinect, ZED 2, and iPhone 13 Pro.
  • The dataset includes visual comparisons of SparseNeRF against other methods on LLFF and NVS-RGBD scenes.
  • iPhone depth maps are smoother than Kinect and ZED 2, but capture object distances only up to 2.5 meters.

C. More Implementation Details

The implementation compares reproduced and reimplemented depth-based baselines, including a scale-invariant depth consistency loss applied to RegNeRF for novel-view synthesis.

  • The experiments reproduce most prior methods and implement DSNeRF using its official codebase.
  • MonoSDF is adapted for novel-view synthesis by applying its scale-invariant depth consistency loss to RegNeRF.
  • MonoSDF achieves lower depth error than RegNeRF but does not achieve better novel-view RGB synthesis.

D. More Visual Comparisons

Visual comparisons across LLFF, DTU, and NVS-RGBD evaluate SparseNeRF against representative few-shot, pretrained, and depth-based methods under three-view training.

  • The supplementary comparisons cover LLFF, DTU, and NVS-RGBD, using five representative methods on LLFF and DTU.
  • MipNeRF can shift object positions because it lacks extra geometric regularization in the under-constrained few-shot setting.
  • PixelNeRF(ft) and MVSNeRF(ft) improve few-shot NeRFs through pretraining on extra scenes and target-scene fine-tuning.
  • RegNeRF and SparseNeRF do not require extra-scene pretraining, while RegNeRF adds continuous geometric constraints from unseen viewpoints.
  • SparseNeRF uses pretrained depth models on LLFF and DTU, depth sensors on NVS-RGBD, and achieves better visual effects than the compared depth-based methods and RegNeRF.

E. Visual Comparisons of Predicted Geometry

Geometry comparisons on Kinect and ZED 2 data show that SparseNeRF improves depth error and reconstructed geometry over the compared baselines.

  • 5.9×10−3 to 3.9×10−3: SparseNeRF reduces scale-invariant depth error on scenes captured by Kinect versus RegNeRF.
  • 3.5×10−3 to 2.4×10−3: SparseNeRF reduces scale-invariant depth error on scenes captured by ZED 2 versus RegNeRF.
  • The depth error measures the quality of reconstructed geometry, linking the quantitative comparisons to geometric fidelity.
  • Across qualitative comparisons, SparseNeRF yields much better geometry than RegNeRF, MonoSDF, and DSNeRF.
  • MonoSDF’s sub-optimal geometry is attributed to coarse depth maps not being linearly scalable to ground-truth depth maps.
Loading 2303.16196v2…