PaperlayerView exact revision on arXiv

Source-linked AI summary

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, Hao Su

arXiv:2103.15595v2cs.CV

TL;DR

Prior neural radiance-field methods rely on expensive per-scene optimization, motivating a more practical reconstruction approach. MVSNeRF uses geometry-aware plane-swept cost volumes and neural rendering to infer radiance fields from three nearby views. It generalizes across datasets for realistic novel-view synthesis, while dense-view fine-tuning can improve quality with substantially less optimization time than NeRF.

  • Problem

    Prior neural radiance-field methods require long per-scene optimization on densely captured images, limiting practicality.

  • Method

    MVSNeRF uses plane-swept cost volumes, 3D CNNs, neural encoding volumes, MLP decoding, and differentiable ray marching to reconstruct radiance fields from sparse views.

  • Results

    MVSNeRF generalizes across diverse datasets and produces realistic novel-view synthesis from only three input images, outperforming concurrent generalizable radiance-field methods.

  • Takeaways & Limitations

    The method serves as a fast radiance-field reconstructor for few images and as an initializer for efficient per-scene optimization when dense images are available.

  • Takeaways & Limitations

    A single neural encoding volume struggles with wide viewing ranges such as 360° rendering because views outside the input range are not directly recovered.

Abstract

from arXiv · show

We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference. Our approach leverages plane-swept cost volumes (widely used in multi-view stereo) for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction. We train our network on real objects in the DTU dataset, and test it on three different datasets to evaluate its effectiveness and generalizability. Our approach can generalize across scenes (even indoor scenes, completely different from our training scenes of objects) and generate realistic view synthesis results using only three input images, significantly outperforming concurrent works on generalizable radiance field reconstruction. Moreover, if dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction with higher rendering quality and substantially less optimization time than NeRF.

1 ShanghaiTech University 2 Adobe Research 3 University of California, San Diego

MVSNeRF generalizes from DTU object scenes to complex indoor scenes using three input images, with quality improved by brief fine-tuning. The work includes a remote-internship contribution by Anpei Chen.

  • MVSNeRF reconstructs a neural radiance field for a complex indoor scene from only three input images.
  • Six minutes of fine-tuning on more images largely improves artifacts in the reconstruction.
  • Anpei Chen’s research was conducted during a remote internship with UCSD.

1. Introduction

MVSNeRF addresses the costly per-scene optimization required by prior neural radiance fields through a generalizable network that reconstructs radiance fields from as few as three views. It combines geometry-aware MVS reasoning with neural rendering, supports realistic novel-view synthesis across scenes, and can be fine-tuned efficiently when more images are available.

  • Prior neural radiance-field methods require long per-scene optimization, limiting their practicality.
  • MVSNeRF reconstructs radiance fields from as few as three unstructured multi-view images through fast network inference.
  • The method builds a cost volume from warped 2D features, then predicts a neural encoding volume for density and radiance estimation.
  • MVSNeRF combines learning-based MVS with neural rendering to support cross-view correspondence reasoning and generalization to unseen scenes.
  • With dense images, the estimated radiance field can be fine-tuned to produce photo-realistic results comparable to or better than per-scene NeRF optimization with substantially less time.

2. Related Work

MVSNeRF connects deep multi-view stereo and neural rendering by replacing direct depth or per-plane color prediction with a continuous neural radiance-field representation. Its framework uses cost volumes, neural encoding volumes, and differentiable volume rendering to support efficient view synthesis.

  • Multi-view stereo: Deep MVS methods use 3D convolutions on plane-swept cost volumes for dense geometry reconstruction and depth estimation.
  • View synthesis: Prior view-synthesis methods use plane-sweep volumes, including MPI-based approaches at reference views and methods constructing plane sweeps at novel viewpoints.
  • MVSNeRF: MVSNeRF constructs a cost volume by warping 2D features, applies a 3D CNN to form a neural encoding volume, and uses an MLP with ray marching for rendering.
  • MVSNeRF: Unlike direct per-plane color prediction, MVSNeRF infers per-voxel neural features and regresses rendering properties at arbitrary 3D locations.
  • Neural rendering: NeRF combines MLPs with differentiable volume rendering to achieve photo-realistic view synthesis.

3. MVSNeRF

MVSNeRF uses a generic, geometry-aware network to reconstruct a neural radiance field from sparse multi-view images and render novel views. Its encoding volume can also be rapidly fine-tuned for higher-quality scene-specific reconstruction.

  • Volume rendering and end-to-end training: Differentiable ray marching renders novel images and supplies the rendering loss used to train the entire network across scenes.This training propagates appearance and correspondence information through the network while avoiding 3D supervision.
  • MVSNeRF: MVSNeRF reconstructs radiance fields from a few input images using a generic network rather than per-scene network memorization.The framework is designed for sparse, nearby, unstructured views and uses three views in practice.
  • Cost volume construction: The method builds a reference-view plane-sweep cost volume by warping CNN features from nearby views across multiple depths.Camera parameters define homographic warping into the reference view, and variance across views forms the cost feature.
  • Radiance field reconstruction: A 3D CNN converts the cost volume into a neural scene encoding volume whose features represent scene geometry and appearance.The encoding volume is learned without 3D supervision through end-to-end volume-rendering training.
  • Radiance field reconstruction: The low-resolution encoding volume makes high-frequency appearance difficult to regress directly, motivating the use of original image pixels and later fine-tuning.The paper identifies this as a limitation of the downsampled 2D feature representation.
  • Radiance field reconstruction: An MLP maps arbitrary 3D locations and viewing directions, with sampled input colors, to volume density and view-dependent radiance.The encoding volume and decoder form a standalone neural radiance-field representation that supports volume rendering.
  • Optimizing the neural encoding volume: Fine-tuning only the encoding volume and MLP avoids repeated CNN processing and substantially reduces optimization time relative to optimizing NeRF from scratch.With dense images, this procedure produces photo-realistic results after a short optimization period and can discard input images afterward.

4. Implementation details

The implementation trains MVSNeRF on DTU and evaluates it across scenes with sparse inputs, additional fine-tuning views, and held-out test views.

  • Dataset: Training uses 88 DTU scenes, while 16 DTU scenes are reserved for testing at 512 × 640 image resolution.The model is additionally tested on Realistic Synthetic NeRF and Forward-Facing data with different scene and view distributions.
  • Dataset: Each test scene provides 20 nearby views: three center views for input, 13 additional views for fine-tuning, and four views for testing.This split supports both sparse reconstruction and dense-image per-scene optimization experiments.
  • Rendering: The rendering procedure samples 128 shading points along each marching ray.The detailed network structure is provided in the supplementary materials.
  • Training: Across-scene training runs on one RTX 2080 Ti GPU with 1,024 randomly sampled pixels per batch and Adam at an initial learning rate of 5e-4.The batch samples pixels from one novel viewpoint.

5. Experiments

MVSNeRF is evaluated against concurrent generalizable radiance-field methods across three datasets, using three input views and additional per-scene fine-tuning experiments. It achieves stronger rendering quality and generalization, while producing high-quality geometry and fast fine-tuned results.

  • Comparisons with three-image input: Three-image evaluations compare MVSNeRF, PixelNeRF, and IBRNet across three datasets using identical input views and four additional test images.Quantitative results use PSNR, SSIM, and LPIPS; visual comparisons are reported in Figure 4.
  • Comparisons with three-image input: MVSNeRF produces realistic view synthesis from three images and generalizes from DTU training scenes to datasets with different scene and view distributions.PixelNeRF shows artifacts or fails on the other datasets, while IBRNet exhibits more visible flicker artifacts than MVSNeRF.
  • Comparisons with three-image input: MVSNeRF consistently outperforms PixelNeRF and IBRNet on PSNR, SSIM, and LPIPS, with especially large advantages on the two non-DTU datasets.The paper attributes this to geometry-aware reasoning in plane-swept cost volumes and a localized neural encoding volume with explicit 3D structure.
  • Per-scene fine-tuning: MVSNeRF fine-tunes its predicted neural encoding volume for 15 minutes using 16 additional input images, producing photo-realistic results after 10k iterations.Figure 4 compares these results with NeRF optimized for 10.2 hours.
  • Depth reconstruction: Depth reconstruction evaluates volume-density-derived geometry against PixelNeRF, IBRNet, and MVSNet on the DTU testing set.The supplied passage introduces the evaluation and its weighted-depth procedure but does not include the numerical depth results.

6. Conclusion

MVSNeRF combines deep multi-view stereo with neural rendering to reconstruct high-quality radiance fields from few views and generalize across datasets. With dense images, its representation can also be fine-tuned for photo-realistic rendering using substantially less optimization time than NeRF.

  • MVSNeRF combines cost-volume-based scene reasoning with physically based neural volumetric rendering for high-quality radiance-field reconstruction.
  • The method reconstructs radiance fields from only three input views and generalizes across diverse testing datasets for realistic view synthesis.
  • With dense input images, MVSNeRF can be fine-tuned to achieve photo-realistic renderings better than NeRF while using substantially less optimization time.

A. Per-scene optimization.

Per-scene fine-tuning optimizes MVSNeRF’s predicted neural encoding volume and MLP, leveraging a strong initialization for rapid quality gains. The representation is limited to the reference-view frustum, so large scenes may require boundary padding and longer optimization.

  • Optimization setup: Fine-tuning optimizes the predicted neural encoding volume together with the MLP decoder for each scene.The neural encoding volume with the MLP serves as the radiance-field representation.
  • Scope and artifacts: The neural encoding volume covers only the reference-view frustum, so large scenes can exhibit artifacts when out-of-frustum content appears in novel views.Boundary padding can address these artifacts, but padded regions may require longer fine-tuning because they are initially poorly reconstructed.
  • Sampling: The method normally samples 128 points per ray and increases sampling to 256 points for some challenging fine-tuning scenes.
  • Optimization progress: MVSNeRF fine-tuning remains better than NeRF’s from-scratch optimization through 200k iterations and can improve further with continued optimization.The strong predicted initialization makes the early fine-tuning results competitive with or better than much longer NeRF optimization.

B. Network Architectures

The network architecture comprises 2D feature extraction, 3D neural encoding-volume prediction, and MLP volume-property regression. Table 3 specifies the layer configurations and positional encoding used by these components.

  • Table 3 specifies the 2D CNN, 3D CNN, and MLP components used for feature extraction, neural encoding-volume prediction, and volume-property regression.
  • The architecture table records kernel size, stride, dilation, and input/output channel counts for each layer.
  • CBR2D, CBR3D, CTB3D, and LR denote ConvBnReLU2D, ConvBnReLU3D, ConvTransposeBn3D, and LinearRelu layer structures, respectively.
  • PE denotes positional encoding as used in NeRF.

C. Limitations.

MVSNeRF reconstructs radiance fields quickly on diverse real scenes, but its direct reconstruction is limited by view-dependent effects and incomplete coverage outside visible reference-view content.

  • C. Limitations.: Highly glossy or specular scenes can require longer fine-tuning because strong view-dependent shading is difficult to recover through network inference.The limitation concerns recovering view-dependent appearance rather than general radiance-field reconstruction speed.
  • C. Limitations.: The reconstructed representation primarily covers content seen by the reference view, leaving unseen or occluded regions unrecovered by the network.Volume padding can add out-of-frustum content, but it does not directly recover unseen regions.
  • C. Limitations.: A single neural encoding volume is challenging to use for 360° rendering, although dense-image fine-tuning can recover uninitialized regions at optimization costs comparable to standard NeRF.Multiple encoding volumes from different views are suggested as a future direction for larger viewing ranges.

D. Per-scene breakdown.

Per-scene results are consistent with the averaged evaluation: MVSNeRF performs strongly on DTU and generalizes substantially better than the comparison methods on novel datasets.

  • D. Per-scene breakdown.: MVSNeRF outperforms PixelNeRF on DTU testing scenes with the same three-image input, achieving higher PSNR and SSIM and lower LPIPS.The breakdown covers Realistic Synthetic, DTU, and LLFF datasets.
  • D. Per-scene breakdown.: MVSNeRF with three input images is quantitatively comparable to IBRNet with ten on DTU, trading lower PSNR in some scenes for better SSIM and LPIPS in others.IBRNet is trained and tested with ten input images for its best performance in the cited setup.
  • D. Per-scene breakdown.: On novel datasets, MVSNeRF generalizes significantly better than PixelNeRF and IBRNet, producing much better quantitative results on Synthetic Data and Forward-Facing scenes.The cited comparisons correspond to the Realistic Synthetic and Forward-Facing evaluations.
Loading 2103.15595v2…