PaperlayerView exact revision on arXiv

Source-linked AI summary

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, Hao Su

arXiv:2103.15595v2cs.CV

TL;DR

Neural radiance field methods have required lengthy per-scene optimization for high-quality fields, limiting practicality. MVSNeRF combines multi-view cost-volume reasoning with neural volume rendering to reconstruct from three views, generalizing across diverse testing datasets and outperforming concurrent generalizable reconstruction works.

  • Problem

    High-quality neural radiance fields have typically required lengthy per-scene optimization, limiting practicality.

    Source: §1 · p. 1

  • Method

    MVSNeRF builds a plane-swept cost volume from warped image features, then uses a 3D CNN and volume rendering to reconstruct a neural radiance field.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    From only three input views, MVSNeRF generalizes across diverse testing datasets and significantly outperforms concurrent generalizable reconstruction works.

    Source: §6 · p. 8

  • Takeaways & Limitations

    MVSNeRF serves as a realistic-view reconstructor from few images or as an initialization for per-scene optimization when dense images are available.

    Source: §1 · p. 2, §6 · p. 8

    The low-resolution scene encoding makes high-frequency appearance challenging to regress from that information alone.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

We present MVSNeRF, a novel neural rendering approach that can efficiently reconstruct neural radiance fields for view synthesis. Unlike prior works on neural radiance fields that consider per-scene optimization on densely captured images, we propose a generic deep neural network that can reconstruct radiance fields from only three nearby input views via fast network inference. Our approach leverages plane-swept cost volumes (widely used in multi-view stereo) for geometry-aware scene reasoning, and combines this with physically based volume rendering for neural radiance field reconstruction. We train our network on real objects in the DTU dataset, and test it on three different datasets to evaluate its effectiveness and generalizability. Our approach can generalize across scenes (even indoor scenes, completely different from our training scenes of objects) and generate realistic view synthesis results using only three input images, significantly outperforming concurrent works on generalizable radiance field reconstruction. Moreover, if dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction with higher rendering quality and substantially less optimization time than NeRF.

1. Introduction

MVSNeRF targets the impractical per-scene optimization required by prior radiance-field methods by reconstructing fields from as few as three unstructured views for fast novel-view synthesis. It combines geometry-aware multi-view stereo with neural rendering and can be further optimized when more images are available.

1. Introduction

  • Prior NeRF works [34] [31] [27] can produce photorealistic views but require lengthy per-scene optimization, which is costly and limits practicality.MVSNeRF is proposed to enable efficient radiance-field estimation through fast network inference instead of tedious per-scene optimization.Source: §1 · pp. 1–2
  • 5–15 minutes of further optimization can let MVSNeRF outperform NeRFs [34] optimized for hours; its fine-tuned fields are comparable to or better than per-scene overfit NeRFs.The estimated three-image radiance field can serve as a strong initialization when denser images are captured.Source: §1 · p. 2
  • A 3D CNN builds a localized neural scene encoding volume from a plane-swept cost volume, and an MLP decodes density and radiance at continuous locations for differentiable rendering.The cost volume is formed by warping 2D features from nearby views onto sweeping planes in the reference view’s frustum; trilinearly interpolated voxel features feed the MLP.Source: §1 · p. 2
  • Using three input images and training on DTU, MVSNeRF synthesizes realistic views on DTU and reasonable results on differently distributed datasets, outperforming concurrent generalizable NeRF works [54] [46].The authors attribute its cross-scene generalization to explicit geometry-aware 3D structure and cross-view correspondence reasoning, unlike approaches relying mainly on 2D image features.Source: §1 · p. 2

2. Related Work

MVSNeRF combines deep MVS cost volumes with differentiable volume rendering to infer continuous neural radiance fields from a few images. It estimates fields across scenes without expensive per-scene optimization and outperforms concurrent methods that use only 2D image features.

Multi-view stereo

  • MVSNeRF combines deep MVS cost volumes with differentiable volume rendering and trains with image loss alone, enabling novel-view synthesis without direct depth supervision.Its depth reconstruction is comparable to MVSNet, despite no direct depth supervision.Source: §2 · p. 2

View synthesis

  • Prior view-synthesis methods using plane-sweep volumes typically predict colors at discrete planes and aggregate them with alpha-blending or learned weights.MPI-based methods build sweeps at reference views, while other methods construct them at novel viewpoints [55] [11] [33] [41] [15].Source: §2 · p. 2
  • Instead of predicting colors at discrete sweep planes, MVSNeRF infers per-voxel neural features and regresses volume-rendering properties at arbitrary 3D locations, forming a continuous radiance field.Differentiable, physically based volume rendering then supports realistic view synthesis.Source: §2 · pp. 2–3

Neural rendering

  • MVSNeRF uses a plane-swept 3D cost volume for geometry-aware scene understanding and generalizes radiance-field reconstruction from few input images, outperforming concurrent methods based only on 2D image features [54] [46].Unlike most prior neural-rendering work that requires expensive per-scene optimization, it estimates fields across scenes.Source: §2 · p. 3

3. MVSNeRF

MVSNeRF learns a generic radiance-field network from sparse nearby views, combining MVS-style geometry reasoning, neural encoding, and differentiable volume rendering; with dense views, its encoding can be fine-tuned per scene.

3.1. Cost volume construction.

  • MVSNeRF warps extracted features from multiple views onto reference-view sweep planes and uses their variance as a cost volume encoding geometry and view-dependent appearance.The cost volume captures appearance variation across input views, including effects from scene geometry and view-dependent shading.Source: §3.1 · p. 4

3.2. Radiance field reconstruction.

  • A 3D UNet learns an unsupervised per-voxel geometry-and-appearance encoding from the cost volume, and an MLP decodes interpolated features into density and view-dependent radiance.The decoder also receives projected input-pixel colors to help recover high-frequency appearance; NDC coordinates normalize scene scales across datasets.Source: §3.2 · pp. 4–5

3.3. Volume rendering and end-to-end training.

  • Differentiable physically based ray marching renders novel-view pixels and supplies end-to-end L2 supervision; training across DTU scenes teaches a generic function that reconstructs radiance fields on novel scenes.The rendering loss is the only supervision used for the full system.Source: §3.3 · p. 5

3.4. Optimizing the neural encoding volume.

  • At 18 minutes, fine-tuned MVSNeRF results are already much better than 215-minute NeRF results, while optimizing only the encoding volume and MLP avoids costly CNN and plane-sweep processing.Appending trainable per-view voxel colors makes the reconstruction independent of input images, which can be dropped after optimization.Source: §3.4 · pp. 5–6, Figure 3 · p. 6

4. Implementation details

The implementation specifies DTU-based training and cross-dataset evaluation, alongside network architecture, input-view selection, and optimization settings.

Implementation details

  • DTU-only training uses 88 scenes and 16 test scenes at 512 × 640; evaluation also covers Realistic Synthetic NeRF and Forward-Facing data.For each test scene, 20 nearby views are divided into 3 center input views, 13 views for per-scene fine-tuning, and 4 test views.Source: §4 · p. 6
  • The model uses 32 feature channels, 128 uniformly sampled depth hypotheses, a six-layer MLP, one radiance field, and 128 shading points per ray.The 32 channels are used in feature extraction, the cost volume, and the neural encoding volume before color channels are appended; coarse-to-fine fields could be added at fine-tuning using two encoding volumes with the same initialization.Source: §4 · p. 6
  • Across-scene training uses one RTX 2080 Ti, 1,024 randomly sampled pixels from a novel viewpoint per batch, and Adam with an initial learning rate of 5 × 10^-4.Source: §4 · p. 6

5. Experiments

With three input views, MVSNeRF generalizes across the tested datasets and outperforms concurrent methods on novel-view synthesis. Fine-tuning its predicted radiance field also enables high-quality per-scene reconstruction with much less optimization time than NeRF, while its geometry reconstruction is comparable to a supervised MVS method.

Comparisons on results with three-image input

  • MVSNeRF consistently outperforms PixelNeRF and IBRNet on PSNRs, SSIMs, and LPIPs, with significantly better results on the two non-DTU datasets.The comparisons use the same three input views across three datasets; Figure 4 shows realistic synthesis across datasets, while PixelNeRF fails on Forward-Facing and IBRNet exhibits more obvious flicker than MVSNeRF.Source: §5 · pp. 6–7, Table 1 · p. 7, Figure 4 · p. 8

Depth reconstruction

  • Without depth supervision, MVSNeRF reconstructs depth significantly more accurately than PixelNeRF and IBRNet and achieves accuracy comparable to depth-supervised MVSNet.Depth is derived from volume density as a weighted sum of sampled depth values along marched rays.Source: Table 2 · p. 7, §5 · pp. 7–8

Comparisons on results with three-image input

  • MVSNeRF uses plane-swept cost volumes for geometry-aware reasoning and encodes localized radiance fields in a volume with explicit 3D structure.The paper attributes its generalization and rendering quality to this approach rather than directly aggregating across-view 2D image features at ray-marching points.Source: §5 · p. 7

Per-scene fine-tuning results

  • 15 minutes of fine-tuning (10k iterations) yields results on par with NeRF on Realistic Synthetic and better on DTU and Forward-Facing, despite NeRF's 10.2 hours of optimization.Fine-tuning uses 16 additional input images and the network-predicted neural encoding volume with its MLP; the paper reports that this is 30 times less optimization time than NeRF.Source: §5 · p. 7, Table 1 · p. 7, Figure 4 · p. 8

6. Conclusion

MVSNeRF combines cost-volume scene reasoning from deep MVS with physically based neural volumetric rendering to reconstruct radiance fields from three views. It generalizes across diverse test datasets and can be fine-tuned with dense images for higher-quality rendering in less optimization time than NeRF.

6. Conclusion

  • From three input views, MVSNeRF produces high-quality radiance fields and realistic view synthesis, generalizes across diverse test datasets, and significantly outperforms concurrent works [54] [46].The approach is intended for generalizable radiance field reconstruction from few views.Source: §6 · p. 8
  • MVSNeRF combines deep MVS cost-volume scene reasoning with physically based neural volumetric rendering.Source: §6 · p. 8
  • When dense input images are available, its reconstruction can be fine-tuned for per-scene optimization, producing renderings better than NeRF with substantially less optimization time.The approach supports neural rendering with either few or dense input images.Source: §6 · p. 8

A. Per-scene optimization.

MVSNeRF’s predicted neural encoding volume can be fine-tuned per scene to reach results comparable to NeRF’s much longer from-scratch optimization, with further improvement possible over more iterations. Large scenes may require padding to address out-of-frustum artifacts.

A. Per-scene optimization.

  • Per-scene fine-tuning optimizes the network-predicted neural encoding volume with its MLP decoder; on one RTX 2080Ti, 15min corresponds to 10k iterations versus NeRF’s 10.2h for 200k.The neural encoding volume with the MLP is described as an effective radiance-field representation.Source: A. Per-scene optimization · p. 11
  • Because the encoding volume covers only the reference-view frustum, large scenes can show artifacts outside it; padding boundary voxels can help but may require longer fine-tuning.Padding voxels are not well reconstructed initially by the network.Source: A. Per-scene optimization · p. 11
  • Ray marching uses uniform sampling without NeRF’s two-stage coarse-and-fine sampling [34]: 128 points suffice for most scenes, while some challenging fine-tuning scenes use 256.Across-scene DTU training uses 128 points, with small per-step randomness.Source: A. Per-scene optimization · p. 11
  • 15min fine-tuning is comparable to NeRF’s 10.2h optimization and stays better through 200k iterations; at 12k iterations, the 18min result already beats NeRF after 48k.The 10.2h NeRF result corresponds to 200k iterations.Source: A. Per-scene optimization · p. 11

B. Network Architectures

The network comprises a 2D CNN for image-feature extraction, a 3D CNN for neural-encoding-volume prediction, and an MLP for volume-property regression; their detailed specifications appear in Table 3.

B. Network Architectures

  • The 2D CNN extracts image features, the 3D CNN predicts a neural encoding volume, and the MLP regresses volume properties.Table 3 presents the architecture specifications for all three models.Source: B. Network Architectures · p. 11, Table 3 · p. 11
  • Table 3 specifies layer kernel sizes, strides, dilations, channel counts, layer structures, and positional encoding (PE) as used in [34].The MLP regression model uses layers indexed i ∈ [1, ..., 5].Source: Table 3 · p. 11

C. Limitations.

MVSNeRF can require longer fine-tuning to recover strong view-dependent effects, and its single encoding volume does not directly reconstruct unseen scene regions needed for wide-range rendering.

C. Limitations.

  • Highly glossy or specular scenes can require longer fine-tuning because strong view-dependent shading is hard to recover through network inference alone.The extra fine-tuning can be required to fully reconstruct these effects.Source: C. Limitations · p. 11
  • The encoding volume initializes content seen by the reference view; padding can add out-of-frustum content, but occluded or invisible regions are not directly recovered.These uninitialized regions make it challenging for a single volume to support rendering across a wide viewing range.Source: C. Limitations · pp. 11–12
  • Dense images captured around a scene can enable 360° rendering through long per-scene fine-tuning, but it can be as slow as training standard NeRF [34] or Sparse Voxel Fields [27].Combining multiple neural encoding volumes from multiple views is proposed as a future direction for fast reconstruction over larger viewing ranges.Source: C. Limitations · p. 12

D. Per-scene breakdown.

The per-scene quantitative breakdown across Realistic Synthetic, DTU, and LLFF is consistent with the paper’s averaged results. It shows stronger generalization on novel datasets and compares MVSNeRF with PixelNeRF and IBRNet on DTU.

D. Per-scene breakdown.

  • On DTU, MVSNeRF outperforms PixelNeRF [54] with the same three-image input, achieving higher PSNR and SSIM and lower LPIPS.Because training uses DTU scenes, all three methods work reasonably well on the DTU testing set.Source: D. Per-scene breakdown · p. 12
  • On DTU, MVSNeRF’s three-image results are quantitatively comparable to IBRNet [46] with 10 images; IBRNet often has better PSNR, while MVSNeRF often has better SSIM and LPIPS.IBRNet is trained and tested with 10 input images to achieve its best performance as used in their paper.Source: D. Per-scene breakdown · p. 12
  • On the novel Synthetic Data and Forward-Facing datasets, MVSNeRF generalizes significantly better than PixelNeRF and IBRNet, with much better quantitative results.The per-scene comparisons are reported for the Realistic Synthetic and Forward Facing datasets.Source: D. Per-scene breakdown · p. 12, Table 5 · p. 13, Table 6 · p. 13

Related papers

Loading 2103.15595v2…