PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast-MVSNet: Sparse-to-Dense Multi-View Stereo With Learned Propagation and Gauss-Newton Refinement

Zehao Yu, Shenghua Gao

arXiv:2003.13017v1cs.CV

TL;DR

Fast-MVSNet targets the efficiency and memory costs of deep learning-based MVS while preserving reconstruction quality. It predicts sparse high-resolution depth, learns image-guided propagation to densify it, and applies differentiable Gauss-Newton refinement. Experiments report comparable or better results with substantially faster runtimes than Point-MVSNet and R-MVSNet.

  • Problem

    Deep learning-based MVS methods improve reconstruction quality but often require memory-intensive 3D volumes, limiting efficient high-resolution reconstruction.

  • Method

    Fast-MVSNet uses a sparse-to-dense coarse-to-fine framework with sparse high-resolution depth prediction, learned image-guided propagation, and differentiable Gauss-Newton refinement.

  • Results

    Fast-MVSNet achieves better or comparable reconstruction results while being about 5× faster than Point-MVSNet and 14× faster than R-MVSNet.

  • Takeaways & Limitations

    Sparse depth representation and lightweight modules make Fast-MVSNet efficient and memory-friendly for high-resolution MVS.

Abstract

from arXiv · show

Almost all previous deep learning-based multi-view stereo (MVS) approaches focus on improving reconstruction quality. Besides quality, efficiency is also a desirable feature for MVS in real scenarios. Towards this end, this paper presents a Fast-MVSNet, a novel sparse-to-dense coarse-to-fine framework, for fast and accurate depth estimation in MVS. Specifically, in our Fast-MVSNet, we first construct a sparse cost volume for learning a sparse and high-resolution depth map. Then we leverage a small-scale convolutional neural network to encode the depth dependencies for pixels within a local region to densify the sparse high-resolution depth map. At last, a simple but efficient Gauss-Newton layer is proposed to further optimize the depth map. On one hand, the high-resolution depth map, the data-adaptive propagation method and the Gauss-Newton layer jointly guarantee the effectiveness of our method. On the other hand, all modules in our Fast-MVSNet are lightweight and thus guarantee the efficiency of our approach. Besides, our approach is also memory-friendly because of the sparse depth representation. Extensive experimental results show that our method is 5$\times$ and 14$\times$ faster than Point-MVSNet and R-MVSNet, respectively, while achieving comparable or even better results on the challenging Tanks and Temples dataset as well as the DTU dataset. Code is available at https://github.com/svip-lab/FastMVSNet.

1. Introduction

MVS seeks dense 3D scene recovery from calibrated images, but deep methods often trade efficiency and memory for reconstruction quality. Fast-MVSNet addresses this with a sparse-to-dense, coarse-to-fine design combining lightweight propagation and Gauss-Newton refinement.

  • MVS recovers dense 3D scene structure from calibrated images and supports applications including 3D reconstruction, augmented reality, autonomous driving, and robotics.
  • Deep CNN-based MVS improves reconstruction quality, but multi-scale 3D CNNs are memory-consuming because 3D volume requirements grow cubically.
  • Fast-MVSNet predicts a sparse high-resolution depth map, propagates it to dense resolution using image-guided local dependencies, and refines it with a differentiable Gauss-Newton layer.
  • The framework uses sparse-to-dense and coarse-to-fine strategies, a small-scale propagation network, and end-to-end learnable Gauss-Newton optimization.
  • Fast-MVSNet is reported to achieve better or comparable reconstruction results while being more efficient and memory-friendly than competing methods.

2. Related Work

MVS methods use volume, point-cloud, or depth-map representations, with learned matching and regularization improving reconstruction. Fast-MVSNet organizes sparse depth prediction, learned propagation, and Gauss–Newton refinement into three stages.

  • Scene representations: Volumetric MVS discretizes 3D space into regular grids, but its high memory consumption limits scalability to large-scale scenarios.
  • Scene representations: Point-cloud methods begin with sparse matched keypoints and densify them through propagation, but sequential propagation makes parallelization difficult.
  • Scene representations: Depth maps reduce reconstruction to per-view depth estimation and can be fused into point clouds or volumetric representations.
  • Learning-based MVS: Deep CNNs have been used to learn patch similarity for matching and to regularize cost volumes constructed from multi-view images.
  • Fast-MVSNet: Fast-MVSNet predicts a sparse high-resolution depth map from a sparse cost volume, propagates it to dense depth, and refines it with a differentiable Gauss–Newton layer.
  • Upsampling and propagation: Learned propagation replaces hand-crafted upsampling strategies by using image guidance and data-adaptive depth propagation.
  • Learned optimization: Differentiable optimization methods unroll iterative nonlinear least-squares algorithms into neural-network layers, including learned Levenberg–Marquardt updates.

3. Method

Fast-MVSNet estimates depth with a sparse-to-dense coarse-to-fine pipeline: sparse high-resolution prediction, learned propagation, and differentiable Gauss-Newton refinement. The design targets efficient, memory-conscious inference while retaining high-resolution details.

  • Fast-MVSNet estimates a per-view depth map using a sparse-to-dense coarse-to-fine strategy.It first predicts sparse high-resolution depth, then densifies and refines it.
  • Sparse high-resolution depth map prediction: The sparse high-resolution representation reduces memory and computation costs while preserving details that low-resolution depth maps may lose.Low-resolution alternatives can misalign with extracted features or introduce discontinuity artifacts during downsampling.
  • Sparse high-resolution depth map prediction: A sparse cost volume is constructed and regularized with 3D CNNs to predict the sparse depth map through differentiable argmax.The framework can also accommodate alternative regularizers such as Convolutional GRU.
  • Depth map propagation: The propagation module densifies the sparse map by learning local weights from the reference image with a CNN.This replaces manually tuned bilateral-filter kernels with data-driven, position-dependent weights.
  • Gauss-Newton refinement: The Gauss-Newton layer refines depth by minimizing multi-view feature inconsistency from image features, camera parameters, and an initial depth map.It is differentiable, has no additional learnable parameters, supports end-to-end training, and converges quickly with one update step.
  • Gauss-Newton refinement: Training supervises both the initial and refined depth maps with mean absolute error on valid ground-truth depths.The two losses are balanced with λ = 1.0 in all experiments.

4. Experiments

Experiments on DTU and Tanks and Temples evaluate reconstruction quality, efficiency, memory use, and component effectiveness. Fast-MVSNet achieves strong reconstruction results while remaining efficient and memory-friendly.

  • DTU dataset: The DTU dataset contains 80 scenes captured from 49 or 64 camera positions under seven lighting conditions.
  • DTU results: Fast-MVSNet outperforms competing methods in DTU reconstruction completeness and overall quality, although Gipuma achieves the best Accuracy.
  • Efficiency comparison: Fast-MVSNet outperforms all compared methods across reconstruction quality, depth resolution, GPU memory, and runtime on DTU.
  • Ablation study: Adding propagation and Gauss-Newton refinement produces denser reconstructions with finer details, while Gauss-Newton refinement improves Overall reconstruction quality by 9.5%.The Overall value changes from 0.409 to 0.370.
  • Generalization: On Tanks and Temples, a model trained on DTU without finetuning achieves results comparable to state-of-the-art methods and produces dense, visually appealing reconstructions.

5. Conclusion

Fast-MVSNet is an efficient MVS framework that combines sparse-to-dense and coarse-to-fine depth estimation. Experiments on two challenging datasets verify its effectiveness and efficiency.

  • Fast-MVSNet first predicts a sparse high-resolution depth map, propagates it densely, and then optimizes it with a differentiable Gauss-Newton layer.

A. Architecture

The architecture comprises sparse high-resolution depth prediction, depth propagation, and Gauss-Newton refinement. Its propagation network predicts local depth dependencies from image features.

  • Fast-MVSNet has three main components: sparse high-resolution depth prediction, depth map propagation, and Gauss-Newton refinement.
  • Sparse depth prediction: The sparse prediction stage builds a sparse spatial cost volume and uses fewer virtual depth planes than the MVSNet-style design.
  • Depth propagation: The propagation module uses a small convolutional network to predict weights for densifying the sparse depth map, with Conv and BR layers specified in Table 5.BR abbreviates batch normalization and ReLU; K, S, and F denote kernel size, stride, and output channels.
  • Gauss-Newton refinement: For Gauss-Newton refinement, Conv 4 and Conv 7 are interpolated to a common size and concatenated as deep feature representations.

B. Depth maps fusion

Depth fusion filters predictions photometrically and geometrically before averaging depths visible across multiple views. Fusion parameters create an Accuracy–Completeness trade-off.

  • Photometric filtering: Photometric filtering removes points whose interpolated sparse-depth probability is below the threshold 0.5.
  • Geometric consistency: Geometric consistency filtering removes points whose depth-map discrepancy exceeds the threshold η = 0.12 pixels.
  • Depth fusion: Depth fusion requires each point to be visible in V = 3 views and averages all reprojected depths.
  • Parameter trade-offs: Increasing η makes reconstructions less accurate but more complete, whereas increasing V makes them more accurate but incomplete.
  • Parameter trade-offs: Because fusion parameters significantly affect reconstruction results, the authors identify integrating a learnable fusion module as future work.

C. Gauss-Newton refinement with more iterations

Gauss-Newton refinement significantly improves reconstruction quality, but additional iterations provide only marginal gains, motivating a single iteration.

  • Gauss-Newton refinement significantly improves reconstruction quality on the DTU evaluation dataset.
  • Additional Gauss-Newton iterations yield only marginal performance improvements.
  • The evaluation compares reconstruction quality across different Gauss-Newton iteration numbers.

D. Reconstruction results

Fast-MVSNet produces dense and accurate reconstructions across all scenes evaluated on the DTU dataset.

  • Fast-MVSNet produces dense reconstructions across all DTU scenes shown in Figure 9.
  • Fast-MVSNet produces accurate reconstructions across all DTU scenes shown in Figure 9.
  • Figure 9 presents additional Fast-MVSNet reconstruction results on the DTU dataset.
Loading 2003.13017v1…