PaperlayerView exact revision on arXiv

Source-linked AI summary

MVSNet: Depth Inference for Unstructured Multi-view Stereo

Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, Long Quan

arXiv:1804.02505v2cs.CV

TL;DR

MVSNet targets incomplete and difficult multi-view reconstruction, especially under low texture, specularity, reflection, and arbitrary camera geometry. It builds camera-frustum cost volumes with differentiable homography warping and regularizes them for reference-image depth inference. The method achieves strong DTU completeness and overall quality, ranks first on Tanks and Temples without fine-tuning, and is substantially faster than prior methods.

  • Problem

    Multi-view stereo must recover dense geometry from overlapping images, but difficult regions and arbitrary camera geometries limit completeness and make learned two-view extensions non-trivial.

  • Method

    MVSNet extracts image features, uses differentiable homography warping to build a camera-frustum cost volume, applies 3D convolutions for depth regression, and refines the result with the reference image.

  • Results

    MVSNet outperforms competing methods in DTU completeness and overall quality, ranks first on Tanks and Temples without fine-tuning, and runs several times or orders of magnitude faster than prior state-of-the-art methods.

  • Takeaways & Limitations

    Depth-map inference with camera-aware cost volumes provides a flexible MVS framework that generalizes from indoor DTU to outdoor Tanks and Temples without fine-tuning.

  • Takeaways & Limitations

    Incomplete ground-truth meshes can falsely render occluded or background triangles as valid training pixels, and missing complete surfaces prevent correct identification of pixels occluded in all other views.

Abstract

from arXiv · show

We present an end-to-end deep learning architecture for depth map inference from multi-view images. In the network, we first extract deep visual image features, and then build the 3D cost volume upon the reference camera frustum via the differentiable homography warping. Next, we apply 3D convolutions to regularize and regress the initial depth map, which is then refined with the reference image to generate the final output. Our framework flexibly adapts arbitrary N-view inputs using a variance-based cost metric that maps multiple features into one cost feature. The proposed MVSNet is demonstrated on the large-scale indoor DTU dataset. With simple post-processing, our method not only significantly outperforms previous state-of-the-arts, but also is several times faster in runtime. We also evaluate MVSNet on the complex outdoor Tanks and Temples dataset, where our method ranks first before April 18, 2018 without any fine-tuning, showing the strong generalization ability of MVSNet.

1 Introduction

MVSNet addresses the difficulty of applying learned stereo methods to unstructured multi-view images by building camera-aware cost volumes for reference-image depth inference. On DTU and Tanks and Temples, it reports strong completeness, overall quality, speed, and generalization without fine-tuning.

  • Traditional MVS methods struggle with dense matching in low-textured, specular, and reflective regions, leaving reconstruction completeness limited.They rely on hand-crafted similarity metrics and engineered regularization.
  • Directly extending learned two-view stereo is non-trivial because MVS images can have arbitrary camera geometries and pairwise merging underuses multi-view information.Pre-rectifying image pairs and merging reconstructions leads to less accurate results.
  • MVSNet uses differentiable homography warping to encode camera geometry, build camera-frustum cost volumes, and enable end-to-end reference-image depth inference.A variance-based metric combines arbitrary numbers of source-image features into one cost feature.
  • MVSNet outperforms competing methods in completeness and overall quality on DTU with simple post-processing.The reported comparison is across all competing methods on these two criteria.
  • MVSNet ranks first on Tanks and Temples before April 18, 2018 without fine-tuning and runs several times or even orders of magnitude faster than prior state-of-the-art methods.The comparison includes open-source MVS methods and commercial software.

2 Related work

MVS methods use point-cloud, volumetric, or depth-map representations, each trading off flexibility, scalability, memory, or processing speed. MVSNet follows depth-map inference while using a camera-frustum cost volume and image-guided refinement.

  • Point-cloud methods progressively densify propagated points, making them difficult to fully parallelize and typically slow to process.
  • Volumetric methods classify surface occupancy in regular grids but incur space discretization error and high memory consumption.
  • Depth-map representations decouple MVS into per-view estimation problems using one reference and several source images, and can be fused into point clouds or volumetric reconstructions.
  • MVSNet constructs its cost volume through 2D feature extraction and differentiable homography warping, then regresses and reference-image-refines the final depth map.
  • Unlike SurfaceNet and LSM, MVSNet focuses on one reference-image depth map at a time, allowing adaptive reconstruction of large scenes directly.SurfaceNet and LSM are restricted to small-scale or low-resolution settings.

3 MVSNet

MVSNet extracts shared deep features from multiple images before matching them in a camera-aware cost-volume pipeline. Its feature design preserves contextual information despite spatial downsampling.

  • MVSNet’s architecture follows camera geometry and previous MVS approaches, with the full network organized around the pipeline shown in Fig. 1.
  • The network begins by extracting deep feature maps from the N input images for dense matching.
  • An eight-layer shared 2D CNN extracts dense-matching features at three scales, using downsampling at layers 3 and 6.Convolutional layers use batch normalization and ReLU except for the last layer.
  • The feature extractor applies shared parameters across all feature towers for efficient learning.
  • Feature maps contain 32 channels and are downsampled by four in each image dimension, while neighboring context is encoded in each remaining pixel descriptor.This preserves useful context for dense matching despite reduced spatial resolution.

3.2 Cost Volume

MVSNet constructs a 3D cost volume on the reference camera frustum by warping source features into frontoparallel planes. A variance-based metric aggregates arbitrary numbers of views, after which 3D CNN regularization refines the cost volume.

  • Cost Volume: The cost volume is built on the reference camera frustum rather than a regular 3D grid.This representation targets depth-map inference for one reference image.
  • Cost Volume: Differentiable homography warping maps feature volumes onto frontoparallel depth planes using camera geometry.Bilinear interpolation makes the warping operation differentiable and supports end-to-end training.
  • Cost Volume: The variance-based cost metric maps multiple feature volumes into one cost volume while treating views symmetrically.It explicitly measures multi-view feature differences and adapts to arbitrary numbers of input views.
  • Cost Volume: The raw cost volume is regularized by a multi-scale 3D CNN to produce a probability volume for depth inference.The regularizer is designed to address noise from non-Lambertian surfaces and occlusions.

Cost Volume Regularization

The probability volume produced during regularization supports both depth estimation and confidence measurement.

  • Cost Volume Regularization: The probability volume is used for per-pixel depth estimation and for measuring estimation confidence.Its probability distributions also support an outlier-filtering strategy.

3.3 Depth Map

MVSNet retrieves the depth map from the regularized probability volume using a differentiable expectation-based alternative to winner-take-all selection.

  • Depth Map: Pixel-wise winner-take-all, or argmax, is the simplest way to retrieve a depth map from the probability volume.The method subsequently motivates a differentiable alternative because argmax cannot provide sub-pixel estimates or support back-propagation.

Initial Estimation

The initial depth estimate is obtained by a probability-weighted sum over depth hypotheses, while the distribution itself indicates estimation quality. Scattered distributions identify falsely matched pixels and support probability-based filtering.

  • Initial Estimation: The expectation value computes depth as a probability-weighted sum over all depth hypotheses.This is the soft argmin operation used instead of non-differentiable argmax.
  • Initial Estimation: The soft argmin is differentiable, approximates argmax, and produces continuous depth estimates from uniformly sampled hypotheses.The resulting depth map is downsampled by four in each dimension relative to the input images.
  • Initial Estimation: Scattered depth-direction probability distributions indicate falsely matched pixels and lower estimation quality.Concentrated distributions correspond to more reliable estimates, whereas outlier distributions do not form a single peak.
  • Initial Estimation: MVSNet measures estimation quality by summing probabilities over the four nearest depth hypotheses.The paper uses this probability sum for outlier filtering and reports no significant improvement from standard deviation or entropy measurements.

Probability Map

MVSNet refines the initial depth map with reference-image guidance to address oversmoothed reconstruction boundaries, while training losses supervise both initial and refined estimates.

  • Probability Map: Reference-image guidance refines the depth map because large-receptive-field regularization can oversmooth reconstruction boundaries.The motivation is compared with boundary problems in semantic segmentation and image matting.
  • Probability Map: Training uses mean absolute depth differences for both the initial and refined depth maps.Only pixels with valid ground-truth depth labels contribute to the loss.
  • Probability Map: The training-loss weight λ is set to 1.0 in the experiments.

4 Implementations

The implementation constructs training depth maps and samples, processes multi-view inputs under hardware and resolution constraints, filters unreliable depths, and fuses consistent estimates into point clouds.

  • 4 Implementations: DTU ground-truth point clouds are converted into mesh surfaces with screened Poisson reconstruction to generate depth maps.The dataset contains more than 100 scenes under different lighting conditions.
  • 4 Implementations: 27097 training samples are formed by using each image as the reference across 49-image scans with seven lighting conditions.
  • 4 Implementations: N = 3 training inputs comprise one reference image and two source images, with image-pair scores based on sparse-track baseline angles.The score uses a piecewise Gaussian favoring a target baseline angle.
  • 4 Implementations: The baseline-angle parameters θ0, σ1, and σ2 are set to 5, 1, and 10, respectively.
  • 4 Implementations: Images are resized from 1600×1200 to 800×600 and center-cropped to W = 640 and H = 512 for divisibility and GPU-memory constraints.
  • 4 Implementations: Depth filtering removes outliers using photometric consistency and geometric consistency across views.Pixels with probability below 0.8 are treated as outliers, while geometric consistency reprojects depths between views.
  • 4 Implementations: Visibility-based fusion integrates depth maps from different views into a unified point cloud while minimizing depth occlusions and cross-view violations.Visible-view depths are averaged before reprojection into space.

5 Experiments

MVSNet is evaluated on DTU and Tanks and Temples, with ablations examining input views, image features, cost metrics, refinement, efficiency, and training constraints. It achieves strong completeness and overall quality, generalizes without fine-tuning, and runs substantially faster than several baselines.

  • Benchmarking on DTU dataset: MVSNet outperforms all methods in DTU completeness and overall quality, although Gipuma has the best accuracy.Overall quality averages mean accuracy and mean completeness.
  • Benchmarking on DTU dataset: MVSNet produces the most complete DTU point clouds, especially in textureless and reflected regions.These regions are described as especially difficult for MVS reconstruction.
  • Generalization on Tanks and Temples dataset: MVSNet ranks first on the Tanks and Temples intermediate set without fine-tuning, despite training on the different indoor DTU dataset.The benchmark ranking is reported as of April 18, 2018.
  • Ablations: Testing with five input views lowers validation loss relative to three views, even though the model was trained with three views.The ablation tests N = 2, 3, and 5 and supports flexible input-view settings.
  • Ablations: Learned 2D image features outperform a single convolutional layer, while the variance cost metric converges faster and reaches lower validation loss than the mean metric.The cost-metric comparison replaces variance with mean operation for training a new model.
  • Ablations: Depth refinement raises DTU f-score from 75.58 to 75.69 at <1mm and from 79.98 to 80.25 at <2mm.The refinement has little effect on validation loss but improves evaluation results.
  • Efficiency: MVSNet reconstructs one scan in around 230 seconds, approximately 5× faster than Gipuma, 100× faster than COLMAP, and 160× faster than SurfaceNet.The reported per-view runtime is 4.7 seconds, using the same machine for all methods.
  • Implementation constraints: Training and validation on DTU fit on an 11 GB GTX 1080 Ti, whereas original-resolution Tanks and Temples reconstruction uses a 16 GB Tesla P100.The memory requirement depends on input image size and the number of depth samples.

6 Conclusion

MVSNet infers reference-image depth from unstructured images end to end by encoding camera parameters through differentiable homography-based cost-volume construction. On DTU it outperforms previous methods and runs several times faster, while achieving state-of-the-art Tanks and Temples results without fine-tuning.

  • MVSNet takes unstructured images as input and infers a depth map for the reference image end to end.
  • Differentiable homography encoding places camera parameters into the cost volume on the camera frustum, bridging 2D feature extraction and 3D cost regularization.
  • On the DTU dataset, MVSNet significantly outperforms previous methods and is several times faster in speed.
  • On Tanks and Temples, MVSNet achieves state-of-the-art results without fine-tuning, demonstrating strong generalization ability.
Loading 1804.02505v2…