Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference

Yao Yao; Zixin Luo; Shiwei Li; Tianwei Shen; Tian Fang; Long Quan

Source-linked AI summary

Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference

Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, Long Quan

arXiv:1902.10556v1Feb 27, 2019cs.CV

TL;DR

Learned MVS is limited by the memory cost of regularizing 3D cost volumes, restricting high-resolution reconstruction. R-MVSNet sequentially regularizes 2D cost maps along depth with a convolutional GRU, reducing memory growth while retaining benchmark quality and supporting large-scale scenes. The method remains limited by input image resolution.

Problem
Learned MVS approaches use memory-expensive 3D cost-volume regularization, making high-resolution and wide-depth-range reconstruction difficult.
Method
R-MVSNet sequentially regularizes the cost volume through the depth direction using a convolutional GRU instead of 3D CNNs.
Results
R-MVSNet achieves benchmark results comparable to or better than MVSNet and applies to large-scale reconstructions that previous learned approaches cannot handle.
Takeaways & Limitations
Sequential recurrent regularization reduces online memory growth from cubic to quadratic in model resolution and enables high-resolution, wide-depth-range reconstruction.
Takeaways & Limitations
R-MVSNet remains limited by input image resolution, handling up to 3072 × 2048 on an 11GB GPU and not covering the ETH3D high-resolution benchmark.

Abstract

from arXiv · show

Deep learning has recently demonstrated its excellent performance for multi-view stereo (MVS). However, one major limitation of current learned MVS approaches is the scalability: the memory-consuming cost volume regularization makes the learned MVS hard to be applied to high-resolution scenes. In this paper, we introduce a scalable multi-view stereo framework based on the recurrent neural network. Instead of regularizing the entire 3D cost volume in one go, the proposed Recurrent Multi-view Stereo Network (R-MVSNet) sequentially regularizes the 2D cost maps along the depth direction via the gated recurrent unit (GRU). This reduces dramatically the memory consumption and makes high-resolution reconstruction feasible. We first show the state-of-the-art performance achieved by the proposed R-MVSNet on the recent MVS benchmarks. Then, we further demonstrate the scalability of the proposed method on several large-scale scenarios, where previous learned approaches often fail due to the memory constraint. Code is available at https://github.com/YoYo000/MVSNet.

1. Introduction

Learned MVS methods improve reconstruction quality but struggle to scale because 3D cost-volume regularization has cubic memory growth. R-MVSNet addresses this by sequentially regularizing cost maps with a convolutional GRU, reducing memory growth to quadratic and enabling high-resolution reconstruction.

3D CNN cost-volume regularization is extremely memory-expensive because its requirement grows cubically with model resolution.
Existing scalable strategies, including octrees, divide-and-conquer processing, and per-view cost volumes, can fail or become slow for high-resolution reconstruction.
R-MVSNet sequentially regularizes the cost volume along depth with a convolutional GRU instead of using 3D CNNs.
The sequential design reduces online memory requirements from cubic to quadratic in model resolution and supports unlimited depth-wise resolution.
R-MVSNet produces results comparable to or better than MVSNet on DTU, Tanks and Temples, and ETH3D, while targeting large-scale scenes.

3. Network Architecture

R-MVSNet extends MVSNet by sequentially regularizing depth-indexed cost maps with a convolutional GRU, replacing full 3D CNN regularization. It produces probability-based depth estimates while reducing test-time memory requirements.

MVSNet review: R-MVSNet extracts image features, warps them into the reference camera frustum, and builds cost maps across depth planes.
Recurrent regularization: The network sequentially processes cost maps along the depth direction, letting each regularized map depend on the current map and preceding maps.
Recurrent regularization: The convolutional GRU combines 2D spatial convolutions with depth-wise temporal context to regularize the cost maps.
Recurrent regularization: A 3-layer stacked GRU maps 32-channel cost maps through 16, 4, and 1 output channels before softmax generates the probability volume.
Training and inference: Training uses inverse-depth sampling and classification with cross-entropy against a one-hot occupancy volume, while testing retrieves depths sequentially from regularized maps.

4. Reconstruction Pipeline

The reconstruction pipeline selects source views and depth samples, predicts a depth map, then refines it through multiview photo-consistency before filtering and fusing depth maps into a point cloud.

Input preparation: The pipeline first prepares neighboring source images, a sparse-point-cloud-derived depth range, and inverse-depth samples.Source images are ranked by baseline-angle scores, while COLMAP determines the depth range.
Depth estimation: Winner-take-all selection retrieves a depth map from the regularized cost maps, but lacks sub-pixel accuracy.The limitation motivates refinement within a small depth range.
Variational refinement: The refinement projects source images into the reference view and minimizes reprojection error using photometric and smoothness terms.ZNCC measures photo-consistency, while bilateral squared depth differences regularize neighboring depths.
Variational refinement: The initial R-MVSNet depth map already achieves a satisfying result, while variational refinement fine-tunes depths within a small range for sub-pixel accuracy.The refinement is analogous to quadratic interpolation in stereo methods and DenseCRF in DeepMVS.
Point-cloud reconstruction: Finally, filtered depth maps are fused into a single 3D point cloud using photometric and geometric consistency checks.The selected depth probability supplies the confidence measurement for filtering.

5. Experiments

Experiments show that R-MVSNet achieves strong benchmark performance while scaling to wide-depth-range and high-resolution reconstructions with lower memory requirements. Ablations and runtime measurements further characterize its design and practical limits.

Benchmark Results: R-MVSNet produces results comparable to or better than MVSNet across DTU, Tanks and Temples, and ETH3D evaluations.It achieves the best reconstruction completeness and overall score on DTU, ranks third on both Tanks and Temples sets, and ranks sixth on ETH3D low-res.
Benchmark Results: R-MVSNet successfully reconstructs all Tanks and Temples advanced scenes, where previous learned methods often fail because of memory constraints.It ranks 3rd on the advanced set and also ranks 3rd on the intermediate set.
Scalability: 8× higher Mem-Util makes R-MVSNet more memory-efficient than MVSNet for wide-range depth reconstruction.Its memory requirement is independent of the depth sample number D, enabling large depth ranges.
Scalability: Increasing DTU depth samples from D = 256 to D = 512 reduces the overall distance score from 0.422mm to 0.419mm.The depth range remains fixed while sampling becomes denser along the depth direction.
Ablation Studies: The 3D CNN variant yields the best depth-map reconstruction in the ablation, while the proposed GRU variant yields the second-best result.Spatial regularization, winner-take-all selection, and classical plane sweeping produce progressively larger depth-map errors.
Runtime and Discussion: R-MVSNet generates a DTU depth map in 9.1s per view, including 2.9s for inference and 6.2s for refinement.Filtering and fusion require negligible runtime.
Runtime and Discussion: The model trained on DTU generalizes to other datasets without fine-tuning, but its performance may improve with more diverse training data.The authors report satisfying generalizability on Tanks and Temples and ETH3D while identifying broader training data as future work.
Runtime and Discussion: R-MVSNet handles inputs up to 3072 × 2048 on an 11GB GPU, but reconstruction scale remains restricted by input image size.This excludes the ETH3D high-res benchmark with 6000 × 4000 images.

6. Conclusions

The paper presents R-MVSNet as a scalable architecture for high-resolution multi-view stereo. Sequential GRU regularization reduces memory requirements while preserving high-quality benchmark performance and enabling large-scale reconstructions.

6. Conclusions: R-MVSNet sequentially regularizes the cost volume through depth with a convolutional GRU instead of using 3D CNNs.This design dramatically reduces memory requirements for learning-based MVS reconstruction.
6. Conclusions: R-MVSNet produces high-quality benchmarking results comparable to the original MVSNet after post-processing.The conclusion also reports applicability to large-scale reconstructions beyond previous learned MVS approaches.
6. Conclusions: The method applies to large-scale reconstructions that previous learning-based MVS approaches cannot handle.

1. Network Architecture

R-MVSNet constructs depth-indexed cost maps and recurrently regularizes them along the depth direction. During testing, sequential retrieval limits GPU memory to a quantity linear in input image resolution.

1. Network Architecture: R-MVSNet constructs cost maps at different depths and recurrently regularizes them through the depth direction.
1. Network Architecture: During testing, R-MVSNet sequentially retrieves regularized cost maps, requiring GPU memory linear to input image resolution.The full probability volume is explicitly computed during training but not required in the same way during testing.
1. Network Architecture: The architecture table defines Conv as 2D convolution and BR as batch normalization plus ReLU.It also identifies N, H, W, and D as input view number, image width, image height, and depth sample number.

2. Depth Sample Number

The method selects inverse-depth samples across the specified depth range and chooses the sample count by matching spatial image resolution with temporal depth resolution.

2. Depth Sample Number: Inverse-depth sampling selects depth values within [dmin, dmax] using the depth-sampling index i and sample count D.The sampling procedure is introduced for a given depth range and distributes samples according to inverse depth.
2. Depth Sample Number: The spatial resolution at depth dmin is defined by the 3D distance ρ between projections of neighboring reference-image pixels.The neighboring pixels are the reference-image center and its adjacent pixel, projected into 3D at dmin.
2. Depth Sample Number: The temporal depth resolution at dmin is defined as d(2) − d(1), linking adjacent inverse-depth samples to depth spacing.This quantity is compared with the spatial resolution when determining D.
2. Depth Sample Number: The depth sample number D is calculated by equating spatial image resolution with temporal depth resolution.This design assumes that spatial image resolution and temporal depth resolution should match.

3. Variational Depth Map Refinement

The refinement procedure iteratively minimizes image reprojection error using photo-consistency and smoothness terms, updating the reference depth map and reprojected images.

3. Variational Depth Map Refinement: The refinement represents each reference-image pixel as a 3D point and projects it into source images to formulate reprojection consistency.The 3D point is defined from the reference camera projection matrix, camera center, and pixel depth.
3. Variational Depth Map Refinement: The photo-consistency derivative with respect to depth uses the projection Jacobian and the derivative of the photometric measurement with respect to pixel coordinates.The formulation assumes equivalent forward and backward image-consistency expressions for the corresponding projection.
3. Variational Depth Map Refinement: The smoothness term penalizes squared depth differences between neighboring pixels using bilateral smoothness weighting.The derivative is taken with d1 represented by the depth map value at the pixel.
3. Variational Depth Map Refinement: Depth-map refinement and fusion have little qualitative influence but reduce the overall quantitative score in the reported post-processing comparison.The comparison is shown for DTU point-cloud reconstructions under different post-processing settings.
3. Variational Depth Map Refinement: Gradient descent minimizes the total image reprojection error with a descending step size λ(t) = 0.9 · λ(t − 1), starting from λ(0) = 10.The reference depth map and all reprojected images are updated at each step, with 20 refinement iterations fixed for the experiments.

4. Sliding Window 3D CNNs

Sliding-window 3D CNN regularization divides the cost volume into overlapping depth sub-volumes, but performs worse and runs slower because sub-volumes lose global volumetric context.

4. Sliding Window 3D CNNs: The cost volume is divided into seven overlapping sub-volumes with Dsub = 64 and Doverlap = 32 before sequential 3D CNN regularization.The setup uses D = 256 and applies the 3D CNNs separately to each sub-volume.
4. Sliding Window 3D CNNs: Volume fusion concatenates truncated regularized sub-volumes before softmax and soft argmin depth regression.The overlap region is handled by truncating each sub-volume with Dtrunc = 16.
4. Sliding Window 3D CNNs: Depth-map fusion independently regresses seven depth and probability maps, then selects depths by winner-take-all probability fusion.This is the second tested strategy for combining regularized sub-volumes.
4. Sliding Window 3D CNNs: Both sliding strategies produce higher errors than GRU and 3D CNN regularization.The comparison is reported for the qualitative and quantitative ablation results.
4. Sliding Window 3D CNNs: ∼10s inference for a 1600 × 1184 × 256 depth map is ∼2× slower than MVSNet and R-MVSNet.The timing applies to both sliding-window strategies.
4. Sliding Window 3D CNNs: Sliding windows create discrepancies between sub-volumes and restrict each sub-volume below the multi-scale 3D CNN receptive field of approximately 256^3.Consequently, the strategies cannot fully benefit from the 3D CNN regularizer.

5. Post-processing

Post-processing combines photometric and geometric filtering with depth-map refinement and fusion to produce point-cloud reconstructions across multiple MVS datasets.

5. Post-processing: Photometric and geometric filtering remove different kinds of outliers and produce visually clean DTU point clouds.The comparison evaluates DTU reconstructions under different post-processing settings.
5. Post-processing: Depth-map refinement and fusion have little influence on qualitative results but reduce the overall score in quantitative evaluation.This effect is reported for the DTU post-processing comparison.
5. Post-processing: R-MVSNet produces visually clean and complete point clouds on DTU, Tanks and Temples, and ETH3D reconstructions.The additional qualitative results are presented in Figures 3, 4, and 5 respectively.