PaperlayerView exact revision on arXiv

Source-linked AI summary

IterMVS: Iterative Probability Estimation for Efficient Multi-View Stereo

Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Marc Pollefeys

arXiv:2112.05126v1cs.CV

TL;DR

Real-world MVS remains challenging, and efficient high-resolution methods face memory, run-time, and coarse-to-fine search limitations. IterMVS iteratively refines pixel-wise depth probabilities in a GRU using multi-scale matching information, then estimates depth and confidence. It reports competitive DTU performance, strong efficiency, and better generalization than many state-of-the-art learning-based methods.

  • Problem

    Real-world MVS remains challenging under occlusion, illumination changes, low texture, and non-Lambertian surfaces, while high-resolution cost-volume methods and coarse-to-fine search have resource or recovery limitations.

    Source: §1 · pp. 1–2

  • Method

    IterMVS iteratively updates per-pixel depth-probability distributions in a GRU hidden state using multi-scale matching information, then combines classification and regression to estimate depth.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    IterMVS reports highest memory and run-time efficiency among learning-based methods, competitive DTU performance, and better generalization on Tanks & Temples and ETH3D than many state-of-the-art methods.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Inference can trade time efficiency for accuracy because performance improves across reported metrics as the number of GRU iterations increases.

    Source: §4.4 · p. 8

    The number of depth samples is fixed by the network structure and cannot be adjusted by scene; the data-term samples cover one quarter of the total inverse-depth range.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

We present IterMVS, a new data-driven method for high-resolution multi-view stereo. We propose a novel GRU-based estimator that encodes pixel-wise probability distributions of depth in its hidden state. Ingesting multi-scale matching information, our model refines these distributions over multiple iterations and infers depth and confidence. To extract the depth maps, we combine traditional classification and regression in a novel manner. We verify the efficiency and effectiveness of our method on DTU, Tanks&Temples and ETH3D. While being the most efficient method in both memory and run-time, our model achieves competitive performance on DTU and better generalization ability on Tanks&Temples as well as ETH3D than most state-of-the-art methods. Code is available at https://github.com/FangjinhuaWang/IterMVS.

1. Introduction

Real-world MVS remains difficult under occlusion, illumination changes, low texture, and non-Lambertian surfaces, while efficient high-resolution reconstruction is constrained by competing methods’ resource and search-range trade-offs. IterMVS uses iterative GRU-based depth-probability estimation and reports competitive DTU performance, improved generalization, and high memory and run-time efficiency.

1. Introduction

  • Real-world MVS remains challenging under occlusion, illumination changes, low-textured areas, and non-Lambertian surfaces.Traditional methods rely on hand-crafted models and matching metrics, while data-driven approaches have improved benchmark performance.Source: §1 · p. 1
  • 3D-CNN cost-volume regularization is memory- and run-time-intensive; recurrent methods reduce memory at a run-time cost, while cascades risk unrecoverable coarse-stage errors.PatchmatchNet is described as highly efficient, but its generalization appears limited compared with other multi-stage methods.Source: §1 · pp. 1–2
  • IterMVS is more efficient in memory and run-time than PatchmatchNet, matches its DTU performance, and generalizes better on Tanks & Temples and ETH3D.The figure compares learning-based methods trained on DTU; its panels show Tanks & Temples F-score and resource use, plus an ETH3D qualitative comparison.Source: Figure 1 · p. 1, §1 · p. 2
  • A GRU-based estimator stores each pixel’s depth-probability distribution in its hidden state and updates it with multi-scale matching information over iterations.The method also combines classification and regression for depth estimation, aiming to handle multi-modal distributions while retaining sub-pixel precision.Source: §1 · p. 2

2. Related Work

Related work spans traditional scene representations, data-driven cost-volume methods, and recurrent iterative updates. IterMVS positions its GRU as a way to model a pixel-wise depth distribution across the full range rather than rely on coarse-to-fine search.

2. Related Work

  • Traditional MVS includes volumetric, point-cloud, and depth-map methods; volumetric approaches use substantial memory and run-time, while hand-crafted depth-map methods struggle under challenging conditions.Point-cloud methods often densify reconstructions through propagation, and depth-map methods decouple estimation from fusion.Source: §2 · p. 2
  • Data-driven MVS commonly regularizes cost volumes with 3D CNNs, while recurrent methods trade run-time for memory and cascades trade broad coarse search for narrower fine-stage search.Coarse-to-fine methods can have difficulty recovering from errors introduced at coarse resolutions; IterMVS instead estimates at relatively high resolution and keeps a full-range probability distribution in its GRU state.Source: §2 · p. 2
  • Iterative-update methods use GRUs for tasks such as optical flow, while IterMVS uses its GRU to model and iteratively refine each pixel’s depth-probability distribution.The hidden state is updated each iteration to model that distribution more accurately.Source: §2 · pp. 2–3

3. Method

IterMVS combines multi-scale feature matching with a GRU that iteratively updates per-pixel depth distributions and estimates depth and confidence. It upsamples the quarter-resolution outputs and trains with classification, regression, and confidence losses.

3.1. Multi-scale Feature Extractor

  • Three FPN feature levels provide multi-scale context for matching at each GRU iteration without an explicit coarse-to-fine structure.Features are stored at 1/2^l resolution, with 16, 32, and 64 channels at levels 1, 2, and 3, respectively.Source: §3.1 · p. 3

3.2. GRU-based Probability Estimator

  • The GRU iteratively updates a hidden state encoding per-pixel depth distributions from multi-scale matching information, then predicts depth and confidence.It operates at quarter resolution; the distributions are sampled from the hidden state, and confidence estimates whether ground-truth depth lies near the prediction.Source: §3.2 · pp. 3–4
  • Depth prediction combines the highest-probability sample with an expectation over a local inverse-depth range, supporting multimodal distributions and subpixel precision.This hybrid addresses the stated limitations of argmax and soft argmax strategies.Source: §3.2 · p. 4

3.3. Spatial Upsampling

  • A feature-guided spatial upsampling module converts the GRU’s quarter-resolution depth to full resolution and bilinearly upsamples confidence.A CNN predicts weights for the nine nearest coarse-resolution neighbors to compute full-resolution depth.Source: §3.3 · pp. 4–5, Figure 2 · p. 4

3.4. Loss Function

  • Training combines classification, regression, confidence, and upsampling losses, with valid-depth pixels considered and regression and confidence losses withheld during the first epoch.The loss also includes an initial-depth regression term to facilitate convergence.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate IterMVS on DTU, Tanks & Temples, and ETH3D, reporting competitive reconstruction alongside low memory and run-time use and cross-dataset generalization results.

4.1. Datasets

  • The cited datasets include indoor DTU scenes (124 scenes, seven lighting conditions) and high-resolution ETH3D images of real-world scenes with strong viewpoint variations.BlendedMVS provides over 17k high-quality training samples.Source: §4.1 · p. 5

4.2. Implementation Details

  • The DTU model uses 640×512 images and five views with per-view depth ranges estimated from sparse ground-truth points; BlendedMVS uses 768×576 images and dataset-provided ranges.Both models train for 16 epochs; inference uses four GRU updates in the reported benchmark settings, followed by confidence and geometric-consistency filtering.Source: §4.2 · p. 5

4.3. Benchmark Performance

  • On Tanks & Temples, IterMVS is best in advanced-set Recall and F-score; on ETH3D, BlendedMVS training yields the best F1-score on both datasets.DTU-trained IterMVS has intermediate-set F-score similar to CasMVSNet on Tanks & Temples; BlendedMVS training improves generalization on both Tanks & Temples datasets and ETH3D metrics.Source: §4.3 · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • At 1600×1152, IterMVS uses 78.0% less memory and 68.7% less run-time than CasMVSNet [11], with competitive DTU quality and higher accuracy than PatchmatchNet [39].The paper also reports lower memory and run-time than UCS-Net [6] and PatchmatchNet [39] at this resolution.Source: Figure 3 · p. 6, §4.3 · p. 6, Table 1 · p. 6

4.4. Ablation Study

  • Hybrid classification and regression outperform either depth-prediction alternative; multi-scale matching features and learned upsampling also improve results on DTU and ETH3D.The reported comparisons include both datasets for the hybrid strategy and learned upsampling, and both datasets for multi-scale context.Source: §4.4 · pp. 7–8
  • Learned confidence performs slightly better than summed probabilities on DTU and ETH3D, while inverse-depth losses improve ETH3D and produce similar DTU results.At object boundaries, summed probabilities can over-smooth confidence and assign high confidence to an unmatchable background pixel.Source: §4.4 · pp. 7–8, Figure 4 · p. 7
  • Beyond four GRU iterations, performance continues improving across metrics; reconstruction quality improves with more views and saturates around six, while pixel-wise view weighting matters on ETH3D.The paper attributes the view-weighting benefit to strong viewpoint variation in ETH3D.Source: §4.4 · p. 8, Table 5 · p. 8, Table 6 · p. 8

4.5. Limitations

  • Inference iterations can trade speed for accuracy, but the number of probability samples and the data-term sampling range are fixed by the network structure.The data-term samples cover one quarter of the total inverse-depth range.Source: §4.5 · p. 8

5. Conclusion

IterMVS combines a GRU-based pixel-wise depth-probability estimator with multi-scale matching and inverse-depth sampling to infer depth and confidence. Experiments on DTU, Tanks & Temples and ETH3D report high memory and run-time efficiency alongside competitive reconstruction quality and better generalization than many state-of-the-art learning-based methods.

5. Conclusion

  • Across DTU, Tanks & Temples and ETH3D, IterMVS reports high memory and run-time efficiency with better generalization than many state-of-the-art learning-based methods.The conclusion also describes its reconstruction quality as competitive.Source: §5 · p. 8
  • The GRU-based estimator explicitly encodes each pixel’s depth probability distribution in its hidden state and receives multi-scale matching information at each iteration.The method samples depth distributions uniformly in inverse-depth range to estimate depth maps and confidence.Source: §5 · p. 8
  • Experiments on DTU, Tanks & Temples and ETH3D support the paper’s conclusion of competitive reconstruction quality.Source: §5 · p. 8

1. Matching Similarities with Multi-scale Features

IterMVS computes matching similarities using features from multiple scales during each iterative update. It aggregates the resulting information spatially before passing it onward.

1. Matching Similarities with Multi-scale Features

  • For each depth-map pixel, the method locates its corresponding position in reference feature levels 1–3 and bilinearly interpolates the reference features.Source: §1 · p. 9
  • For each level, new depth hypotheses, camera parameters and source features are used to warp the reference position into source views and compute matching similarities.Source: §1 · p. 9
  • Upsampled pixel-wise view weights integrate the matching similarities, which a level-wise 2D U-Net then uses to aggregate neighborhood information.Source: §1 · p. 9

2. Depth Upsampling

IterMVS upsamples its depth map from quarter resolution to full resolution using learned weights from the reference feature map. Each high-resolution pixel combines depth values from nine coarse-resolution neighbors.

2. Depth Upsampling

  • Each full-resolution depth value is a convex combination of the nine neighboring coarse-resolution depth values.The weights are learned from the reference feature map.Source: §2 · p. 9, Figure 5 · p. 9
  • The depth map is upsampled from 1/4 resolution to full resolution.Source: §2 · p. 9

3. Point Cloud Reconstruction

Before fusing depth maps, IterMVS filters unreliable estimates using geometric consistency across views and a learned-confidence threshold.

3. Point Cloud Reconstruction

  • The reconstruction applies geometric consistency filtering and confidence filtering before depth-map fusion.Source: §3 · p. 9
  • A depth estimate is geometrically consistent with a source view when its reprojection satisfies image-space and depth-distance thresholds.An estimate is accepted as reliable when it is consistent in at least Ngeo source views; the thresholds are δ = 1 and ε = 0.01.Source: §3 · p. 9
  • The confidence filter removes pixels whose learned confidence is below τ = 0.3.The threshold is used throughout the experiments.Source: §3 · p. 9

4. Visualization of Probability

IterMVS represents each pixel’s depth distribution in a GRU hidden state and updates it with multi-scale matching information. Across iterations, distributions become more focused and depth estimates more precise.

4. Visualization of Probability

  • A GRU hidden state encodes each pixel’s depth distribution, which a 2D CNN maps to probabilities over uniformly sampled inverse-depth values.Multi-scale matching information is injected into the hidden state at each iteration.Source: §4 · p. 9
  • With more GRU iterations, local maxima are suppressed, distributions become more focused, and depth estimates become more precise.The authors report that the hidden state models each pixel’s depth distribution more accurately with each iteration.Source: §4 · p. 9
  • For challenging textureless areas with non-peaky or multimodal distributions, the hybrid depth-estimation strategy still produces estimates as accurate as possible.Source: §4 · p. 9

5. Visualization of Pixel-wise View Weight

The visualizations show pixel-wise view weights assigning higher values to co-visible areas and lower values to occluded or unreliable regions. The paper also presents point-cloud reconstructions across three benchmarks and identifies current limits and possible extensions for learned confidence.

5. Visualization of Pixel-wise View Weight

  • Visible areas receive higher view weights than occluded or invisible areas; reflective windows and strong perspective or specular reflections can also receive low weights.Fronto-parallel, textured regions receive higher weights; the authors conclude the weights can determine co-visible areas.Source: §5 · pp. 9–10, Figure 8 · p. 11

6. Visualization of Point Clouds

7. Future Work

  • Learned confidence currently only filters unreliable estimates before depth fusion; proposed directions include refining uncertain depths from confident areas or focusing computation on uncertain areas.Source: §7 · p. 11
Loading 2112.05126v1…