PaperlayerView exact revision on arXiv

Source-linked AI summary

AA-RMVSNet: Adaptive Aggregation Recurrent Multi-view Stereo Network

Zizhuang Wei, Qingtian Zhu, Chen Min, Yisong Chen, Guoping Wang

arXiv:2108.03824v1cs.CV

TL;DR

AA-RMVSNet targets incomplete reconstruction in challenging regions and unreliable matching under varying occlusion. It combines LSTM-based recurrent regularization with intra-view adaptive feature aggregation and inter-view pixel-wise cost-volume aggregation. The method achieves competitive DTU results and ranks first on Tanks and Temples with a 61.51 mean F-score.

  • Problem

    Learning-based MVS still struggles with challenging regions, severe occlusion, and reliable feature and cost aggregation.

  • Method

    AA-RMVSNet uses an LSTM-based recurrent MVS network with intra-view adaptive feature aggregation and inter-view pixel-wise attention for cost-volume aggregation.

  • Results

    AA-RMVSNet ranks 1st on the Tanks and Temples benchmark with 61.51 mean F-score and achieves competitive overall performance on DTU.

  • Takeaways & Limitations

    The complementary adaptive aggregation modules improve reconstruction of thin objects, large low-textured surfaces, and scenes with varying occlusion while supporting generalizability and scalability.

Abstract

from arXiv · show

In this paper, we present a novel recurrent multi-view stereo network based on long short-term memory (LSTM) with adaptive aggregation, namely AA-RMVSNet. We firstly introduce an intra-view aggregation module to adaptively extract image features by using context-aware convolution and multi-scale aggregation, which efficiently improves the performance on challenging regions, such as thin objects and large low-textured surfaces. To overcome the difficulty of varying occlusion in complex scenes, we propose an inter-view cost volume aggregation module for adaptive pixel-wise view aggregation, which is able to preserve better-matched pairs among all views. The two proposed adaptive aggregation modules are lightweight, effective and complementary regarding improving the accuracy and completeness of 3D reconstruction. Instead of conventional 3D CNNs, we utilize a hybrid network with recurrent structure for cost volume regularization, which allows high-resolution reconstruction and finer hypothetical plane sweep. The proposed network is trained end-to-end and achieves excellent performance on various datasets. It ranks $1^{st}$ among all submissions on Tanks and Temples benchmark and achieves competitive results on DTU dataset, which exhibits strong generalizability and robustness. Implementation of our method is available at https://github.com/QT-Zhu/AA-RMVSNet.

1. Introduction

AA-RMVSNet addresses incomplete reconstruction in challenging regions and unreliable matching under occlusion through complementary intra-view and inter-view adaptive aggregation in a recurrent MVS network.

  • Deep-learning MVS improves accuracy and completeness, but challenging regions and reconstruction quality remain unresolved.
  • Fixed-grid, fixed-receptive-field features struggle with thin structures and textureless surfaces because context-aware features are insufficiently adapted to regional texture.
  • Pixel-wise visibility issues during cost aggregation can deteriorate reconstruction quality under severe occlusion, while existing view-weighting approaches remain imperfect.
  • AA-RMVSNet combines LSTM-based recurrent regularization with intra-view and inter-view adaptive aggregation to improve feature extraction and multi-view matching.The intra-view module aggregates context-aware features across scales and regions; the inter-view module assigns higher weights to well-matched view pairs.
  • The method introduces deformable-convolution and multi-scale feature aggregation, plus pixel-wise attention maps for adaptive cost-volume aggregation.

2. Related Work

Learning-based MVS evolved from volumetric and point-based representations toward depth-based reconstruction, while recurrent methods address memory efficiency but challenging regions and occlusion remain difficult.

  • Volumetric methods discretize 3D space into regular cubes, whereas point-based methods densify sparse matched points through propagation.
  • Depth-based methods estimate per-view depth maps and can fuse them into point clouds or volumetric reconstructions.
  • Early learning-based volumetric methods use multi-view cost volumes and 3D CNNs but are restricted to small-scale reconstructions.
  • Recurrent MVS methods regularize cost volumes sequentially, while D2HC-RMVSNet uses ConvLSTMCells and dynamic consistency checking.
  • Despite promising results, learning-based MVS methods still struggle with challenging regions and severe occlusion.

3. Methodology

AA-RMVSNet follows a recurrent multi-view stereo pipeline that extracts shared image features, builds and aggregates per-view cost volumes, recurrently regularizes them, and predicts depth probabilities. Its adaptive aggregation modules address texture variation and unreliable views, while the hybrid regularizer processes depth-sliced cost volumes for memory-efficient reconstruction.

  • Overall architecture: AA-RMVSNet extracts H × W × F features for one reference and N −1 source images, warps source features through differentiable homography, and constructs an H × W × D × F cost volume.The regularized volume produces an H × W × D probability volume for depth prediction.
  • Intra-view Adaptive Aggregation: Intra-view adaptive aggregation uses deformable convolutions and multi-scale feature maps to adjust receptive fields across regions with different texture richness.The module interpolates smaller maps and concatenates 16-, 8-, and 8-channel features into H × W × 32.
  • Inter-view Adaptive Aggregation: Inter-view adaptive aggregation assigns pixel-wise attention to per-view cost volumes, suppressing unreliable matches caused by occlusion or difficult viewing and lighting conditions.The reweighted cost volumes are summed and divided by N −1; the factor 1 + ω(·) is used to better prevent over-smoothing.
  • Recurrent Cost Regularization: A RNN-CNN hybrid regularizer slices the H × W × D × 32 cost volume along depth, using CNN processing horizontally and five parallel RNNs vertically.This recurrent regularization converts matching costs into a pixel-wise probability distribution over D depth hypotheses.
  • Loss Function: Cross-entropy training compares one-hot encoded ground-truth depth probabilities with predicted probabilities over valid pixels having reliable depth.The loss sums over depth hypotheses and the set of valid pixels.

4. Experiments

AA-RMVSNet is evaluated on DTU, Tanks and Temples, and BlendedMVS, with experiments examining reconstruction quality, generalizability, memory efficiency, and adaptive-aggregation ablations.

  • DTU evaluation: AA-RMVSNet achieves the best completeness and competitive overall performance on the DTU evaluation set.Compared with R-MVSNet and D2HC-RMVSNet, it improves both accuracy and completeness.
  • Tanks and Temples benchmark: 61.51 mean F-score places AA-RMVSNet 1st on the Tanks and Temples leaderboard.The method outperforms existing MVS methods and improves reconstruction at low-textured, occluded, and thin regions.
  • BlendedMVS evaluation: AA-RMVSNet reconstructs both wide-range aerial scenes and small objects on the BlendedMVS validation set.The network is fine-tuned on BlendedMVS before validation testing.
  • Ablation study: The intra-view and inter-view adaptive aggregation modules each lower depth error, while their combination achieves the best training performance.The ablation compares architectures with or without the proposed modules using DTU validation results.
  • Ablation study: Using both modules reduces overall error from 0.391 to 0.357, while full AA-RMVSNet uses 4.25GB for 800 × 600 depth maps.Intra-view aggregation adds about 1.74GB and improves completeness by 0.28; inter-view aggregation adds 0.11GB and improves accuracy by 0.31.

5. Conclusion

AA-RMVSNet combines recurrent cost-volume regularization with complementary intra-view and inter-view adaptive aggregation modules. The method achieves competitive DTU results and a significant margin over other Tanks and Temples submissions.

  • AA-RMVSNet is a recurrent multi-view stereo network with complementary adaptive aggregation modules.The intra-view module targets feature extraction, while the inter-view module addresses pixel-wise view aggregation under varying occlusion.
  • The intra-view module improves performance on thin objects and large low-textured surfaces through adaptive multi-scale and context-aware feature integration.
  • The inter-view module handles varying occlusion in complex scenes through adaptive pixel-wise view aggregation.
  • AA-RMVSNet achieves competitive results on DTU and outperforms other submissions by a significant margin on Tanks and Temples.The reported results indicate generalizability and scalability.

A. Network Details

The network applies the same feature-extraction procedure across input images and the same cost-volume-processing procedure across depth hypotheses. Its intra-view aggregation adapts receptive fields to regional texture and structure.

  • Feature extraction is identical for all N input images, and cost-volume processing is identical for all D depth hypotheses.
  • Rich-texture regions use smaller receptive fields to preserve tiny details during matching.
  • Low-textured or textureless regions use larger receptive fields to aggregate more context for reliable matching.
  • Deformable convolution adapts sampling locations by region, keeping boundary samples on the same surface and expanding the receptive field in textureless areas.The paper visualizes these adaptive offsets for thin objects, weak-textured regions, and object boundaries.

C. View Reweighting in Inter-view AA

The inter-view AA module produces pixel-wise attention maps to reweight per-view cost volumes according to local visibility. Higher weights enhance well-captured regions, while lower weights suppress occluded matching.

  • The inter-view AA module uses a CNN to generate pixel-wise attention maps for per-view cost volumes.
  • Occluded areas receive lower weights, while well-captured and unoccluded areas receive higher weights.
  • Attention maps show larger values for areas well-captured in corresponding source images.Brighter areas in the visualization represent higher assigned weights.
  • Adaptive reweighting enhances reliably matched cost-volume regions and suppresses unreliable occluded regions.

D. Depth Comparison in Ablation Experiments

Ablation depth comparisons show distinct benefits from the two adaptive aggregation modules. Their integration produces accurate and complete depth maps under varying conditions.

  • The intra-view AA module eliminates noise on textureless surfaces and object boundaries.
  • The inter-view AA module preserves more details in regions that are easily occluded, such as the handle.
  • Combining both modules enables accurate and complete depth maps under varying conditions.

E. Ablation Study on Experiment Settings

The ablation study examines how input views, depth hypotheses, and image resolution affect AA-RMVSNet reconstruction. More views, finer depth division, and larger images improve reconstruction quality under the tested settings.

  • Number of Views: Increasing input views from N = 3 to 5 to 7 improves reconstruction across all metrics.The comparison fixes the number of depth hypotheses and image resolution.
  • Number of Views: Inter-view adaptive aggregation enhances valid information from neighboring views while suppressing information from occluded views.This supports the observed improvement with more input views.
  • Number of Depth Hypotheses: Using D = 512 instead of D = 256 lowers reconstruction error with N = 7 and image resolution 480 × 360.The recurrent RNN-CNN hybrid regularizer reduces memory usage, allowing finer depth-space division.
  • Resolution of Images: Increasing image resolution from 480 × 360 to 800 × 600 benefits reconstruction with N = 7 and D = 512.Memory-efficient recurrent cost-volume regularization enables use of larger-resolution images.

F. More Point Cloud Results

The paper visualizes complete point-cloud results for the DTU evaluation set, the intermediate Tanks and Temples set, and the BlendedMVS validation set.

  • Figure 12 presents all point-cloud results for the DTU evaluation set.
  • Figure 13 presents all point-cloud results for the intermediate set of the Tanks and Temples benchmark.
  • Figure 14 presents all point-cloud results for the BlendedMVS validation set.
Loading 2108.03824v1…