PaperlayerView exact revision on arXiv

Source-linked AI summary

TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers

Yikang Ding, Wentao Yuan, Qingtian Zhu, Haotian Zhang, Xiangyue Liu, Yuanjiang Wang, Xiao Liu

arXiv:2111.14600v1cs.CV

TL;DR

MVS feature matching lacks global context and inter-image interaction, limiting robust estimation in challenging regions. TransMVSNet introduces a Feature Matching Transformer with intra- and inter-attention, supported by receptive-field adaptation and cross-scale feature propagation. The method achieves state-of-the-art performance on DTU, Tanks and Temples, and BlendedMVS, while inference remains sensitive to depth-related hyperparameters.

  • Problem

    MVS feature matching lacks global context and inter-image interaction, limiting robust estimation in challenging regions.

  • Method

    TransMVSNet uses a Feature Matching Transformer with intra- and inter-attention, plus Adaptive Receptive Field adaptation and a transformed feature pathway.

  • Results

    TransMVSNet achieves state-of-the-art performance on DTU, Tanks and Temples, and BlendedMVS.

  • Takeaways & Limitations

    The results support using global context-aware information within and between views for MVS matching.

  • Takeaways & Limitations

    Inference is sensitive to hyperparameters including the number of depth hypotheses, depth interval, and depth-interval decay factor.

Abstract

from arXiv · show

In this paper, we present TransMVSNet, based on our exploration of feature matching in multi-view stereo (MVS). We analogize MVS back to its nature of a feature matching task and therefore propose a powerful Feature Matching Transformer (FMT) to leverage intra- (self-) and inter- (cross-) attention to aggregate long-range context information within and across images. To facilitate a better adaptation of the FMT, we leverage an Adaptive Receptive Field (ARF) module to ensure a smooth transit in scopes of features and bridge different stages with a feature pathway to pass transformed features and gradients across different scales. In addition, we apply pair-wise feature correlation to measure similarity between features, and adopt ambiguity-reducing focal loss to strengthen the supervision. To the best of our knowledge, TransMVSNet is the first attempt to leverage Transformer into the task of MVS. As a result, our method achieves state-of-the-art performance on DTU dataset, Tanks and Temples benchmark, and BlendedMVS dataset. The code of our method will be made available at https://github.com/MegviiRobot/TransMVSNet .

1. Introduction

TransMVSNet addresses limited global context and missing inter-image interaction in MVS feature matching with a Transformer-based architecture. It reports state-of-the-art performance across DTU, Tanks and Temples, and BlendedMVS.

  • Motivation: MVS recovers dense 3D representations from calibrated images, but convolutional locality limits global context for difficult regions such as poor texture and repetitive patterns.The paper also identifies non-Lambertian surfaces as challenging regions.
  • Motivation: MVS matching compares each reference pixel against warped source images along epipolar lines, yet conventional feature comparisons omit potential inter-image correspondences.
  • Method: TransMVSNet uses a Feature Matching Transformer with intra- and inter-attention to aggregate long-range context within and across images.
  • Method: Adaptive Receptive Field modules smooth the transition from CNN-local features to Transformer-global features, while a transformed feature pathway connects scales and propagates transformed features and gradients.
  • Method: The method applies pair-wise feature correlation and ambiguity-aware focal loss within a coarse-to-fine volume-regularization pipeline.
  • Results: TransMVSNet achieves state-of-the-art results on DTU, Tanks and Temples, and BlendedMVS, with simultaneous improvement in DTU reconstruction accuracy and completeness.

2. Related Work

Related work frames learning-based MVS as a progression toward better reconstruction quality and efficiency, while highlighting persistent robustness challenges. Transformer-based feature matching provides relevant context for addressing global and cross-image information.

  • Learning-based MVS: Learning-based MVS methods improve reconstruction accuracy and completeness, but early 3D cost-volume regularization can incur substantial memory and computation costs.
  • Learning-based MVS: Recurrent and coarse-to-fine multi-stage networks were proposed to reduce the costs associated with regularizing 3D cost volumes.
  • Open challenges: Robust estimation remains challenging in non-Lambertian, low-texture, and severely occluded regions.
  • Transformer-based matching: Transformer attention captures global context and has been applied to feature matching, including sparse matching with SuperGlue and dense coarse-to-fine matching with LoFTR.

3. Methodology

TransMVSNet treats MVS as one-to-many feature matching and combines multi-scale CNN features with Transformer-based global context aggregation, correlation volumes, and focal supervision.

  • Network Overview: The network predicts each reference-view depth map from calibrated neighboring images, then filters and fuses all depth maps into a dense point cloud.It uses the reference image, neighboring images, and camera intrinsics and extrinsics as inputs.
  • Attention Mechanism: Linear Transformer attention reduces attention complexity from quadratic to linear, enabling attention computation on higher-resolution images.The reduction follows from using a feature channel count smaller than the input-sequence length.
  • Feature Matching Transformer: FMT uses intra-attention for within-image global context and inter-attention for cross-image feature searching and matching in MVS’s one-to-many setting.The reference feature remains unchanged during inter-attention, while source features are updated.
  • Feature Pathway: A transformed feature pathway interpolates low-resolution FMT features to higher resolutions and passes gradients from all scales through FMT.This pathway addresses the memory and computation limits of applying Transformer operations at higher resolutions.
  • Adaptive Receptive Field: ARF uses deformable convolution to adaptively enlarge receptive fields between locally focused FPN features and globally contextual Transformer features.This module addresses the receptive-field gap between FPN and FMT and smooths feature transition.
  • Correlation and Loss: Differentiable warping aligns source images to the reference view across D depth hypotheses, after which pair-wise correlations form one-channel correlation volumes for regularization.Pixel-wise weights based on maximum depth-wise correlation aggregate the N −1 pair-wise volumes, while focal loss emphasizes ambiguous boundary pixels.

4. Experiments

Experiments evaluate TransMVSNet on DTU, Tanks and Temples, and BlendedMVS using point-cloud and depth-map metrics, plus qualitative comparisons and ablations. The method achieves strong benchmark performance while its components expose accuracy–efficiency trade-offs.

  • DTU evaluation: TransMVSNet outperforms all known methods in DTU Overall while achieving competitive Accuracy and Completeness.Accuracy measures reconstruction-to-ground-truth distance, Completeness measures the reverse, and Overall averages them.
  • Tanks and Temples: TransMVSNet achieves state-of-the-art performance on both the intermediate and advanced Tanks and Temples leaderboards using mean F-score.Qualitative results show more reliable points in low-textured areas and on sophisticated surfaces.
  • BlendedMVS: TransMVSNet achieves impressive depth-map results on the BlendedMVS validation set, evaluated with EPE, e1, and e3.EPE is average absolute depth error; e1 and e3 measure the percentages of pixels exceeding errors of 1 and 3.
  • Ablation study: Focal loss improves Overall performance by 1.7% without changing computational costs.FMT adds little memory and MAC cost but makes inference nearly 1.4 times slower; ARF adds considerable computation while keeping inference within one second.
  • Ablation study: The transformed feature pathway boosts Completeness and Overall performance with almost no increase in memory occupancy.The full model with ARF achieves state-of-the-art performance by a large margin, though ARF increases computational costs.

5. Discussions

The discussion distinguishes TransMVSNet from related MVS and matching methods by adapting Transformer attention to multi-view, one-to-many feature matching. It also identifies slower inference and sensitivity to coarse-to-fine inference settings as limitations.

  • Comparisons: TransMVSNet extends CasMVSNet’s coarse-to-fine pattern with Transformer-based long-range context for multi-view feature matching.The coarse-to-fine design improves computational efficiency while retaining strong performance.
  • Comparisons: FMT adapts self- and cross-attention to MVS’s one-to-many matching structure, unlike LoFTR’s pairwise dense matching formulation.MVS requires each reference pixel to search across multiple source images.
  • Comparisons: Unlike STTR’s epipolar-line attention, TransMVSNet applies attention across flattened feature maps because MVS lacks line-to-line correspondence.This design brings global context into feature matching over multiple views.
  • Limitations: Transformer use slows inference, while coarse-to-fine inference remains sensitive to depth hypotheses, depth intervals, and decay factors.These limitations constrain speed and dependence on inference hyperparameters.

6. Conclusion

TransMVSNet is a Transformer-based MVS network designed to aggregate long-range context within and across images. Its formulation is motivated by viewing MVS as a one-to-many feature-matching task evaluated through ablations and matching-pattern analysis.

  • TransMVSNet aggregates global long-range context through a Feature Matching Transformer within and across images.Its FMT uses Transformer-based context aggregation for learning-based MVS.
  • MVS can be treated as one-to-many matching, where each reference pixel searches epipolar-line hypotheses across neighboring source images for an optimal depth.The matching pattern is illustrated for N = 3 views and plane-sweep depth hypotheses.
  • The reported ablation evaluates the number of input views and image resolution on the DTU evaluation set.Table 5 reports these settings with lower-is-better metrics.

B.2. Focusing Parameter

The focusing parameter γ has dataset-dependent effects in focal-loss training. γ = 0 fits DTU best, whereas γ = 2 is beneficial for the more diverse BlendedMVS scenes.

  • γ = 0 obtains the best Accuracy and Overall scores on the DTU dataset.The authors attribute this fit to DTU scene complexity.
  • γ = 2 makes a big difference on the more diverse and complicated BlendedMVS dataset.The comparison uses the same inference settings as the reported case in the main paper.
  • Table 6 compares focal-loss values of γ on the DTU evaluation set and BlendedMVS validation set.The table reports lower-is-better metrics.

C. Ablation Study on FMT Design

The FMT design is examined through an ablation study focused on attention-head configuration. The supplied evidence identifies DTU evaluation as the test setting but does not report individual head-count outcomes.

  • The study further explores the architecture design of the Feature Matching Transformer.
  • The FMT ablation concerns the number of attention heads Nh.The corresponding evaluation is reported on the DTU evaluation set.
  • Table 7 reports lower-is-better metrics for different attention-head counts on DTU.The supplied passage does not state which head count performs best.

C.2. Number of Attention Blocks

The attention-block ablation evaluates performance alongside memory occupancy and inference time. Four attention blocks provide the reported balance between performance and efficiency.

  • Na = 4 achieves a balance between performance and efficiency.This conclusion is based on the reported evaluation results, memory occupancy, and inference time.
  • The experiment adjusts the number of attention blocks Na.
  • Table 8 reports evaluation results, memory occupancy, and inference time for different attention-block counts on DTU.The table uses lower-is-better metrics.

C.3. Design of Attention Block

The attention-block study compares four ways to handle the reference feature under inter-attention. TransMVSNet selects the design using only reference-to-source inter-attention, which is more efficient while supporting complementary global and cross-image context aggregation.

  • Candidate designs: The four candidate designs differ primarily in how the reference feature F0 is handled during inter-attention.Intra-attention is applied to both reference and source features in all designs.
  • Candidate designs: Design (a) updates F0 only through intra-attention while applying reference-to-source inter-attention, and this is TransMVSNet’s final choice.
  • Candidate designs: Designs (b), (c), and (d) additionally update F0 through source-to-reference inter-attention, differing in sequential, parallel, or duplicated one-to-one processing.
  • Ablation study: Among designs (a), (b), and (c), memory consumption is identical, while design (a) is more efficient in inference time.
  • Attention behavior: Intra-attention globally seeks context and inter-attention matches features across images, making the mechanisms complementary for challenging textureless or non-Lambertian regions.
  • Feature evolution: After several attention modules, FMT encodes more global, position-dependent context into feature maps, benefiting matching on textureless and non-Lambertian surfaces.

F. More Point Cloud Results

The paper visualizes reconstructed point clouds and transformed features across multiple datasets and scenes. These results illustrate TransMVSNet’s feature evolution and its robustness and scalability across varying scenes.

  • Point-cloud results: The visualized results demonstrate TransMVSNet’s robustness and scalability on scenes with varying characteristics.
  • Feature visualization: Figure 11 visualizes transformed-feature evolution by reducing feature channels to three with PCA and coloring them using RGB.
  • Implementation: The implementation is based on CasMVSNet and uses preprocessed DTU and Tanks and Temples images and camera parameters from official MVSNet and R-MVSNet repositories.
  • Point-cloud results: Figures 12–14 show reconstructed point clouds for all 22 DTU scans, all Tanks and Temples intermediate and advanced scenes, and all 7 BlendedMVS validation scenes.
Loading 2111.14600v1…