PaperlayerView exact revision on arXiv

Source-linked AI summary

MatchFormer: Interleaving Attention in Transformers for Feature Matching

Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen

arXiv:2203.09645v3cs.CVcs.ROeess.IV

TL;DR

Local feature matching remains computationally intensive, while detector-based and sequential extract-to-match approaches struggle with low-texture scenes or underuse encoder matching capacity. MatchFormer addresses this with a hierarchical encoder that interleaves self-attention for extraction and cross-attention for matching, achieving state-of-the-art results across four benchmark families. Its main scope boundary is that different efficient attention modules are better suited to indoor versus outdoor inputs, leaving a uniform module for future work.

  • Problem

    Existing detector-based methods are computationally intensive, while extract-to-match methods neglect encoder matching capacity and overburden the decoder.

  • Method

    MatchFormer interleaves self- and cross-attention within each stage of a hierarchical encoder to perform feature extraction and matching synchronously, with multi-scale fusion in a lightweight decoder.

  • Results

    MatchFormer achieves state-of-the-art performance on indoor and outdoor pose estimation, homography estimation, image matching, and visual localization across ScanNet, MegaDepth, HPatches, and InLoc.

  • Takeaways & Limitations

    The extract-and-match strategy provides a multi-win solution in matching efficiency, robustness, and precision, especially for low-texture scenes and limited outdoor training data.

  • Takeaways & Limitations

    Efficient attention choices remain setting-dependent: LA suits dense high-resolution outdoor scenes, whereas SEA suits sparse low-resolution indoor scenes; a uniform module is left for future work.

Abstract

from arXiv · show

Local feature matching is a computationally intensive task at the subpixel level. While detector-based methods coupled with feature descriptors struggle in low-texture scenes, CNN-based methods with a sequential extract-to-match pipeline, fail to make use of the matching capacity of the encoder and tend to overburden the decoder for matching. In contrast, we propose a novel hierarchical extract-and-match transformer, termed as MatchFormer. Inside each stage of the hierarchical encoder, we interleave self-attention for feature extraction and cross-attention for feature matching, yielding a human-intuitive extract-and-match scheme. Such a match-aware encoder releases the overloaded decoder and makes the model highly efficient. Further, combining self- and cross-attention on multi-scale features in a hierarchical architecture improves matching robustness, particularly in low-texture indoor scenes or with less outdoor training data. Thanks to such a strategy, MatchFormer is a multi-win solution in efficiency, robustness, and precision. Compared to the previous best method in indoor pose estimation, our lite MatchFormer has only 45% GFLOPs, yet achieves a +1.3% precision gain and a 41% running speed boost. The large MatchFormer reaches state-of-the-art on four different benchmarks, including indoor pose estimation (ScanNet), outdoor pose estimation (MegaDepth), homography estimation and image matching (HPatch), and visual localization (InLoc).

1 Introduction

The introduction identifies inefficiencies in detector-based and extract-to-match pipelines and proposes MatchFormer, which interleaves self- and cross-attention throughout a hierarchical encoder. This extract-and-match design targets efficiency, robustness, and precision across low-texture and multi-scale matching settings.

  • Motivation: Matching underlies SfM, SLAM, relative pose estimation, and visual localization, but classical local-feature pipelines are computationally intensive.The high dimensionality of local features contributes to this computational cost.
  • Motivation: Detector-based and extract-to-match methods leave the encoder’s matching capacity underused while placing matching demands on the decoder.The introduction specifically describes attention-based decoders as computationally inefficient in this pipeline.
  • MatchFormer: Earlier cross-attention improves robustness particularly in low-texture indoor scenes and when fewer outdoor training samples are available.Multi-level encoder extraction and multi-scale decoder fusion also contribute to matching robustness.
  • Results: MatchFormer achieves state-of-the-art performance on low-texture indoor matching and across the ScanNet, MegaDepth, HPatches, and InLoc benchmarks.The paper investigates lite and large variants and introduces PosPE for continuous patch and location information.
  • MatchFormer: MatchFormer interleaves self-attention and cross-attention within each hierarchical encoder stage to synchronize feature extraction and matching.The design forms a match-aware encoder and an extract-and-match pipeline rather than a sequential extract-to-match pipeline.
  • Results: MatchFormer is reported as superior to previous detector-based and extract-to-match methods in pose estimation, homography estimation, and visual localization.The cited result passage summarizes superiority across these task categories.

2 Related Work

Related work includes detector-based pipelines and CNN-based extract-to-match methods for learning dense, discriminative features. These approaches use staged detection, description, matching, refinement, and geometric estimation or CNN feature fusion.

  • Detector-based methods: Detector-based local feature matching typically detects interest points, computes descriptors, searches nearest neighbors, rejects incorrect matches, and estimates geometric transformations.This five-step pipeline is described as the usual structure of detector-based methods.
  • Extract-to-match methods: CNN-based extract-to-match methods learn dense and discriminative features, often combining multi-resolution CNN maps or multiple frames and scales.Examples include interpolation-based pixel descriptors, descriptor refinement, and fusion of multi-resolution matching tensors.

3 Methodology

MatchFormer uses a four-stage hierarchical transformer that interleaves self- and cross-attention for feature extraction and matching, then fuses multi-scale features for coarse-to-fine matching. Its design combines positional patch embedding, efficient attention, and a lightweight decoder.

  • Architecture: MatchFormer’s four-stage hierarchical encoder produces high-resolution coarse and low-resolution fine features before coarse-to-fine matching.An FPN-like decoder fuses features from the progressively shrinking stages.
  • Attention design: Interleaved self-attention extracts non-local features within each image, while cross-attention explores similarities between paired images.The two attention types operate throughout the encoder rather than being confined to a decoder.
  • Patch embedding: Positional patch embedding uses strided convolutions and depth-wise convolution to capture local information and encode positional information with few parameters.The first stage uses a 7×7 convolution; later stages use 3×3 convolutions, all with stride 2, and depth-wise 3×3 convolution enhances local features.
  • Efficient attention: O(N^2) attention complexity is reduced to O(N^2/R) with Spatial Efficient Attention or O(N) with Linear Attention.These efficient-attention variants allow larger input feature maps to be processed with a pure transformer encoder.
  • Feature fusion and variants: Multi-scale feature fusion generates dense, match-aware features while keeping the decoder lightweight and promoting robust coarse and fine matching.The architecture investigates lite and large models with Linear Attention and Spatial Efficient Attention variants.

4 Experiments

Experiments evaluate MatchFormer across indoor and outdoor pose estimation, image matching, homography estimation, visual localization, and structural and efficiency studies. Results show strong performance and robustness in challenging low-texture settings, while attention interleaving and positional encoding improve matching.

  • Indoor Pose Estimation: 89.5% matching precision is achieved by MatchFormer-large-SEA on low-texture indoor pose estimation, exceeding SuperGlue by +5.1% and LoFTR by +1.6%.The evaluation reports pose-error AUC at 5°, 10°, and 20° thresholds alongside matching precision.
  • Attention Analysis: Cross-attention learns similarity across paired images and progressively narrows the matching range, while self-attention detects image features and supports long-range dependencies.Visualization across four stages shows self-attention connecting local and semantic information as depth increases.
  • Indoor Pose Estimation: MatchFormer improves low-texture indoor matching by capturing global information, producing more matches and lower maximum angle and translation errors.The hierarchical transformer and interleaved attention provide a larger receptive field than CNN-based methods.
  • Outdoor Pose Estimation: 97.55% matching precision is reached by MatchFormer-lite-LA in outdoor pose estimation, while MatchFormer-large-LA achieves state-of-the-art AUC and precision.The memory-efficient LA variant is recommended for high-resolution outdoor scenes because large-SEA can cause an out-of-memory issue.
  • Image Matching, Homography, and Localization: MatchFormer is evaluated on HPatches image matching and homography estimation and on InLoc visual localization using established benchmark metrics and pipelines.HPatches reports MMA and match or feature counts for image matching, corner-error accuracy for homography, and correctly localized queries for InLoc.
  • Structural Study: Interleaved self- and cross-attention remains effective when models are trained with 10% of ScanNet data, supporting robustness under reduced training resources.The ablation compares backbone structures, attention arrangements, and patch embedding modules.
  • Structural Study: +5.2% matching precision is obtained with transformer self-attention over a convolutional extractor, while adding cross-attention yields a further +3.1% gain.Interleaving self- and cross-attention across multi-scale stages improves overall performance, and PosPE adds +0.8% over standard positional encoding.

5 Conclusions

MatchFormer combines feature extraction and feature similarity learning synchronously through a matching-aware encoder, replacing a complex matching decoder with lightweight multi-scale fusion. It achieves state-of-the-art performance across indoor and outdoor pose estimation, HPatches matching and homography estimation, and InLoc localization.

  • 5 Conclusions: MatchFormer interleaves self- and cross-attention in a matching-aware encoder to perform feature extraction and feature similarity learning synchronously.Its lightweight FPN-like decoder fuses multi-scale features instead of using the complex decoder common in extract-to-match methods.
  • 5 Conclusions: MatchFormer achieves state-of-the-art performance on ScanNet, MegaDepth, HPatches image matching and homography estimation, and InLoc visual localization.The reported benchmarks cover indoor and outdoor pose estimation, image matching, homography estimation, and visual localization.

A Implementation Details

MatchFormer uses a four-stage hierarchical Transformer that interleaves self- and cross-attention, then performs coarse-to-fine matching on multi-scale features.

  • Architecture: The hierarchical Transformer has four stages, positional patch embedding, three attention layers per stage, increasing channels, and progressively lower resolutions.The large version uses resolutions from 1/2 to 1/16, while the lite version uses 1/4 to 1/32.
  • Attention arrangement: 81.8% precision is reached when three stages include cross-attention, while excessive cross-attention usage reduces performance.Using only self-attention limits the encoder’s matching capacity.
  • Attention arrangement: Each encoder stage interleaves self-attention for feature extraction with cross-attention for feature matching.The implementation represents attention choices with per-stage cross_flags, using [False, False, True] in all four stages.
  • Matching pipeline: The decoder fuses multi-scale features, which are then passed to coarse-to-fine matching.Coarse matching uses low-resolution feature pairs, while fine matching crops paired windows and estimates local correspondences.
  • Matching pipeline: Coarse matching computes probabilities with a two-dimensional softmax over similarity scores, followed by fine matching within paired windows.The similarity uses an inner product and temperature coefficient; fine matching maps the estimated correspondence back to the original resolution.

B Indoor Pose Estimation.

MatchFormer is evaluated for indoor matching under reduced training data and challenging scene conditions, where qualitative results emphasize dense matching in difficult views.

  • Robustness: MatchFormer maintains consistent indoor pose-estimation performance across training-set sizes of 10%, 30%, 50%, 70%, and 100%.The evaluation compares MatchFormer-large-LA with LoFTR on ScanNet under progressively reduced training data.
  • Qualitative comparison: MatchFormer produces dense feature matching in challenging indoor scenes with large viewing-angle changes.The qualitative comparison includes SuperGlue, LoFTR, and MatchFormer models trained with either 10% or all ScanNet data.

C Outdoor Pose Estimation

On MegaDepth outdoor scene matching, MatchFormer-large-LA outperforms LoFTR and SuperGlue, while lite variants perform comparably to those baselines.

  • Qualitative comparison: MatchFormer-large-LA outperforms LoFTR and SuperGlue in outdoor matching performance on MegaDepth.MatchFormer-lite-SEA and MatchFormer-lite-LA are reported to be on par with both comparison methods.

D Homography Estimation

On HPatches homography estimation, MatchFormer-large-LA is compared qualitatively with LoFTR and SuperGlue, emphasizing matching density and confidence.

  • Qualitative comparison: MatchFormer produces denser and more confident matching than SuperGlue on HPatches.It also yields more matches than LoFTR in the qualitative comparison.

E Image Matching

The HPatches evaluation follows Patch2Pix’s setup across 108 sequences and reports matching accuracy, matches, and features under pixel thresholds.

  • The evaluation uses the same 108 HPatches sequences as Patch2Pix, including viewpoint-change sequences.Each sequence contains six images.
  • Mean matching accuracy is reported at thresholds from pixels.
  • The evaluation also reports the number of matches and features.
  • Images are resized to 1024, with a matching threshold of 0.2 and a RANSAC threshold of 2 pixels.

F InLoc Visual Localization

The InLoc evaluation follows Patch2Pix on 40 retrieval pairs and uses temporal consistency and RANSAC-based pose estimation settings.

  • The InLoc evaluation uses the same first 40 retrieval pairs as Patch2Pix.
  • A temporal consistency check is applied to limit the retrievals.
  • Images are adjusted to 1024 pixels on the long side for evaluation.
  • Pose estimation uses a RANSAC threshold of 48 pixels.
  • MatchFormer-large-LA is evaluated qualitatively on the InLoc benchmark.

G Limitations and Future Work

MatchFormer uses different efficient attention modules for indoor and outdoor scenes, while a uniform module and alternative decoder remain future work.

  • Attention modules: MatchFormer employs spatial efficient attention (SEA) and linear attention (LA) for indoor and outdoor scenes.The two attention types provide different computational reductions and feature-extraction abilities.
  • Attention modules: LA is more suitable for outdoor scenes with dense high-resolution input, whereas SEA is more appropriate indoors with sparse low-resolution input.
  • Future work: Developing one efficient attention module for both indoor and outdoor inputs with different resolutions is left for future work.
  • Future work: The efficient FPN-like decoder combines match-aware feature maps, while an alternative decoder such as an MLP-decoder could be explored.
Loading 2203.09645v3…