PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed

Yifan Wang, Xingyi He, Sida Peng, Dongli Tan, Xiaowei Zhou

arXiv:2403.04765v2cs.CV

TL;DR

LoFTR provides strong detector-free matching but is inefficient because it transforms entire coarse feature maps, limiting practical large-scale use. Efficient LoFTR reduces redundant attention with adaptive token aggregation and improves refinement with two-stage correlation, achieving ∼2.5 times faster inference than LoFTR while maintaining comparable or better accuracy.

  • Problem

    LoFTR handles challenging image pairs effectively but its large transformer token size limits efficiency for practical applications such as image retrieval and structure from motion.

  • Method

    The method uses aggregated attention with adaptively selected tokens and a two-stage correlation layer for accurate subpixel correspondence refinement.

  • Results

    ∼2.5 times faster than LoFTR, the method achieves comparable or better matching accuracy than competitive detector-free baselines across homography estimation, relative pose recovery, and visual localization.

  • Takeaways & Limitations

    The method provides a semi-dense matching pipeline with considerable efficiency benefits for large-scale or latency-sensitive applications.

  • Takeaways & Limitations

    The method may fail on strong repetitive structures because its focus on local features lacks sufficient global semantic context.

Abstract

from arXiv · show

We present a novel method for efficiently producing semi-dense matches across images. Previous detector-free matcher LoFTR has shown remarkable matching capability in handling large-viewpoint change and texture-poor scenarios but suffers from low efficiency. We revisit its design choices and derive multiple improvements for both efficiency and accuracy. One key observation is that performing the transformer over the entire feature map is redundant due to shared local information, therefore we propose an aggregated attention mechanism with adaptive token selection for efficiency. Furthermore, we find spatial variance exists in LoFTR's fine correlation module, which is adverse to matching accuracy. A novel two-stage correlation layer is proposed to achieve accurate subpixel correspondences for accuracy improvement. Our efficiency optimized model is $\sim 2.5\times$ faster than LoFTR which can even surpass state-of-the-art efficient sparse matching pipeline SuperPoint + LightGlue. Moreover, extensive experiments show that our method can achieve higher accuracy compared with competitive semi-dense matchers, with considerable efficiency benefits. This opens up exciting prospects for large-scale or latency-sensitive applications such as image retrieval and 3D reconstruction. Project page: https://zju3dv.github.io/efficientloftr.

1. Introduction

Image matching supports 3D vision applications, but detector-based methods struggle with challenging image pairs and LoFTR remains inefficient. Efficient LoFTR reduces redundant computation through adaptive token aggregation and improves refinement accuracy with two-stage correlation.

  • Image matching establishes accurate image correspondences for 3D reconstruction, SLAM, and visual localization.
  • Detector-based matchers are efficient but struggle to detect repeatable keypoints under extreme viewpoint changes and in texture-poor regions.
  • LoFTR matches challenging pairs without keypoint detection but suffers from low efficiency because transformers process the entire coarse feature map.
  • Adaptive token aggregation performs attention on compact, selected tokens because local regions share similar attention information.
  • Two-stage correlation first locates pixel-level matches with mutual-nearest-neighbor matching, then refines them to subpixel accuracy within tiny patches.
  • ∼2.5 times faster than LoFTR, the method achieves comparable or better accuracy than competitive detector-free baselines across multiple tasks.Experiments cover homography estimation, relative pose recovery, and visual localization.

2. Related Work

Image matching methods range from detector-based sparse pipelines to detector-free semi-dense and dense approaches. Efficient LoFTR addresses detector-free efficiency by reducing attention tokens while retaining coarse-to-fine matching and semi-dense correspondences.

  • Detector-Based Image Matching: Detector-based methods detect and describe keypoints before matching them, while LightGlue adapts attention to matching difficulty for efficient sparse matching.
  • Detector-Free Image Matching: Detector-free methods directly match images without detected keypoints, producing semi-dense or dense correspondences through correlation-based and coarse-to-fine approaches.
  • Detector-Free Image Matching: LoFTR introduced transformers for detector-free matching and strong long-range modeling, but dense transformation of coarse feature maps makes it inefficient.
  • Efficient LoFTR: Efficient LoFTR transforms coarse features with aggregated self- and cross-attention, adaptively reducing token size before attention.
  • Efficient LoFTR: The pipeline correlates transformed coarse features, uses mutual-nearest-neighbor search for coarse matches, and refines cropped patches to subpixel correspondences.
  • Dense Matching: Dense matchers provide strong robustness but are generally slower than sparse and semi-dense methods, whereas this method targets competitive performance with higher efficiency.
  • Transformer Efficiency: Hardware-specific transformer optimizations are orthogonal to this method and can be adapted into its pipeline for further efficiency improvement.

3. Method

The method uses a coarse-to-fine detector-free matching pipeline, replacing costly dense transformation with aggregated attention and refining matches through a two-stage correlation layer. It also removes dual-softmax during inference for improved efficiency while retaining it during training.

  • The pipeline establishes coarse matches on downsampled feature maps and refines them for high-accuracy correspondences.
  • A lightweight reparameterized single-branch backbone replaces LoFTR’s heavy multi-branch ResNet for more efficient feature extraction.Training uses a multi-branch residual CNN, which is converted losslessly into a single-branch network for inference.
  • Efficient Local Feature Transformation: The method uses vanilla attention on reduced tokens because LoFTR’s dense linear attention remains costly and has reduced representational power.The proposed aggregation reduces the token count by s^2 during the attention phase.
  • Efficient Local Feature Transformation: Aggregated attention combines neighboring query tokens and selects salient key tokens before applying vanilla attention to reduce redundant computation.Relative positional encoding is used during self-attention, and transformed features are upsampled and fused with the original features.
  • Coarse-level Matching: Coarse correspondences are selected from a dual-softmax score matrix under a score threshold and mutual-nearest-neighbor constraint.Dual-softmax is retained for training but can be skipped during inference using the score matrix directly for MNN matching.
  • Subpixel-Level Refinement Module: Two-stage refinement first obtains pixel-level MNN matches and then performs local correlation and expectation for subpixel accuracy.This design avoids the spatial variance introduced when expectation is computed over an entire noisy correlation patch.

4. Experiments

Experiments evaluate the method on homography estimation, relative pose recovery, and visual localization against sparse, semi-dense, and dense baselines. The method maintains competitive accuracy while offering substantial efficiency advantages, including faster relative-pose matching and strong homography results under a restricted match budget.

  • Evaluation Setup: Experiments cover homography estimation, relative pose estimation, and visual localization using HPatches, MegaDepth, ScanNet, InLoc, and Aachen v1.1.The evaluation includes indoor and outdoor scenes with viewpoint changes, illumination changes, repetitive patterns, and texture-less regions.
  • Baselines: The method is compared with sparse keypoint pipelines, semi-dense matchers, and the dense matcher ROMA across the evaluated tasks.Baselines include SuperPoint with NN, SuperGlue, or LightGlue; LoFTR and other semi-dense methods; and ROMA.
  • Relative Pose Estimation: ∼2 times faster than AspanFormer on ScanNet while achieving lower but comparable performance, whereas it outperforms AspanFormer on all MegaDepth metrics.Relative pose accuracy is measured by AUC of pose error at 5°, 10°, and 20° thresholds.
  • Homography Estimation: The method outperforms sparse methods significantly on HPatches and surpasses semi-dense methods with significantly higher efficiency despite restricting semi-dense methods to their top 1000 matches.HPatches reports AUC of corner-point reprojection error at 3 px, 5 px, and 10 px thresholds.
  • Visual Localization: The method achieves competitive visual-localization results across detector-based and detector-free methods, with performance comparable to many accuracy-oriented approaches.InLoc reports DUC1 and DUC2 separately, while Aachen v1.1 reports daytime and nighttime divisions.
  • Ablation Studies: The ablations report that aggregated attention improves efficiency with competitive or better accuracy, while two-stage refinement improves accuracy with negligible latency.Removing the second refinement stage degrades pose accuracy, especially at the strict AUC@5° metric; replacing RepVGG with a multi-branch ResNet reduces efficiency with similar accuracy.

5. Conclusions

The paper introduces a semi-dense local feature matcher that revises LoFTR for efficiency and accuracy. Aggregated attention reduces redundant computation, while two-stage correlation refinement improves correspondence precision, yielding faster matching with better accuracy and comparable efficiency to LightGlue.

  • 5. Conclusions: The method revisits LoFTR to improve both efficiency and matching accuracy in semi-dense local feature matching.Its design targets redundant transformer computation and spatial variance in LoFTR’s refinement stage.
  • 5. Conclusions: Aggregated attention transforms reduced tokens and improves efficiency while retaining competitive performance.The approach addresses redundancy from applying the Transformer to the entire coarse feature map.
  • 5. Conclusions: Two-stage correlation refinement addresses location variance in LoFTR’s refinement design and improves matching accuracy.The layer supports accurate subpixel-level correspondence refinement.
  • 5. Conclusions: ∼2.5 times faster than LoFTR with better matching accuracy, while achieving comparable efficiency to LightGlue.The paper connects these efficiency gains to large-scale and latency-sensitive downstream tasks such as image retrieval and 3D reconstruction.

Supplementary Material

The supplementary discussion explains how aggregated-and-upsampling attention preserves high-resolution information and why the query transformation and refinement design choices matter for matching.

  • Related Designs: Pooling-based prior designs differ because some cannot support cross-attention between non-aligned images or cannot provide the high-resolution features required for matching.The discussion contrasts PoolFormer and MViT with the proposed design.
  • Efficient Feature Transformation: Aggregate-and-upsample attention performs attention on aggregated features, then upsamples before the feed-forward network and fusion with the input feature.Upsampling before fusion is described as important for combining interpolated messages with detailed feature maps.
  • Efficient Feature Transformation: The method uses convolution rather than pooling on Q because salient tokens should not represent neighboring query regions.The transformer is used to enhance non-salient features, while pooling on Q can make texture-less-region attention dominated by salient neighbors.

B. Implementation Details

The implementation uses a RepVGG feature backbone, 2D rotary position encoding, and relative positional attention to transform coarse features efficiently and robustly.

  • Backbone: The four-stage RepVGG backbone outputs 1/8-resolution features for the efficient local feature transformer modules.Stage widths are 64, 64, 128, and 256, with block counts [14].
  • Position Encoding: 2D RoPE computes attention from projected features and their relative spatial coordinates.The attention score uses q_i, k_j, and a rotation matrix R based on coordinate differences.
  • Evaluation Setting: The MegaDepth position-encoding comparison reports averaged running times for high-resolution 1200 × 1200 image pairs.
  • Position Encoding: Relative position encoding emphasizes feature interactions rather than specific locations and is more robust to rotation, translation, and scaling.This robustness is important for matching local features across different views.

C.1. More Ablation Studies

Ablations assess position encoding, token aggregation, image resolution, linear attention, and component contributions, showing efficiency–accuracy trade-offs across settings.

  • Position Encoding: 2D RoPE achieves better performance than sinusoidal position encoding on MegaDepth.
  • Aggregation Range: Using a 4 × 4 aggregation range reduces token size before attention, while smaller ranges produce more tokens and significantly slower matching.Smaller ranges cause only slight performance changes but substantially increase matching time.
  • Qualitative Results: Figure 5 compares qualitative matches against SuperPoint + LightGlue and AspanFormer using dataset-specific error or confidence coloring.Red denotes epipolar error beyond the stated thresholds on ScanNet and MegaDepth; on InLoc, red indicates higher predicted confidence.
  • Image Resolution: Larger test images improve accuracy but slow matching, whereas 640 × 640 images provide the fastest speed with competitive performance.The results support flexible resolution choices for real-world applications.
  • Linear Attention After Aggregation: Linear attention after aggregation provides a minor high-resolution efficiency gain but reduces accuracy.
  • Additional ScanNet Studies: The ScanNet ablation study reports averaged image-pair running times for a comprehensive component analysis.

C.2. More Qualitative Results

Additional qualitative results are presented for ScanNet, InLoc, and MegaDepth.

  • Qualitative Results: More qualitative matching results on ScanNet, InLoc, and MegaDepth are shown in Figure 5.

C.3. Additional Results on other RANSAC setting

Additional experiments examine relative-pose evaluation under LightGlue’s RANSAC setting and compare latency across datasets and methods.

  • Relative Pose Estimation: Following LightGlue’s RANSAC setting, tuned thresholds increase the performance gap between the proposed method and LightGlue under naive RANSAC.The setting uses OpenCV RANSAC and LO-RANSAC with carefully tuned inlier thresholds.
  • Timing Protocol: Running times are averaged over test-dataset image pairs after a 50-pair warm-up on an RTX 3090 GPU and 14-core Xeon Gold 6330 CPU.
  • Optimization: The optimized model uses mixed precision and drops the dual-softmax operator in coarse matching to reduce time cost.
  • Cross-Dataset Latency: Across the reported latency datasets, the speed rankings remain consistent with the main results.The comparisons cover HPatches, Aachen, and InLoc settings.
  • HPatches Latency: On HPatches, both matching and RANSAC latency are lower than LoFTR with a similar number of matches.

E. Limitations and Future Works

The method may fail on image pairs with strong repetitive structures, and the authors identify missing global semantic context as a likely cause and future improvement direction.

  • The method may fail when strong repetitive structures appear, such as different scenes containing the same chair.
  • The authors attribute this failure to the model’s stronger focus on local features, with insufficient global semantic context.
  • Adding high-level context mechanisms is proposed to improve performance on ambiguous scenes.
Loading 2403.04765v2…