Source-linked AI summary
LoFTR: Detector-Free Local Feature Matching with Transformers
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, Xiaowei Zhou
TL;DR
Detector-based local matching can miss repeatable points in indistinctive regions. LoFTR uses globally conditioned self- and cross-attention to establish dense coarse matches and refine them finely, achieving state-of-the-art results across relative pose estimation and visual localization. Its CNN backbone and downsampling keep computation manageable, but the method depends on CNN locality and translation-equivariance inductive biases.
Problem
Detector-based matching can fail to produce repeatable interest points in poor-texture, repetitive, blurred, or changed-view regions.
Method
LoFTR uses a detector-free coarse-to-fine pipeline with Transformer self- and cross-attention that makes dense features context- and position-dependent.
Results
LoFTR achieves state-of-the-art performance on relative pose estimation and visual localization across multiple datasets, including first place among published methods on two visual-localization benchmarks.
Takeaways & Limitations
LoFTR provides a detector-free direction for local image matching and produces high-quality matches in low-texture or repetitive regions.
Takeaways & Limitations
LoFTR relies on CNN translation-equivariance and locality for local feature extraction, while downsampling is needed to keep computation manageable.
Abstract
from arXiv · showhide
We present a novel method for local image feature matching. Instead of performing image feature detection, description, and matching sequentially, we propose to first establish pixel-wise dense matches at a coarse level and later refine the good matches at a fine level. In contrast to dense methods that use a cost volume to search correspondences, we use self and cross attention layers in Transformer to obtain feature descriptors that are conditioned on both images. The global receptive field provided by Transformer enables our method to produce dense matches in low-texture areas, where feature detectors usually struggle to produce repeatable interest points. The experiments on indoor and outdoor datasets show that LoFTR outperforms state-of-the-art methods by a large margin. LoFTR also ranks first on two public benchmarks of visual localization among the published methods.
1. Introduction
Local feature matching supports core 3D vision tasks, but detector-based pipelines can fail in low-texture or repetitive regions because they lack repeatable interest points. LoFTR addresses this with detector-free, Transformer-based coarse-to-fine matching and reports strong performance across matching and localization tasks.
- Local feature matching underpins structure from motion, simultaneous localization and mapping, and visual localization.
- Detector-based pipelines first detect interest points, describe their neighborhoods, and then find point-to-point correspondences.
- Poor texture, repetitive patterns, viewpoint or illumination changes, and motion blur can prevent detectors from finding repeatable interest points, especially indoors.
- LoFTR uses self- and cross-attention Transformers to process dense features, establish coarse matches at 1/8 image resolution, and refine confident matches to sub-pixel precision.
- The Transformer provides global receptive fields and positional conditioning, enabling matches in indistinctive low-texture or repetitive regions.
- LoFTR outperforms detector-based and detector-free baselines by a large margin and ranks first among published methods on two public visual-localization benchmarks.
2. Related Work
Prior local matching methods largely rely on detected interest points or dense descriptors, while LoFTR combines detector-free matching with Transformer attention to address indistinctive regions and computational cost.
- Traditional and learning-based detector methods extract interest points and descriptors, then commonly match them with nearest-neighbor search.
- SuperGlue uses a graph neural network with self- and cross-attention to match detected points, but remains limited by detector repeatability and attention only over detected points.
- LoFTR instead uses a detector-free design with Transformer attention to process dense local features and reduce computation with an efficient attention variant.
- Detector-free methods remove feature detection and directly produce dense descriptors or dense feature matches, often followed by nearest-neighbor matching.
- Transformer computation grows quadratically with input-sequence length, motivating efficient variants for vision applications.
3. Methods
LoFTR extracts multilevel CNN features, transforms them with self- and cross-attention, establishes coarse dense matches, and refines selected matches to sub-pixel accuracy.
- 3.1. Local Feature Extraction: LoFTR extracts coarse features at 1/8 and fine features at 1/2 of the original image dimension using a convolutional architecture with FPN.The CNN provides multilevel features from both images, while downsampling helps keep LoFTR computation manageable.
- 3.2. Local Feature Transformer (LoFTR) Module: The LoFTR module transforms image features into position- and context-dependent representations that are easier to match.It processes the dense local features extracted by the convolutional backbone.
- 3.2. Local Feature Transformer (LoFTR) Module: Positional encoding is added to backbone features so transformed representations contain unique position information useful for matching indistinctive regions.The paper uses a 2D positional encoding and adds it to the backbone output once.
- 3.2. Local Feature Transformer (LoFTR) Module: Self-attention and cross-attention layers are interleaved Nc times, with self-attention operating within an image and cross-attention between the two images.Transformer attention provides global receptive fields, while linear attention reduces the sequence-length complexity from O(N^2) to O(N).
- 3.3. Establishing Coarse-level Matches: Coarse matches are obtained from a score matrix using optimal transport or dual-softmax, then filtered by a confidence threshold and mutual-nearest-neighbor criterion.The filtering step is intended to remove possible outlier coarse matches.
- 3.4. Coarse-to-Fine Module: Each selected coarse match is refined with local correlation windows, producing a heatmap whose expectation yields a final correspondence at sub-pixel accuracy.The final objective combines coarse-level and fine-level losses, with coarse supervision derived from camera poses and depth maps.
4. Experiments
LoFTR is evaluated on homography estimation, indoor and outdoor pose estimation, and visual localization across challenging datasets. It consistently outperforms competing methods, with detector-free matching and Transformer-based global context supporting performance in difficult regions.
- 4.1. Homography Estimation: LoFTR notably outperforms other baselines under all HPatches corner-error thresholds, with a larger gap at stricter correctness thresholds.The authors attribute this to more match candidates, Transformer-based global receptive fields, and sub-pixel coarse-to-fine refinement.
- 4.2. Relative Pose Estimation: LoFTR achieves the best indoor pose-estimation accuracy against all competitors on ScanNet.Models using optimal transport or dual-softmax perform comparably, and the MegaDepth-trained LoFTR† model also exceeds DRC-Net† by a large margin.
- 4.2. Relative Pose Estimation: 61% at AUC@10° separates LoFTR from DRC-Net on MegaDepth outdoor pose estimation, while LoFTR exceeds SuperGlue by 13% at AUC@10°.LoFTR-DS performs better than LoFTR-OT on MegaDepth.
- 4.3. Visual Localization: LoFTR-DS outperforms all baselines in the local-feature visual-localization track, while LoFTR-OT outperforms all published methods on InLoc.InLoc includes appearance changes, texture-less areas, and symmetric or repetitive elements; Aachen performance is comparable to SuperPoint plus SuperGlue at night but slightly worse by day.
- 4. Experiments: Qualitatively, LoFTR produces more correct matches and fewer mismatches than SuperGlue and DRC-Net indoors and outdoors.The comparisons include low-texture regions and large viewpoint and illumination changes.
5. Conclusion
The paper concludes that LoFTR provides accurate semidense matching through a detector-free, Transformer-based coarse-to-fine approach. Its experiments show state-of-the-art relative pose estimation and visual localization across multiple datasets.
- 5. Conclusion: LoFTR establishes accurate semidense matches with Transformers in a coarse-to-fine, detector-free pipeline.Self- and cross-attention make local features context- and position-dependent.
- 5. Conclusion: Context- and position-dependent features support high-quality matches in low-texture or repetitive regions.The conclusion identifies these regions as a central setting where the approach is effective.
- 5. Conclusion: LoFTR achieves state-of-the-art performance on relative pose estimation and visual localization across multiple datasets.The paper presents detector-free matching as a new direction that may extend to more challenging scenarios such as severe seasonal changes.