Source-linked AI summary
ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer
Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David Mckinnon, Yanghai Tsin, Long Quan
TL;DR
Image matching needs robust correspondences despite viewpoint changes and textureless regions, while existing approaches trade off global context against fine-grained interaction. ASpanFormer uses hierarchical global-local attention with uncertainty-driven adaptive spans guided by flow prediction. It reports state-of-the-art benchmark results and outperforms detector-free and detector-based baselines in two-view pose estimation, while running slightly slower than LoFTR.
Problem
Image matching methods face limitations in extreme conditions and a persistent trade-off between global receptive field and fine-grained interaction.
Method
ASpanFormer is a detector-free Transformer matcher that uses hierarchical global-local attention and uncertainty-driven flow-guided adaptive attention spans.
Results
ASpanFormer achieves state-of-the-art results and outperforms detector-free and detector-based matching baselines in two-view pose estimation.
Takeaways & Limitations
The method captures long-range dependencies and fine-grained local details by adapting context to different matching locations.
Takeaways & Limitations
The proposed method is overall slightly slower than LoFTR because of its more complicated attention operation.
Abstract
from arXiv · showhide
Generating robust and reliable correspondences across images is a fundamental task for a diversity of applications. To capture context at both global and local granularity, we propose ASpanFormer, a Transformer-based detector-free matcher that is built on hierarchical attention structure, adopting a novel attention operation which is capable of adjusting attention span in a self-adaptive manner. To achieve this goal, first, flow maps are regressed in each cross attention phase to locate the center of search region. Next, a sampling grid is generated around the center, whose size, instead of being empirically configured as fixed, is adaptively computed from a pixel uncertainty estimated along with the flow map. Finally, attention is computed across two images within derived regions, referred to as attention span. By these means, we are able to not only maintain long-range dependencies, but also enable fine-grained attention among pixels of high relevance that compensates essential locality and piece-wise smoothness in matching tasks. State-of-the-art accuracy on a wide range of evaluation benchmarks validates the strong matching capability of our method.
1 Introduction
ASpanFormer addresses limitations of detector-based and existing Transformer matching by combining global context with fine-grained local interaction through hierarchical attention and adaptive attention spans. The method adjusts context to matching difficulty and reports state-of-the-art performance across benchmarks.
- Detector-based matching struggles with large viewpoint changes and textureless areas because it relies on keypoint detection and context-limited descriptions.
- Existing detector-free Transformers model long-range dependencies but can lack fine-level local interaction, with attention maps diffusing across large regions.
- ASpanFormer uses hierarchical attention: coarse global attention captures long-range dependencies, while fine local attention focuses around correspondences predicted by dense flow.
- Adaptive attention spans use probabilistic uncertainty to assign smaller contexts to reliable matches and larger contexts to uncertain or impossible matches.
- The framework performs attention at different scales and adaptively adjusts local span using flow-prediction uncertainty.
- The method achieves state-of-the-art results and outperforms detector-free and detector-based baselines in two-view pose estimation.
2 Related Works
Related matching methods trade off global receptive fields against fine-grained interaction, motivating detector-free and hierarchical approaches. ASpanFormer uses flow prediction and uncertainty modeling to guide localized interaction while retaining broader context.
- Detector-free matching directly produces correspondences from image pairs, avoiding keypoint detection and leveraging richer image context.
- Deep detector-free methods include cost-volume and Transformer-based approaches that exploit correlations between image features.
- Global interaction is computationally expensive, so existing methods downsample features or use low-rank projections, limiting fine-grained feature updates.
- Local correlation methods restrict cost volumes to neighborhoods around correspondence estimates to process fine-level features within limited computation.
- Flow maps encode correspondence coordinates, and ASpanFormer regresses a flow map for each attention block to guide local regions and adapt span size.
- The network extracts initial features with a CNN, updates them through iterative GLA blocks, and determines final matches with a matching module.
3 Methodology
ASpanFormer updates CNN features with hierarchical global-local attention, using flow estimates and uncertainty to adapt local search regions before constructing refined correspondences.
- Network overview: The network encodes each image with a CNN, adds positional and two-view context initialization, then iteratively processes features with global-local attention blocks.The Transformer module supplements independently extracted CNN features with long-range and cross-view context.
- Flow-guided attention: Flow maps predict correspondence coordinates and uncertainty, which guide local attention regions instead of serving directly as final correspondences.The model uses uncertainty to adjust subsequent search regions rather than thresholding flow estimates.
- Global-local attention: Each global-local attention block applies global attention at fixed coarse resolution and local attention between medium- and fine-level features.Messages from the three levels are upsampled, concatenated, and fused to update source features.
- Adaptive local attention: For local attention, query maps are partitioned into S × S cells, and each cell attends to w^2 uniformly sampled tokens from a flow-centered rectangle.The rectangle has width n × σ_x and height n × σ_y, while the fixed sample count preserves linear complexity.
- Adaptive local attention: Adaptive spans assign small contexts to high-confidence or distinctive regions and larger contexts to uncertain regions, balancing locality with matching uncertainty.After feature updating, dual-direction softmax scores yield mutual-nearest-neighbor coarse matches, followed by correlation-based refinement.
4 Experiments
Experiments evaluate ASpanFormer on two-view pose estimation, visual localization, ablations, uncertainty estimation, and runtime. It achieves the best reported pose-estimation accuracy across indoor and outdoor scenes, strong localization results, and improvements from both hierarchical and adaptive attention, with slightly higher runtime than LoFTR.
- Evaluation Setup: Experiments cover two-view pose estimation and visual localization across indoor and outdoor datasets, alongside ablations and component analyses.ScanNet and MegaDepth support pose estimation; InLoc and Aachen Day-Night v1.1 support visual localization.
- Evaluation Setup: Pose accuracy is measured by AUC at 5°, 10°, and 20° thresholds, requiring both rotation and translation errors to remain below each threshold.
- Two-view Pose Estimation: ASpanFormer consistently achieves the best accuracy in indoor and outdoor two-view pose estimation.The comparison includes detector-based and detector-free matching methods.
- Visual Localization: On InLoc, ASpanFormer achieves the overall best localization results; on Aachen V1.1, it outperforms all methods except SuperGlue.The authors partially attribute the Aachen limitation to using only coarse matches for database reconstruction.
- Ablation Study: Both hierarchical global-local attention and adaptive attention span improve overall ScanNet performance by a considerable margin.The ablation compares single-level attention, fixed-span multi-level attention, and the full adaptive-span design.
- Understanding ASpanFormer: Flow-regression precision improves across attention iterations and converges after four iterations, while uncertainty decreases for matchable pixels and rises for unmatchable pixels.Textureless regions generally receive larger uncertainty values, indicating a need for larger cross-attention context.
- Runtime Evaluation: ASpanFormer is overall slightly slower than LoFTR because its attention operation is more complicated.Runtime is compared on 100 ScanNet image pairs sized 640×480 using an NVIDIA V100 GPU.
5 Conclusion
The paper concludes that ASpanFormer uses feature hierarchy and adaptive attention spans to capture long-term dependencies and fine-grained local details. It reports state-of-the-art results and anticipates wider real-world application with further engineering optimizations.
- ASpanFormer is a Transformer framework whose feature hierarchy adaptively determines attention spans for long-term dependencies and fine-grained local details.
- The authors report state-of-the-art results and anticipate wider real-world application after further engineering optimizations.
A Implementation Details
Implementation uses LoFTR-compatible feature extraction and training components around four Global-Local Attention blocks. Flow coordinates and uncertainties are regressed per block, while localization follows a hierarchical pipeline with dataset-specific matching procedures.
- Network Implementation: Four Global-Local Attention blocks update features, using coarsest resolutions of 15×20 indoors and 36×36 outdoors.
- Flow and Uncertainty: Each attention block regresses flow coordinates and standard deviations with an MLP; coordinates use sigmoid rescaling, while standard deviations use exponentiated log predictions.The image height and width define the coordinate recovery range.
- Training: Training follows LoFTR's multi-step schedule: linear warm-up during the first epoch, then learning-rate halving every two or three epochs.
- Visual Localization Details: Visual localization follows a hierarchical localization pipeline, with Aachen using coarse-match triangulation and InLoc projecting reference-image matches into 3D through depth maps.InLoc image pairs with fewer than 25 matches are omitted.
- Training: The learning-rate schedule is illustrated across iterations in Fig. 6.
A.5 Some Effective Designs
The paper evaluates several network-design choices, including learnable temperatures, convolutional FFNs, normalized positional encoding, and probabilistic flow supervision.
- Learnable Temperature: Learnable temperatures τf, τm, and τc balance softmax distributions across fine-, medium-, and coarse-level attention with different token counts.The temperatures address the effect of varying token numbers on the final softmax distribution.
- Convolutional FFN: Replacing self attention and MLP-based FFNs with 3 × 3 convolution in the FFN improves overall performance in experiments.The network otherwise uses cross attention for cross-view message passing, without self attention.
- Normalized Positional Encoding: Normalized positional encoding addresses unseen coordinates when testing and training resolutions differ, improving precise localization and boundary awareness.The normalization aligns testing and training positional encodings and is especially important for precise flow prediction.
- Normalized Positional Encoding: The normalized positional encoding ablation uses MegaDepth images resized to 1152 resolution with models trained at 832 resolution.Results are reported in Table 9.
- Flow Loss: Flow supervision formulates the predicted flow parameters as a Gaussian maximum-likelihood objective.The resulting loss is a weighted L2 distance between estimated and ground-truth flows, with a regularizer encouraging lower uncertainty.
C Additional Quantitative Results
Additional experiments evaluate the method on YFCC100M and the Image Matching Challenge 2022 benchmark.
- Additional Quantitative Results: Additional experiments are conducted on YFCC100M and Image Matching Challenge 2022.These results extend the evaluation beyond the main experiments.
C.1 Results on YFCC100M
On YFCC100M, the method is evaluated for two-view pose estimation using landmark image pairs and outperforms all comparative methods.
- Results on YFCC100M: 4000 YFCC100M image pairs are constructed from four landmark sequences, with OpenCV RANSAC used for two-view pose estimation.The RANSAC threshold is 5×10−4 in normalized image coordinate space.
C.2 Results on Image Matching Challenge 2022
On Image Matching Challenge 2022, the method is evaluated under a fixed input resolution and robust fundamental-matrix estimation, consistently outperforming strong baselines.
- Results on Image Matching Challenge 2022: The method consistently outperforms other strong comparative baselines on Image Matching Challenge 2022.Inputs are resized to [1472,832], and OpenCV USAC MAGSAC estimates the fundamental matrix with a 0.2-pixel RANSAC threshold.
- Results on Image Matching Challenge 2022: Table 11 reports the Image Matching Challenge 2022 results.The benchmark evaluation uses fixed-resolution inputs and fundamental-matrix estimation.
D Additional Visualizations
The visualizations compare matching quality, show flow refinement across GLA iterations, and illustrate how uncertainty drives adaptive attention spans.
- Table 11 reports two-view pose estimation results on the IMC 2022 Kaggle benchmark, including results for MatchFormer and QuadTree attention from a Kaggle discussion solution.
- Figure 7 qualitatively compares matches from SuperGlue, LoFTR, and ASpanFormer, with ASpanFormer producing more accurate and denser matches.
- Across GLA iterations, flow predictions are visualized at 1/8 resolution and progressively prune occluded or non-overlapping regions as more blocks update features.
- Figure 9 shows uncertainty heatmaps alongside adaptive spans that are small for distinctive textures and larger for low-texture or uncertain regions.
- Very large attention spans are generated for non-overlapping or occluded areas, preventing attention from falsely focusing on particular regions.