PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Feature Matching Using Deep Learning: A Survey

Shibiao Xu, Shunpeng Chen, Rongtao Xu, Changwei Wang, Peng Lu, Li Guo

arXiv:2401.17592v2cs.CVcs.AI

TL;DR

Local feature matching remains difficult under scale, viewpoint, illumination, pattern, and texture variation. This survey organizes recent deep-learning methods and reviews their applications, datasets, and evaluation; it reports that these methods have demonstrated impressive performance across matching tasks and benchmarks.

  • Problem

    Scale, viewpoint, illumination, repeated patterns, and texture variation make precise local correspondences difficult to establish across images of the same physical space.

    Source: §1 · p. 1

  • Method

    The survey reviews deep-learning matching methods, grouping them as detector-based or detector-free and covering datasets, evaluation standards, and applications.

    Source: §1 · p. 3, §8 · p. 23

  • Results

    The reviewed algorithms have demonstrated impressive performance in various local feature-matching tasks and benchmark tests.

    Source: §8 · p. 23

  • Takeaways & Limitations

    The survey identifies challenges and future research directions as guidance for further work in local feature matching.

    Source: §1 · p. 3

    Existing local feature-matching datasets have significant gaps that need to be addressed.

    Source: §5.6 · p. 15

Abstract

from arXiv · show

Local feature matching enjoys wide-ranging applications in the realm of computer vision, encompassing domains such as image retrieval, 3D reconstruction, and object recognition. However, challenges persist in improving the accuracy and robustness of matching due to factors like viewpoint and lighting variations. In recent years, the introduction of deep learning models has sparked widespread exploration into local feature matching techniques. The objective of this endeavor is to furnish a comprehensive overview of local feature matching methods. These methods are categorized into two key segments based on the presence of detectors. The Detector-based category encompasses models inclusive of Detect-then-Describe, Joint Detection and Description, Describe-then-Detect, as well as Graph Based techniques. In contrast, the Detector-free category comprises CNN Based, Transformer Based, and Patch Based methods. Our study extends beyond methodological analysis, incorporating evaluations of prevalent datasets and metrics to facilitate a quantitative comparison of state-of-the-art techniques. The paper also explores the practical application of local feature matching in diverse domains such as Structure from Motion, Remote Sensing Image Registration, and Medical Image Registration, underscoring its versatility and significance across various fields. Ultimately, we endeavor to outline the current challenges faced in this domain and furnish future research directions, thereby serving as a reference for researchers involved in local feature matching and its interconnected domains. A comprehensive list of studies in this survey is available at https://github.com/vignywang/Awesome-Local-Feature-Matching .

1. Introduction

Local feature matching must establish precise correspondences despite substantial changes in how the same physical space appears across images. This survey reviews learning-based methods, organizing them by detector use and covering their applications, datasets, evaluation, and research needs.

1. Introduction

  • Local correspondences underpin tasks including image fusion, visual localization, Structure from Motion, SLAM, optical flow, and image retrieval.The introduction presents these applications as examples of tasks built on matching similar features across images.Source: §1 · p. 1
  • Scale, viewpoint, illumination, repeated patterns, and texture variation make precise correspondences difficult even when images depict the same physical space.The introduction identifies accuracy and dependability as continuing challenges.Source: §1 · p. 1
  • Conventional matching pipelines comprise feature detection, description, matching, and geometric transformation estimation, while handcrafted approaches were constrained by researcher-imposed prior knowledge.Before deep learning, many methods addressed only one or several pipeline stages.Source: §1 · p. 1
  • Detector-based pipelines depend on detected and described sparse keypoints, whereas detector-free methods use image context and match end to end.The survey says detector-based efficacy depends substantially on the keypoint detectors and descriptors.Source: §1 · p. 2
  • The survey reviews deep-learning local matching methods proposed since 2018 and organizes them into detector-based and detector-free categories.It follows the prevalent image-matching pipeline in structuring its review.Source: §1 · p. 3
  • It examines local matching deployments in Structure from Motion, remote-sensing registration, and medical registration, and discusses multimodal matching in connection with information fusion.The survey also considers traditional methods alongside deep learning, dataset gaps, and large foundation models.Source: §1 · p. 3
  • The survey classifies datasets by computer-vision task and analyzes evaluation metrics alongside quantitative comparisons of key matching methods.The task-based organization is intended to address specific research requirements within each domain.Source: §1 · p. 3
  • Earlier surveys focused on manual methods, while newer reviews had not timely covered recent learning-based literature.The survey presents its review as a more recent and comprehensive overview.Source: §1 · p. 3

2. Detector-based Models

The survey organizes detector-based matching into four method families and reviews how their pipelines learn, combine, or contextualize keypoints and descriptors. It covers supervised and weakly supervised approaches, joint and reversed detection-description workflows, and graph-based matching.

2. Detector-based Models

  • Detector-based methods match sparsely distributed keypoints using detection and description, and are grouped as Detect-then-Describe, Joint Detection and Description, Describe-then-Detect, or Graph Based.Traditional pipelines detect keypoints, describe them, then establish correspondences; deep-learning methods extend these approaches.Source: §2 · pp. 3–4

2.1. Detect-then-Describe

  • In Detect-then-Describe, keypoints are detected first and descriptors are extracted from centered patches, with metric learning bringing similar points together in feature space.Detectors often focus on low-level structures in small regions, while descriptors capture higher-level information over larger patches.Source: §2.1 · p. 4

2.1.1. Fully-Supervised

  • Fully supervised work develops CNN-learned descriptors through geometric and similarity constraints, while context-aware methods incorporate keypoint, local, and regional information.Frequency-domain descriptors such as RIFT target cross-modal matching; SRIFT adds a nonlinear diffusion scale space for scale and rotation invariance.Source: §2.1.1 · pp. 5–6

2.1.2. Weakly Supervised and Others

  • Weakly and semi-supervised methods use camera poses, geometric constraints, or alignment-based rewards to learn features without densely annotated labels.AffNet emphasizes descriptor matchability alongside repeatability, while CAPS uses epipolar constraints and differentiable matching layers.Source: §2.1.2 · pp. 6–7

2.2. Joint Detection and Description

  • Joint Detection and Description learns both tasks within one model, allowing information from detection and description to be shared during optimization.Examples include SuperPoint, which predicts pixel-level keypoints and descriptors, and D2Net, which shares parameters between the tasks.Source: §2.2 · p. 7

2.3. Describe-then-Detect

  • Describe-then-Detect first describes local image regions and then selects keypoints from those descriptors.D2D generates dense descriptors before selecting keypoints; weakly supervised variants include decoupled training and shared coupling bridges.Source: §2.3 · p. 8

2.4. Graph Based

  • Graph-based methods treat keypoints as nodes and use self- and cross-attention to exchange visual and geometric context, producing matches from a soft assignment matrix.SuperGlue uses optimal transport for partial assignment, but its complexity grows quadratically with keypoint count; SGMNet and ClusterGNN reduce redundant processing through sparse graphs or clustering.Source: §2.4 · p. 8

3. Detector-free Models

Detector-free methods remove the detector and produce dense matches, with CNN, Transformer, and patch-based approaches offering different ways to use image context and refine correspondences. The survey describes their mechanisms and representative methods.

3. Detector-free Models

  • Detector-free methods eliminate feature detectors and extract descriptors on a dense image grid, enabling dense matches and repeatable keypoints in challenging image pairs.Detection-based approaches can struggle with substantial viewpoint changes and textureless regions.Source: §3 · p. 9

3.1. CNN Based

  • CNN-based methods use correlation or cost volumes to find neighborhood consistency, with NCNet analyzing candidate correspondences in four-dimensional space without a global geometric model.Sparse-NCNet applies sparse convolutions to reduce memory consumption and execution time.Source: §3.1 · p. 9, Figure 6 · p. 9

3.2. Transformer Based

  • Transformer-based matching brings global receptive fields and long-range dependencies to matching, including dense matches in low-texture regions with LoFTR.Aspanformer adapts local attention span by position; SE2-LoFTR adds a steerable CNN for translation and rotation equivariance.Source: §3.2 · pp. 9–10

3.3. Patch Based

  • Patch-based matching compares descriptors from local image regions to accommodate large displacements, then refines point correspondences within matched areas.Patch2Pix uses a two-stage detection-refinement strategy, while PATS supports multiple-to-multiple relationships between patches.Source: §3.3 · p. 11, Figure 7 · p. 11

4. Local Feature Matching Applications

The survey traces local feature matching across SfM, remote sensing, and medical image registration, describing methods and their application-specific challenges. It highlights both advances and unresolved integration or data-generalization needs.

4.1. Structure from Motion

  • SfM is shifting from sparse feature identification toward dense matching, including detector-free methods that defer keypoint selection and show efficacy in texture-poor scenes.Integrating these methods into contemporary SfM systems remains challenging.Source: §4.1 · p. 11

4.2. Remote Sensing Image registration

  • Remote-sensing registration combines learning with traditional methods, modality transformation, or direct transformation-parameter regression to address geometric distortions and nonlinear radiometric disparities.Methods include CNN-SIFT combinations, coarse-to-fine registration, and networks for SAR-optical matching; dataset construction and generalization remain challenges.Source: §4.2 · pp. 11–12

4.3. Medical Image Registration

  • Medical registration covers motion estimation and 2D-3D alignment, with deep-learning approaches addressing anatomical motion and registration ambiguity.Examples include one-shot methods for 3D/4D motion tracking and self-supervised or two-stage frameworks for X-ray–CT registration.Source: §4.3 · pp. 12–13

5. Local Feature Matching Datasets

The survey groups local feature matching datasets by downstream task and describes their contents and evaluation scope. It identifies gaps in environmental, dynamic-scene, and application-specific coverage.

5. Local Feature Matching Datasets

  • The survey organizes datasets into image matching, relative pose estimation, visual localization, optical flow estimation, and SfM groups.The groups reflect downstream tasks used to evaluate local feature matching methods.Source: §5 · p. 13

5.1. Image Matching Datasets

  • ZEB provides 46,000 cross-domain evaluation pairs across five overlap ratios, while the IMC series adds datasets reflecting varied real-world matching challenges.ZEB combines eight real-world and four simulated datasets; IMC 2023 includes UAV-to-ground, day-night, repeated-pattern, wiry-object, and scale-change challenges.Source: §5.1 · pp. 13–14

5.2. Relative Pose Estimation Datasets

  • Relative-pose datasets cover varied matching conditions, including indoor weak textures in ScanNet, extreme viewpoints in MegaDepth, and wide-baseline acquisition differences in WxBS.ScanNet includes ground truth and depth images; MegaDepth provides known poses and depth information.Source: §5.2 · p. 14

5.3. Visual Localization Datasets

  • Visual-localization benchmarks include day-night, indoor temporal-change, and varied-weather conditions, while LaMAR targets realistic AR settings with multisensor data.LaMAR includes images, depth, and IMU streams across indoor and outdoor environments.Source: §5.3 · p. 14

5.4. Optical Flow Estimation Datasets

  • KITTI is an image-matching dataset collected in urban traffic: KITTI-2012 has 194 training and 195 test pairs at 1226×370; KITTI-2015 has 200 each at 1242×375 and presents dynamic-scene, complex-scenario challenges.The dataset includes sparse ground-truth disparities obtained using a laser scanner.Source: §5.4 · pp. 14–15

5.5. Structure from Motion Datasets

  • SfM datasets span RGB-D indoor sequences, controlled multiview stereo scenes, realistic reconstruction benchmarks, and synthetic scenes with extensive ground truth.Examples include SUN3D, DTU, Tanks and Temples, ETH3D, and ENRICH.Source: §5.5 · p. 15

5.6. Dataset Gaps and Future Needs

  • Dataset gaps include limited coverage of extreme weather, highly dynamic environments, and domains such as underwater or aerial imagery.The survey calls for specialized datasets that address these gaps and evolving application needs.Source: §5.6 · pp. 15–16

6. Performance Review

The survey explains task-specific evaluation metrics and reports that its benchmark review compiles published results under stated evaluation conditions. It covers matching, geometry, localization, flow, and reconstruction measures.

6.1.1. Image Matching

  • Image-matching metrics include repeatability, M-score, MMA, feature and match counts, and PCK, measuring detector consistency or matching performance.Repeatability divides matching feature areas by the smaller detected-feature count and expresses the result as a percentage.Source: §6.1.1 · p. 16

6.1.2. Homography Estimation

  • Homography estimation evaluates corner error against ground-truth warps and uses AUC over error thresholds to quantify matching accuracy and stability.The cited passage gives typical corner-error thresholds of 1 to 10 pixels.Source: §6.1.2 · p. 16

6.1.3. Relative Pose Estimation

  • Relative-pose evaluation measures rotation and translation angular deviations at thresholds and commonly reports AUC, matching accuracy, and matching score.The maximum translation and angular errors are usually noted as pose error.Source: §6.1.3 · p. 16

6.1.4. Visual Localization

  • Visual-localization evaluation counts images successfully positioned within a predefined tolerance and often reports cumulative pose-error AUC.The protocol uses custom features with an image-registration framework such as COLMAP.Source: §6.1.4 · p. 16

6.1.5. Optical Flow Estimation

  • Optical-flow evaluation includes AEPE, flow outlier ratio, and PCK; AEPE averages Euclidean correspondence error over valid target-image pixels.Flow outliers exceed either 3 pixels or 5% of ground-truth flow.Source: §6.1.5 · pp. 16–17

6.1.6. Structure from Motion

  • SfM evaluation uses reconstruction measures such as registered images, sparse points, observations, feature-track length, and reprojection error, alongside ETH3D pose and triangulation metrics.ETH3D reports pose-error AUC and Accuracy and Completeness at distance thresholds.Source: §6.1.6 · p. 17

6.2. Quantitative Performance

  • The quantitative review compiles methods’ reported benchmark performances from original authors or best reported results under the same evaluation conditions.The survey omits results from non-standard benchmarks or databases and from certain subsets of popular test sets.Source: §6.2 · p. 17

7. Challenges and Opportunities

The survey identifies efficiency and adaptability as future directions for local feature matching. It discusses attention costs and adaptive mechanisms for varying matching difficulty.

7. Challenges and Opportunities

  • The survey frames deep-learning matching as having remaining challenges and introduces research directions for advancing image-based algorithms.Source: §7 · p. 17

7.1. Efficient Attention and Transformer

  • Transformer attention exchanges global visual and geometric information, but basic matrix multiplication remains costly with many keypoints.Future work aims to reduce computational complexity while maintaining matching performance for real-time and resource-constrained settings.Source: §7.1 · p. 18

7.2. Adaptation strategy

  • Adaptive strategies can vary network depth and width with overlap or appearance variation, adjust patch allocation and subdivision, and adapt attention spans to matching difficulty.The survey presents these mechanisms as ways to address latency, scale variation, and patch-level geometric inconsistency.Source: §7.2 · p. 18

7.3. Weakly Supervised Learning

Weakly supervised and self-supervised learning aim to reduce reliance on precise correspondence labels, but weak signals and descriptor uncertainty remain challenges. Decoupled training pipelines and weak-match mining are among the approaches reviewed.

7.3. Weakly Supervised Learning

  • Weakly supervised learning can reduce reliance on precise labels, but direct weakly supervised losses can cause noticeable performance drops.Some methods use relative camera poses and epipolar loss; decoupled pipelines train description and detection networks separately until high-quality descriptors are obtained.Source: §7.3 · p. 18
  • Self-supervised methods such as SuperPoint train on image pairs generated through virtual homography transformations, which may not work effectively in complex scenarios.Source: §7.3 · p. 18
  • Chen et al. use feature shape estimation, orientation assignment, and a weak match finder to support self-supervised learning without known matching relationships.The weak match finder increases feature-appearance variability to improve descriptor invariance.Source: §7.3 · pp. 18–19
  • Weak supervision may support training on larger, more diverse datasets and more generalized descriptors, but effectively using its signals and addressing descriptor uncertainty remain challenges.Future directions include finer loss designs, better use of weak signals, and combining weakly and fully supervised methods.Source: §7.3 · p. 19

7.4. Foundation Segmentation Models

The section describes how semantic segmentation and large foundation models can contribute semantic information to local feature matching. It highlights category-agnostic guidance and lightweight adaptation as distinct approaches.

7.4. Foundation Segmentation Models

  • Semantic segmentation models trained on Cityscapes and MIT ADE20k provide semantic information used to enhance detection and description in specific environments.Source: §7.4 · p. 19
  • SAMFeat [42] uses SAM’s category-agnostic semantic knowledge, fine-grained semantic relations, and edge focus to improve local feature description and accuracy.Source: §7.4 · p. 19
  • SelaVPR adapts DINOv2 with lightweight adapters for visual place recognition, matching local features without extensive spatial verification.Source: §7.4 · p. 19
  • The survey presents large foundation models as a complementary approach for feature matching in open-world applications.Their broad pre-training is described as supporting zero-shot generalization across diverse scenarios.Source: §7.4 · pp. 19–20

7.5. Mismatch Removal

Mismatch removal addresses outliers in image matching, where direct matching can be computationally demanding and erratic and descriptor-based pipelines face challenges under multimodal imagery or major viewpoint and lighting changes. The survey reviews learning-driven methods alongside traditional geometric approaches and identifies hybrid methods as a future direction.

7.5. Mismatch Removal

  • Direct point-set registration and graph matching can be computationally demanding and erratic, motivating descriptor-based preliminary matches followed by local and global geometric constraints.Source: §7.5 · pp. 20–21
  • Descriptor-based strategies face constraints with multimodal images and substantial viewpoint or lighting variation.Source: §7.5 · p. 21
  • Traditional RANSAC variants improve outlier-rejection efficiency and accuracy, but face computational-time constraints and limits in non-rigid contexts.ICF is described as effective for geometric distortions.Source: §7.5 · p. 21
  • Learning-driven methods include networks for distinguishing inliers from outliers, clustering correspondences, and refining geometric estimates.Examples include CNe, ACNe, OANet, and NM-Net.Source: §7.5 · pp. 21–22
  • Shape-Former uses ShapeConv to learn sparse matches, supporting outlier estimation and consensus representation.Source: §7.5 · p. 22
  • The survey identifies generalized methods for non-rigid and multimodal settings, hybrid geometric-learning approaches, and tailored architectures or losses as future directions.Source: §7.5 · p. 22

7.6. Deep Learning and Handcrafted Analogies

Recent image-matching methods combine handcrafted principles with deep learning, while the section notes that rotation handling remains a gap for many modern methods. Rotation-equivariant approaches are presented as a way to bridge that gap.

7.6. Deep Learning and Handcrafted Analogies

  • Hybrid Pipeline (HP), HarrisZ+ [275], and Slime combine handcrafted concepts with learning-based matching in different ways.HP preserves rotational invariance for photogrammetric surveys; HarrisZ+ adapts Harris corners, while Slime combines local overlapping planes with global matching constraints.Source: §7.6 · p. 22
  • Modern methods such as LoFTR and SuperGlue can fall short on in-plane rotations, a capability inherently integrated into handcrafted detectors.Source: §7.6 · p. 22
  • SE2-LoFTR and S-TREK exemplify rotation-equivariant approaches intended to bridge the gap under orientation changes.Source: §7.6 · p. 22
  • The survey presents combining classical concepts with modern computational methods as a potential avenue for robust image matching solutions.Source: §7.6 · pp. 22–23

7.7. Utilizing geometric information

Geometric information is presented as a way to improve local matching in difficult scenes, while its deeper modeling and confidence estimation remain open challenges.

7.7. Utilizing geometric information

  • Geometric priors combined with image appearance can improve matching robustness and accuracy in textureless, occluded, or repetitive-pattern scenes, as studies indicate.The methods capture geometric relationships between pixels more accurately.Source: §7.7 · p. 23
  • More profound geometric modeling for large displacements, occlusions, and textureless regions, alongside more reliable confidence estimation, remains a research challenge.Source: §7.7 · p. 23
  • Geometric priors broaden matching beyond appearance similarity to account for object behavior across viewpoints, and dense matching is presented as promising for large displacements and appearance variations.Future geometric matching may increasingly focus on dense feature matching with geometric information and prior knowledge.Source: §7.7 · p. 23

7.8. Advancing Cultural Heritage Preservation

Deep learning methods are described as promising for matching historical imagery despite radiometric and geometric differences, but rotations, scale changes, and computational demands remain obstacles.

7.8. Advancing Cultural Heritage Preservation

  • SuperGlue, LoFTR, and DISK are identified as particularly effective for historical image matching, surpassing classical methods in robustness to severe illumination and viewpoint shifts.Deep learning methods also show resilience to radiometric and geometric discrepancies between historical and contemporary imagery, facilitating accurate co-registration of multi-temporal scenes.Source: §7.8 · p. 23
  • Extensive rotations and scale variations in historical datasets, plus the computational intensity of high-resolution images, remain obstacles, especially for real-time web and VR/AR applications.Source: §7.8 · p. 23
  • Future work can explore evolving methods and adapt current approaches to historical imagery’s distinctive characteristics to improve matching accuracy and efficiency.The paper links this potential to digital preservation and exploration of cultural heritage.Source: §7.8 · p. 23

8. Conclusions

The conclusion surveys deep-learning local matching methods, their benchmark performance, and future challenges. It organizes methods by detector use and reviews datasets and evaluation standards.

8. Conclusions

  • The survey groups deep-learning local matching into detector-based methods, which use keypoint detection and description, and detector-free methods, which generate dense matches from raw-image context.Source: §8 · p. 23
  • It discusses method strengths and weaknesses, datasets, evaluation standards, and quantitative performance on HPatches, ScanNet, YFCC100M, MegaDepth, and Aachen Day-Night.Source: §8 · p. 23
  • The conclusion identifies open challenges and potential research avenues, aiming to inform researchers and guide future work in local feature matching.Source: §8 · p. 23

Related papers

Loading 2401.17592v2…