Source-linked AI summary
ALIKE: Accurate and Lightweight Keypoint Detection and Descriptor Extraction
Xiaoming Zhao, Xingming Wu, Jinyu Miao, Weihai Chen, Peter C. Y. Chen, Zhengguo Li
TL;DR
Existing methods cannot directly optimize keypoint positions through back-propagation because detection is non-differentiable. ALIKE introduces differentiable sub-pixel detection and descriptor extraction with specialized losses, together with a lightweight network; it achieves comparable or superior task performance with substantially lower runtime.
Problem
Existing keypoint detectors use non-differentiable operations, preventing direct back-propagation-based optimization of keypoint positions.
Method
ALIKE uses differentiable sub-pixel keypoint detection, reprojection and dispersity peak losses, neural reprojection error for dense descriptor training, and a lightweight hierarchical-feature network.
Results
ALIKE achieves comparable or superior results on homography estimation, camera pose estimation, and visual (re-)localization while reducing inference time; ALIKE-N reaches 95 FPS at 640×480.
Takeaways & Limitations
The method provides a lightweight local-feature pipeline for real-time visual applications without sacrificing reported performance relative to state-of-the-art approaches.
Takeaways & Limitations
Extreme illumination changes and large viewpoint differences remain failure cases, partly because the shallow network emphasizes low-level descriptors and sacrifices some representation capabilities.
Abstract
from arXiv · showhide
Existing methods detect the keypoints in a non-differentiable way, therefore they can not directly optimize the position of keypoints through back-propagation. To address this issue, we present a partially differentiable keypoint detection module, which outputs accurate sub-pixel keypoints. The reprojection loss is then proposed to directly optimize these sub-pixel keypoints, and the dispersity peak loss is presented for accurate keypoints regularization. We also extract the descriptors in a sub-pixel way, and they are trained with the stable neural reprojection error loss. Moreover, a lightweight network is designed for keypoint detection and descriptor extraction, which can run at 95 frames per second for 640x480 images on a commercial GPU. On homography estimation, camera pose estimation, and visual (re-)localization tasks, the proposed method achieves equivalent performance with the state-of-the-art approaches, while greatly reduces the inference time.
I. INTRODUCTION
ALIKE addresses limitations in learned local-feature extraction by making keypoint localization partially differentiable, stabilizing sub-pixel descriptor training, and using a lightweight network for real-time applications.
- Application context: Keypoints and descriptors are compact representations used for efficient image matching in applications including SLAM and HDRI.The paper motivates learned local features as replacements for handcrafted methods that can be unstable or produce confusable descriptors in complex images.
- Differentiable detection: The proposed differentiable keypoint detection module back-propagates gradients and produces accurate sub-pixel keypoints.Its formulation couples output positions to the score map so keypoint locations can be optimized directly.
- Motivation: Existing score-map methods use NMS, which is non-differentiable and prevents direct optimization of keypoint positions.NMS selects the maximum-scoring pixel, while the surrounding score patch also influences the desired keypoint location.
- Descriptor extraction: Sub-pixel descriptors are sampled from dense descriptor maps and trained with neural reprojection error for more stable convergence than triplet loss.Neural reprojection error covers the entire descriptor map rather than only sampled keypoints.
- Efficient architecture: A lightweight CNN concatenates multi-level features to improve running efficiency while retaining localization accuracy and representation capabilities.The network is designed for keypoint detection and descriptor extraction in real-time applications.
B. Score map based methods
The method estimates score and descriptor maps with a lightweight hierarchical-feature network, then uses partially differentiable local-window processing to produce sub-pixel keypoints and descriptors.
- Network architecture: The network estimates a score map S and descriptor map D, extracts sub-pixel keypoints from S, and samples corresponding descriptors from D.The feature extraction head operates at the original image resolution.
- Differentiable keypoint detection: NMS selects local score maxima but is non-differentiable, so conventional score-map methods cannot directly optimize keypoint positions through back-propagation.The output position is decoupled from the score map in the conventional formulation.
- Differentiable keypoint detection: DKD first applies NMS and thresholding, then uses softargmax over local N × N windows to obtain sub-pixel coordinates.The local scores are normalized with softmax before computing the expected keypoint position.
- Differentiable keypoint detection: The DKD module is partially differentiable because gradients flow through the local soft-coordinate term to scores within each keypoint’s window.Optimizing the output position is therefore equivalent to optimizing local score values.
C. Learning accurate keypoints
The paper trains accurate keypoints by combining differentiable geometric warping with losses that directly constrain sub-pixel locations and score peaks.
- Learning accurate keypoints: Reprojection and dispersity peak losses are introduced to directly optimize keypoint positions and ensure maximal scores at those positions.Together they target precise, repeatable locations such as corners.
- Learning accurate keypoints: For two images, detected keypoints are warped between images using a differentiable warp function and matched to corresponding detected keypoints.The warp may be a homography, 3D perspective projection, or optical flow.
- Learning accurate keypoints: The reprojection distance compares each warped keypoint with its closest detected correspondence within a threshold of thgt pixels.This distance is used to construct the reprojection-based training objective.
1) Reprojection loss:
The symmetric reprojection loss adjusts differentiably extracted keypoints so warped points approach their corresponding detections, also encouraging repeatability.
- Reprojection loss: The reprojection loss is formulated symmetrically for the two image directions.The formulation is inspired by symmetric epipolar distance.
- Reprojection loss: Minimizing reprojection loss pulls warped keypoints toward corresponding keypoints by adjusting scores within their local windows.Because DKD is differentiable, keypoint positions are adjusted in a single optimization step.
2) Dispersity peak loss:
The dispersity peak loss regularizes local score patches using their spatial distribution, aligning soft keypoint optimization with NMS and concentrating scores around the keypoint.
- The reprojection loss optimizes local-window scores through the soft detected keypoint term.
- A peaky score patch places high scores at the keypoint and low scores around it, stabilizing positions when the NMS window shifts.
- In the toy example, distributions with identical max-mean statistics receive different dispersity peak losses because their spatial distributions differ.
- The proposed dispersity peak loss accounts for score spatial distribution rather than only maximum and average values.It produces higher scores at the keypoint and lower scores farther away.
D. Learning discriminative descriptor
The method trains sub-pixel descriptors with neural reprojection error by comparing dense reprojection and matching probability maps, providing broader constraints than triplet loss.
- NRE loss trains descriptors against the dense descriptor map rather than only sampled keypoint descriptors.Triplet loss uses the corresponding point and hardest negative, whereas NRE constrains the entire map.
- The reprojection probability map is defined from a reprojected keypoint using bilinear interpolation.Inconsistent warped keypoints outside the image are assigned an outlier probability of 1.
- The matching probability map is obtained by softmax-normalizing descriptor similarities, with an outlier category for absent correspondences.The temperature t_des controls the map’s sharpness.
- NRE minimizes cross-entropy between reprojection and matching probability maps.
- The descriptor loss is defined symmetrically for the two image directions.
E. Learning reliable keypoint
A reliability loss supplements accurate keypoint and descriptor training by identifying keypoints whose matching probabilities are low, especially in low-texture or repetitive regions.
- The reliability loss addresses unreliable keypoints that may occur in non-discriminative low-texture areas.
- Reliability is computed from the matching probability map obtained by normalizing the descriptor similarity map.A sharpness parameter controls the normalization.
- The reliability score is sampled bilinearly at the keypoint position.
- A small reliability score indicates unreliable low-texture or repetitive regions because their overall similarities are high and normalized values are lower.
- Reliability loss is defined over all valid keypoints and symmetrically across the two images.
IV. EXPERIMENTS
The experiments analyze the proposed architecture and losses, then compare the method with state-of-the-art approaches across homography, pose estimation, and visual localization tasks.
- The evaluation includes ablations of network architecture and loss terms, followed by comparisons with state-of-the-art methods.
- The network-configuration table reports channel counts, feature-aggregation layers, parameter counts, and GFLOPS for 640 × 480 images.
- HPatches provides planar illumination and viewpoint scenes with ground-truth homographies for ablation studies and homography estimation.
- IMW2020 supplies the standard pipeline used to compare camera pose estimation methods.
- Aachen Day-Night evaluates descriptors for visual localization using nighttime queries and a daytime-image 3D model.
- The implementation detects 400 keypoints with a window size of N = 5 and samples another 400 non-salient keypoints.
2) Training setups:
ALIKE is trained with differentiable keypoint objectives and descriptor losses, then evaluated through ablations of architecture and loss terms. The study favors reprojection, dispersity-peak, reliability, and NRE losses, while reporting resource and benchmark settings.
- Training setup: Training uses 480 × 480 crops, ADAM, a learning rate warmed up to 3e−3, batch size one, and gradients accumulated over 16 batches.The model converges on an NVIDIA Titan RTX in about two days.
- Evaluation metrics: The evaluation reports repeatability, matching score, mean matching accuracy, and mean homography accuracy from reprojection-based keypoint and match criteria.Ablations evaluate MMA@3 and MHA@3 on HPatches with up to 5000 detected keypoints.
- Loss ablation: Reprojection loss increases all reported metrics in the compared configurations, while dispersity peak loss improves homography accuracy but decreases repeatability and matching accuracy.Dispersity peak loss makes score maps more peaky, improving localization certainty; reprojection-loss effects are shown visually and quantitatively.
- Reported comparisons: The reported comparisons cover parameters, GFLOPs, inference FPS, MMA, MHA, and pose-estimation metrics across the benchmark evaluations.The relevant comparison tables include network complexity and homography or pose-estimation results.
- Descriptor-loss ablation: NRE loss is adopted over triplet loss because triplet training with sub-pixel keypoints can converge unstably to a local minimum, whereas NRE provides stable convergence.NRE requires more GPU memory and training time because it computes a dense similarity map.
E. Comparisons with the state-of-the-arts
ALIKE is compared with state-of-the-art methods on homography estimation using HPatches and on pose-related evaluation using IMW2020. The comparisons emphasize downstream accuracy together with computational efficiency.
- Comparison protocol: ALIKE variants are denoted by model size—ALIKE-[T/S/N/L]—with “MS” indicating multi-scale keypoint detection.The comparison includes homography estimation, camera pose estimation, and visual (re-)localization tasks.
- Complexity: ALIKE-N has 318K parameters, fewer than R2D2’s 484K and ALIKE-L’s 653K, while parameter count does not alone determine computational cost.R2D2 is given as an example with 484K parameters but 464.55 GFLOPs.
- Homography estimation: Homography evaluation reports MMA and MHA at stricter one-to-three-pixel thresholds because homography accuracy is treated as the downstream objective.The evaluation follows HPatches protocols and prioritizes MHA alongside stricter-threshold MMA.
- Homography estimation: Compared with ASLFeat(MS), ALIKE-N and ALIKE-L improve MHA@3 by 2.22% and 3.33%, respectively, while reducing computational complexity by about 5.6 and 2.2 times.ASLFeat(MS) is identified as the method with the highest MHA@3 among the comparison baseline methods.
- Camera pose estimation: IMW2020 pose evaluation uses up to 2048 keypoints and reports mAA integrated to 5° and 10°, together with GFLOPs and performance per cost.The protocol uses the best configuration for each method and builtin mutual nearest neighbor matching.
3) Camera pose estimation:
ALIKE achieves strong camera-pose and localization performance while using substantially less computation than several competing methods. Its lightweight models are intended for resource-constrained real-time applications, though extreme matching conditions remain challenging.
- Camera pose estimation: ALIKE outperforms all previous methods except DISK on IMW2020 pose estimation when performance per cost is not considered.The evaluation uses up to 2048 keypoints per image and reports GFLOPs and PPC.
- Camera pose estimation: ALIKE-L and ALIKE-N require 7.91 and 19.68 GFLOPs, respectively, compared with 98.97 GFLOPs for DISK.Considering PPC, ALIKE-N and ALIKE-L are approximately 10 and 5 times more efficient than DISK.
- Visual localization: In visual localization, ALIKE-L maintains superior performance with fewer keypoints, while ALIKE-N achieves comparable localization accuracy using fewer computational resources.The Aachen Day-Night evaluation compares limited top-2048 and unlimited-keypoint settings.
- Visual localization: D2-Net, ASLFeat, and DISK degrade dramatically with limited keypoints, whereas ALIKE-L remains effective without heuristically fine-tuned configurations.This supports the use of ALIKE in resource-constrained applications.
- Limitations: Extreme illumination changes and large viewpoint differences are observed failure cases, reflecting limitations of the shallow lightweight network’s low-level descriptors.In randomly selected challenging cases, ALIKE-N’s matching accuracy is twice that of SuperPoint while using one-third its GFLOPs.