PaperlayerView exact revision on arXiv

Source-linked AI summary

ALIKED: A Lighter Keypoint and Descriptor Extraction Network via Deformable Transformation

Xiaoming Zhao, Xingming Wu, Weihai Chen, Peter C. Y. Chen, Qingsong Xu, Zhengguo Li

arXiv:2304.03608v2cs.CV

TL;DR

Conventional convolutions lack the geometric invariance required for robust descriptors, while dense descriptor extraction is computationally expensive. The paper proposes ALIKED with the Sparse Deformable Descriptor Head, which learns deformable support positions at sparse keypoints and trains them with sparse NRE loss. Experiments report efficient and powerful performance across image matching, 3D reconstruction, and visual relocalization, with limitations under large combined scale and viewpoint changes and on mobile platforms.

  • Problem

    Conventional convolutions do not provide the geometric invariance required for descriptors, and dense descriptor-map extraction incurs redundant computation.

  • Method

    ALIKED uses SDDH to learn deformable support positions, extract descriptors only at sparse keypoints, and relax NRE loss from dense to sparse.

  • Results

    ALIKED achieves excellent performance across image matching, 3D reconstruction, and visual localization while reducing redundant descriptor computations and running time.

  • Takeaways & Limitations

    Sparse deformable descriptors provide a paper-supported route to combining geometric modeling with more efficient descriptor extraction and training.

  • Takeaways & Limitations

    ALIKED may fail when images have significant differences in both scale and viewpoint, and its 32-bit floating descriptors may not suit mobile platforms.

Abstract

from arXiv · show

Image keypoints and descriptors play a crucial role in many visual measurement tasks. In recent years, deep neural networks have been widely used to improve the performance of keypoint and descriptor extraction. However, the conventional convolution operations do not provide the geometric invariance required for the descriptor. To address this issue, we propose the Sparse Deformable Descriptor Head (SDDH), which learns the deformable positions of supporting features for each keypoint and constructs deformable descriptors. Furthermore, SDDH extracts descriptors at sparse keypoints instead of a dense descriptor map, which enables efficient extraction of descriptors with strong expressiveness. In addition, we relax the neural reprojection error (NRE) loss from dense to sparse to train the extracted sparse descriptors. Experimental results show that the proposed network is both efficient and powerful in various visual measurement tasks, including image matching, 3D reconstruction, and visual relocalization.

I. INTRODUCTION

Deep learning methods improve keypoint and descriptor robustness, but dense map extraction and fixed convolutions limit efficiency and geometric invariance. ALIKED addresses these issues with sparse deformable descriptors and sparse NRE training.

  • Motivation: Deep learning methods often extract more robust keypoints and discriminative descriptors than hand-crafted methods through score and descriptor maps.Map-based methods estimate both maps and sample keypoints and descriptors from them.
  • Motivation: Fixed-size vanilla convolutions lack the geometric invariance needed for image matching, while scale and orientation modeling covers only affine transformations.Deformable convolution adjusts offsets for individual convolution pixels to model broader geometric transformations.
  • Contributions: ALIKED relaxes neural reprojection error from dense to sparse by training sparse descriptors with sparse probability vectors.The sparse formulation also reduces redundant computations during training and saves GPU memory.
  • Contributions: SDDH extracts deformable descriptors efficiently by modeling geometric transformations and avoiding redundant computations.Its design uses deformable positions of supporting features for each keypoint rather than a dense descriptor map.
  • Results: Experimental results show excellent ALIKED performance across image matching, 3D reconstruction, and visual localization.These tasks are presented as representative visual measurement applications.

B. Joint Keypoint and Descriptor Learning

Joint keypoint and descriptor learning commonly relies on dense descriptor maps, which are costly. ALIKED instead computes deformable descriptors only at sparse keypoints to improve efficiency while modeling geometric variation.

  • Joint learning: Joint learning methods often estimate score and descriptor maps, detect keypoints from the score map, and sample descriptors from the descriptor map.This map-based pipeline underlies several established methods.
  • ALIKED architecture: ALIKED encodes multi-scale features, aggregates them, and uses a score-map head with differentiable keypoint detection before SDDH descriptor extraction.The SDDH extracts deformable invariant descriptors at detected keypoints.
  • Efficiency: Dense descriptor-map extraction is computationally expensive, motivating sparse descriptor extraction on deformable local features.The stated goal is to improve a lightweight network while saving computational budget.
  • Deformable operations: Deformable convolution introduces learnable kernel offsets, enabling the network to attend to selected sample positions rather than processing dense descriptor locations.The paper connects this sparse sampling philosophy to computationally efficient vision architectures.

III. NETWORK ARCHITECTURE OF ALIKED

ALIKED combines multi-scale feature encoding and aggregation with differentiable keypoint detection, then uses SDDH to extract deformable invariant descriptors at detected keypoints.

  • Overall architecture: ALIKED comprises feature encoding, feature aggregation, and keypoint-and-descriptor extraction components.The architecture aggregates multi-scale features for localization and representation.
  • Feature encoding and aggregation: The feature encoder produces four multi-scale features, while ublocks align and concatenate them into an aggregated feature F.Average pooling expands receptive fields and improves computational efficiency in the encoder.
  • Keypoint detection: The SMH transforms aggregated feature F into a sigmoid score map S for keypoint detection.It reduces channels with a 1 × 1 convolution and applies successive 3 × 3 convolutions.
  • Keypoint detection: DKD applies NMS and thresholding, then refines local maxima with softargmax to produce differentiable subpixel keypoints.These keypoints enable direct optimization of reprojection error between images.
  • Descriptor extraction: SDDH extracts deformable invariant descriptors using offsets that provide more degrees of freedom than affine transformations.The paper introduces SDDH in the architecture and models local geometric transformations with per-pixel offsets.

B. The Design of Sparse Deformable Descriptor Head

SDDH replaces dense descriptor-map extraction with sparse, keypoint-centered sampling and deformable support positions, reducing redundant computation while retaining expressive descriptors.

  • Descriptor Map Head Revisited: Conventional methods encode dense feature maps into dense descriptor maps before sampling descriptors at keypoint locations.This dense-map pipeline can be computationally inefficient.
  • Sparse Deformable Descriptor Head: SDDH extracts a K × K feature patch around each keypoint, estimates M deformable sample positions, and samples supporting features from the feature map.Figure 2 illustrates K = 5 as an example.
  • Sparse Deformable Descriptor Head: The sampled supporting features are encoded and aggregated with convM over M flexible positions rather than K × K fixed positions.This operation follows the design philosophy of deformable convolution while targeting sparse descriptors.
  • Sparse Deformable Descriptor Head: Although DCN models deformable invariant features, its dense convolutions are inefficient for sparse descriptor extraction.SDDH is designed to address this efficiency limitation.
  • Descriptor Map Head Revisited: Dense descriptor maps are unnecessary when descriptors are required only at sparse keypoints.Removing the dense map reduces computational effort and permits more powerful descriptors.

3) Sparse Deformable Descriptor Head:

The SDDH extracts deformable descriptors directly at sparse keypoints by learning flexible supporting positions, reducing redundant dense-map computation while retaining geometric flexibility.

  • For each keypoint, SDDH extracts a K × K feature patch and estimates M deformable sample positions for descriptor construction.The sampled supporting features are obtained through bilinear sampling.
  • Unlike DCN’s dense feature-map processing, SDDH computes deformable features only for sparse keypoints, reducing computation when keypoints are fewer than pixels.
  • The SDDH replaces fixed K × K convolution positions with M flexible positions and uses a weighted summation over those positions.Its descriptor transformation applies Φ(x) = SELU(conv1x1(x)).
  • The efficiency comparison evaluates DMH and SDDH on H × W × C feature maps with N keypoints, using M = K^2 for an equivalent K × K configuration.
  • In typical configurations, DMH spends substantial resources producing dense descriptor maps, whereas SDDH computes only on sparse keypoint patches.
  • ALIKED combines SDDH with sparse descriptors and relaxes NRE training from dense maps to sparse probability vectors.

A. Reprojection Loss

The reprojection loss trains differentiable keypoint positions by projecting keypoints between paired images and penalizing disagreement with nearby corresponding keypoints.

  • A keypoint pA is warped from image IA to image IB using 3D perspective projection with relative pose and depth.
  • The projected point pAB is matched to the nearest keypoint pB in IB when their distance is below thgt pixels.
  • The procedure projects pB back to IA and defines reprojection loss from the paired forward and backward keypoint distances.
  • The overall reprojection loss Lrp averages the reprojection losses of matching keypoints in both images.

C. Sparse Neural Reprojection Error Loss

Because SDDH produces sparse descriptors rather than a dense descriptor map, ALIKED replaces dense NRE supervision with sparse matching and adds reliability-aware score supervision.

  • Dense NRE uses reprojection and matching probability maps, but SDDH lacks a dense descriptor map for constructing the matching probability map.
  • Sparse NRE represents reprojection correspondence as a binary vector over keypoints and descriptor matching as a similarity-derived probability vector.
  • The sparse NRE loss is cross-entropy between reprojection and matching probability vectors, averaged across descriptors in both images.
  • Reliability supervision complements keypoint scores because low-texture regions may be nondiscriminative and unreliable.
  • The reliability loss weights keypoint scores by reliability, encouraging higher scores for more reliable keypoints.
  • ALIKED trains with a weighted sum of reprojection, dispersity peak, sparse NRE, and reliability losses.

VI. EXPERIMENTS

The experiments compare ALIKED with established keypoint and descriptor networks across visual measurement settings, using specified training data, implementation settings, and evaluation hardware.

  • The evaluation covers image matching, 3D reconstruction, visual relocalization, ablation studies, and limitation analysis.
  • ALIKE-N serves as the baseline because it balances running time and matching performance.
  • Training uses resized 800 × 800 images, batch size 2, gradient accumulation over 6 batches, and perspective plus homographic image pairs.
  • The perspective training data come from MegaDepth, while homographic training data come from R2D2-derived Oxford, Paris, and Aachen pairs.
  • Evaluation uses an Intel i7-10700F CPU and NVIDIA GeForce RTX 2060 GPU with CUDA 10.2 and PyTorch 1.11.0.
  • Comparisons include D2-Net, LF-Net, SuperPoint, R2D2, ASLFeat, DISK, and ALIKE.

1) Real-time Performance:

ALIKED combines sparse descriptor extraction with competitive matching accuracy and substantially lower computational cost. Its tiny variant reaches real-time speed while retaining comparable matching and reconstruction performance.

  • Efficiency: 0.192M parameters: ALIKED-T(16) is the smallest reported network configuration.The parameter count is reported in the Hpatches performance comparison.
  • Efficiency: Sparse descriptor extraction gives ALIKED networks the lowest GFLOPs among the compared methods.The comparison uses 640×480 images with 1K keypoints.
  • Matching performance: 78.70% MHA@3: ALIKED-T(16) achieves the highest mean homography accuracy on Hpatches.This exceeds ALIKE-N(16) at 77.22% and ALIKE-L at 76.85%.

3) Pose Estimation and 3D Reconstruction:

ALIKED performs strongly on stereo matching, multiview reconstruction, and visual relocalization benchmarks. Its deformable descriptors improve matching, while its tiny variant provides high performance per computational cost.

  • Stereo matching: 1.5%, 0.81%, and 1.06%: ALIKED-N(16) exceeds DISK in stereo matching for Rep, mAA(5°), and mAA(10°), respectively.The authors attribute the improvement to deformable feature modeling.
  • 3D reconstruction: ALIKED-N(32) performs better than most existing methods on multiview 3D reconstruction, except DISK.DISK obtains more matches, providing additional bundle-adjustment constraints.
  • Computational trade-off: 36.77 and 51.74 PPC: ALIKED-T(16) achieves these performance-per-cost values for stereo matching and multiview reconstruction.These values are about six times higher than the best existing method, ALIKE-N.
  • Qualitative comparison: ALIKED contains fewer false matches than DISK and recovers more matches than ALIKE in the visualized reconstruction examples.Its keypoints concentrate on building edges and corners.
  • Visual relocalization: ALIKED-N(32) achieves the best visual relocalization performance on Aachen with up to 1024 and 2048 keypoints.Reducing the limit from 2048 to 1024 causes only a slight degradation.

1) Rotation Invariance:

ALIKED-N(16) provides strong rotation and scale robustness, while deformable descriptors adapt their focus to corresponding structures across geometric transformations. Training choices affect the trade-off between invariance and reconstruction performance.

  • Rotation invariance: ALIKED-N(16, rot) achieves the best rotation invariance among the evaluated methods.The rotation-trained variant performs slightly worse in 3D reconstruction than ALIKED-N(16).
  • Rotation invariance: ALIKED-N(16) outperforms all methods except SuperPoint in rotation invariance without rotation augmentation.SuperPoint benefits from Homography Adaptation, which includes rotation augmentation.
  • Scale invariance: Among single-scale methods, ALIKED-N(16) has the best matching accuracy for scale differences.All single-scale methods degrade to zero when the scale difference exceeds 4.
  • Scale invariance: ALIKED-N(16, MS) remains effective until scale differences as large as 8, outperforming R2D2(MS) under larger scale changes.The passage reports degradation at a scale difference of 8.
  • Deformable focus: The deformable descriptor focuses on corresponding local structures across rotation, scale, homography, and perspective image pairs.The focus area rotates with the image for the ship example and changes relative receptive-field size across scales.

2) Ablation Studies on Network Architecture:

The architectural ablations show that sparse, deeper descriptor heads improve matching and pose metrics while keeping descriptor computation proportional to the number of keypoints. Deformable convolutions and efficient score-head design further support the lightweight network.

  • Feature extraction: Replacing vanilla convolutions with DCN in the last two blocks increases computation by only 0.1 GFLOPs.The change targets geometric-invariant feature extraction.
  • Score head: 1.06 GFLOPs: SH3 saves this amount compared with SH2 while achieving similar matching performance.SH3 first reduces feature channels to 8 with a 1×1 convolution before score-map estimation.
  • Descriptor head: 1.97% and 0.19%: SDH3 improves MMA@3 and MHA@3 over SDH1 on Hpatches.SDH3 also improves mAA(10°) by 1.68% and MS@3 by 1.45% on IMW-validation.
  • Descriptor head: Descriptor-head computation is proportional to the number of keypoints because descriptors are extracted only at sparse locations.Deeper and wider descriptor heads provide better overall performance in the ablation.
  • Limitations: ALIKED may fail when images combine significant scale and viewpoint differences because its single deformable-position-estimation layer limits deformation modeling.The authors note that this challenge is shared by other state-of-the-art keypoint descriptor methods.
  • Limitations: ALIKED uses grid sampling and 32-bit floating descriptors, which may be unsuitable for mobile platforms.Developing a hardware-friendly descriptor network is identified as future work.
Loading 2304.03608v2…