Source-linked AI summary
ASLFeat: Learning Local Features of Accurate Shape and Localization
Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, Long Quan
TL;DR
ASLFeat targets two limitations in joint local feature learning: neglected local shape estimation and insufficient keypoint localization accuracy for camera geometry. It introduces deformable convolutional transformation prediction, multi-level detection using feature hierarchies, and peakiness-based scoring. Extensive evaluations across practical scenarios report state-of-the-art results and improvements over the D2-Net backbone.
Problem
Joint local feature learning often lacks local shape awareness and sufficiently accurate keypoint localization, limiting geometric invariance and robust camera-geometry recovery.
Method
ASLFeat combines deformable convolutions for dense local transformation prediction, multi-level detection for resolution and detail recovery, and peakiness-based keypoint scoring.
Results
ASLFeat achieves state-of-the-art results across image matching, 3D reconstruction, and visual localization, with improvements over D2-Net and strong generalization to unseen scenarios.
Takeaways & Limitations
The proposed modifications substantially improve local-shape modelling and keypoint localization accuracy across varied practical applications.
Takeaways & Limitations
The reported image-matching metric was initially computed without a symmetric check, which was later fixed in the released code.
Abstract
from arXiv · showhide
This work focuses on mitigating two limitations in the joint learning of local feature detectors and descriptors. First, the ability to estimate the local shape (scale, orientation, etc.) of feature points is often neglected during dense feature extraction, while the shape-awareness is crucial to acquire stronger geometric invariance. Second, the localization accuracy of detected keypoints is not sufficient to reliably recover camera geometry, which has become the bottleneck in tasks such as 3D reconstruction. In this paper, we present ASLFeat, with three light-weight yet effective modifications to mitigate above issues. First, we resort to deformable convolutional networks to densely estimate and apply local transformation. Second, we take advantage of the inherent feature hierarchy to restore spatial resolution and low-level details for accurate keypoint localization. Finally, we use a peakiness measurement to relate feature responses and derive more indicative detection scores. The effect of each modification is thoroughly studied, and the evaluation is extensively conducted across a variety of practical scenarios. State-of-the-art results are reported that demonstrate the superiority of our methods.
1. Introduction
ASLFeat addresses missing local shape awareness and insufficient keypoint localization accuracy in jointly learned local features. It introduces deformable dense transformation estimation, multi-level detection, and peakiness-based scoring, then evaluates these choices across practical scenarios.
- Motivation: Jointly learned local features remain limited by weak shape awareness and inaccurate keypoint localization for robust camera geometry.Shape awareness supports stronger geometric invariance, while localization accuracy matters for geometry-sensitive tasks.
- Motivation: Dense extraction makes prior patch-wise shape estimation inapplicable because no predefined keypoint is given.Earlier approaches estimated scale, rotation, or affine shape separately before describing transformed patches.
- Motivation: Low-resolution feature maps can produce imprecise detections, causing low precision in matching or large reprojection error in SfM.The cited examples derive detections from maps at approximately 1/4 of the original image size.
- Method: ASLFeat uses deformable convolutional networks for pixel-wise local transformation estimation and progressive shape modelling.Stacking multiple deformable convolutions enables the progressive modelling described in the introduction.
- Method: Its multi-level detection mechanism restores spatial resolution and low-level details without extra learning weights, while peakiness yields more selective detection scores.These modifications target accurate localization and more indicative keypoint selection within the dense prediction framework.
- Evaluation: ASLFeat is evaluated on image matching, 3D reconstruction, and visual localization, with state-of-the-art results reported on popular benchmarks.The study also compares deformation parameterizations and feature-fusion choices.
2. Related works
Related work covers learned local-shape estimation, joint detector-descriptor learning, and approaches to improving keypoint localization. ASLFeat adopts deformable convolutions for dense transformation prediction and modifies D2-Net to recover spatial resolution and low-level details.
- Local shape estimation: Many descriptor-learning methods omit explicit local-shape modelling and instead use augmentation or hand-crafted scale and rotation estimation.Learned alternatives include canonical orientation prediction and affine-parameter estimation.
- Local shape estimation: Deformable convolution networks provide tunable grid sampling locations for dense local transformation prediction in a single forward pass.The paper uses this approach as an efficient alternative in its dense feature-learning context.
- Joint local feature learning: Joint detector-descriptor learning shares computations for fast inference, but no general consensus exists on an effective detector-learning loss.Descriptor learning commonly uses ranking loss, whereas detector supervision is harder to obtain without biased ground truth.
- Joint local feature learning: D2-Net derives keypoints from the same feature maps used for descriptors, avoiding extra detector weights but coupling detection and description.This produces a clean framework without complex heuristic loss formulation.
- Keypoint localization: ASLFeat addresses D2-Net’s low-resolution localization issue by cheaply restoring spatial resolution and low-level details through the feature hierarchy.The modification avoids extra learning weights while preserving fine structures such as corners and edges.
3. Methods
ASLFeat combines deformable convolutions, multi-level detection, and peakiness-based scoring to estimate local shape and improve keypoint localization in dense feature learning.
- DCN with Geometric Constraints: ASLFeat builds on D2-Net with deformable convolutional networks that learn dense spatial transformations for local feature extraction.The backbone uses DCNs to predict and apply spatial transformations, supporting progressive shape modelling when multiple DCNs are stacked.
- DCN with Geometric Constraints: DCN variants impose similarity, affine, or homography constraints, while free-form offsets retain the capacity to model complex geometric deformation.The comparison addresses whether local descriptors require geometrically constrained transformations or free-form modelling.
- Selective and Accurate Keypoint Detection: Peakiness replaces D2-Net’s ratio-to-max channel score with a measurement that better relates detection scores to the distribution of channel responses.A channel-wise softmax alternative degraded performance, whereas the proposed peakiness is activated with softplus and combined with the spatial score.
- Selective and Accurate Keypoint Detection: Multi-level detection upsamples and combines score maps from hierarchical feature levels to restore spatial resolution without extra learning weights.It preserves low-level structures by combining detections from multiple semantic levels rather than directly fusing low- and high-level features.
- Learning Framework: The detector and descriptor are trained jointly using a hardest-contrastive ranking formulation adopted from FCGF for improved convergence during training from scratch.The loss uses correspondence pairs obtained by densely warping image pairs with ground-truth depths and camera parameters.
4. Experiments
ASLFeat is evaluated through ablations and comparisons spanning image matching, 3D reconstruction, and visual localization. The results show gains from peakiness scoring, multi-level detection, and deformable convolutions, with strong performance across practical scenarios.
- Ablations: Peakiness measurement improves all reported HPatches evaluation metrics and transfers across FM-Bench scenarios.The authors use this model as the basis for later modifications.
- Ablations: Multi-level detection markedly improves performance by restoring spatial resolution, while outperforming U-Net with fewer learning weights and less computation.Combining it with explicit multi-scale detection is particularly advantageous for handling scale changes.
- Ablations: All investigated deformable-convolution variants notably boost performance, with free-form deformation slightly outperforming constrained modelling on HPatches.Stacking multiple deformable convolutions is beneficial because shape estimation can be learned progressively.
- Image Matching: 72.64 vs. 68.644 for MMA@3 overall, ASLFeat outperforms R2D2 on HPatches while using fewer computations for spatial-resolution restoration.ASLFeat and its multi-scale variant achieve the best overall results across illumination and viewpoint variations at different error thresholds.
- Image Matching: ASLFeat outperforms other joint-learning approaches on FM-Bench and generalizes to unseen indoor and driving scenarios.It also improves state-of-the-art results on the T&T and CPC multi-view-stereo datasets, whose scenarios match the training data.
- 3D Reconstruction: ASLFeat produces the most complete reconstructions by registered images and dense points, with reprojection error on par with SuperPoint and smaller than D2-Net.Hand-crafted keypoints still yield notably smaller reprojection error than learning methods.
5. Conclusions
ASLFeat introduces three lightweight modifications to jointly learned local feature detectors and descriptors, targeting local-shape modelling and accurate keypoint localization. Extensive experiments study each modification and demonstrate performance and practical advantages across applications.
- ASLFeat uses three lightweight modifications to improve local-shape modelling and keypoint localization in joint feature learning.The modifications target stronger geometric invariance and robust camera geometry.
- The method jointly learns local feature detectors and descriptors using D2-Net as its backbone architecture.
- Extensive experiments study the effect of each modification across various applications.The paper reports evaluations in multiple practical scenarios.
A.1 Implementation Details of DCN
The DCN implementation predicts deformation parameters and applies them across channels to reduce learning weights. Affine constraints can model scaling, rotation, and residual shape, while the paper reports that free-form DCN is preferable despite limited separation in experiments.
- One deformation-parameter set is predicted and applied across all channels, reducing the number of learning weights.This corresponds to using one deformable group.
- Affine-constrained DCN models scaling, rotation, and residual shape through bounded predicted parameters.The implementation predicts a bounded scalar for scaling, cosine and sine terms for rotation, and a residual shape matrix with determinant one.
- Free-form DCN is preferred over geometrically constrained deformation parameterizations for local feature learning.The paper notes that this difference is not obvious in Table 2 and attributes the result to limited meaningful supervision for complex deformation.
- A specialized loss may be needed to guide local shape estimation, but the tested AffNet loss produced no consistent improvement.
A.2 Implementation Details of MulDet
The appendix compares pyramid, in-network multi-scale, and multi-level implementations for detection. Pyramid inference increases runtime, while in-network multi-scale detection was not useful when combined with DCN; multi-level detection derives keypoints from high-resolution maps.
- The pyramid variant processes an image pyramid from size 2048 down to a longest side below 128 pixels.Scales are downsampled by 2 and blurred with a Gaussian kernel factor of 0.8.
- Final pyramid detections retain thresholded keypoints from all scales and select the top-K combined scores.The inference time doubles when this configuration is enabled.
- The in-network multi-scale variant resizes conv8 feature maps into scales from 1/R to R with N = 5 and R = 2.Score maps are merged by softmax-weighted summation.
- In-network multi-scale detection was not useful when combined with the proposed methods because DCN already handled in-network scale invariance.
- The multi-level variant uses conv1 and conv3 skip connections, concatenates features, and derives keypoints from high-resolution feature maps.
A.3 Additional Experiments
Additional experiments evaluate ASLFeat in dense reconstruction, image retrieval, and learned-matcher-assisted visual localization. The reported results show consistent dense-reconstruction improvements, good retrieval performance, and further localization gains with OANet integration.
- Dense reconstruction: ASLFeat delivers consistent improvements on dense reconstruction across seven T&T scans measured by F-score.F-score jointly measures reconstruction accuracy and completeness; without multi-scale detection, ASLFeat performs best overall on this dataset.
- Image retrieval: ASLFeat performs well on Oxford buildings and Paris image retrieval using mean average precision.The evaluation uses a VocabTree pipeline without query expansion or re-ranking.
- Visual localization: ASLFeat combined with OANet further boosts localization results on the Aachen Day-Night dataset.The learned matcher rejects outlier matches to improve camera-pose recovery.
A.4 Discussions
The discussion reports that separate training of deformation parameters performs better than end-to-end training, while feature-count effects are examined on HPatches.
- End-to-end learning with DCN: 72.64 MMA@3 on HPatches is obtained with two-stage deformation-parameter training, compared with 70.45 for end-to-end training.The comparison is reported in Table 2.
- End-to-end learning with DCN: No tested end-to-end training strategy, including reducing the deformation-parameter learning rate by 10, outperformed separate training.The authors leave potential benefits of end-to-end learning for future exploration.
- Performance regarding different feature number: Performance changes in matching score and mean matching accuracy are plotted on HPatches when limiting the maximum number of features.The figure reports results across different feature counts.
- Performance regarding different feature number: Figure 6 reports matching score and MMA at a 3px error threshold across feature counts, highlighting results at 5K features for the proposed methods.The 5K-feature results are marked in red.
A.5 More Visualizations
The paper provides visual comparisons of keypoints detected by SIFT, D2-Net, and ASLFeat.
- A.5 More Visualizations: Figure 7 visualizes keypoints from SIFT, D2-Net, and the proposed method for direct comparison.The visualization compares keypoint detections across the three local-feature methods.