Source-linked AI summary
Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters
Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, Krystian Mikolajczyk
TL;DR
Keypoint detectors need robust, repeatable features across geometric and photometric changes, while fully learned detectors have shown limited repeatability gains. Key.Net combines handcrafted anchors, learned CNN filters, multiscale processing, and a differentiable loss; it reports improved repeatability, state-of-the-art matching performance, and low complexity.
Problem
Fully CNN-based keypoint detectors had limited improvements over handcrafted detectors on repeatability, with scale robustness remaining particularly problematic.
Method
Key.Net combines handcrafted and learned CNN filters in a shallow multiscale architecture, using M-SIP and a loss that combines local and global information across scales.
Results
The evaluation reports improved repeatability over other state-of-the-art keypoint detectors, state-of-the-art matching performance with a descriptor, and a detector using 280 learnable parameters at 175 frames per second.
Takeaways & Limitations
Combining handcrafted and learned features with multiscale analysis improves repeatability while enabling a low-complexity detector.
Takeaways & Limitations
Anchors can prevent the network from proposing new keypoints when no anchor exists nearby.
Abstract
from arXiv · showhide
We introduce a novel approach for keypoint detection task that combines handcrafted and learned CNN filters within a shallow multi-scale architecture. Handcrafted filters provide anchor structures for learned filters, which localize, score and rank repeatable features. Scale-space representation is used within the network to extract keypoints at different levels. We design a loss function to detect robust features that exist across a range of scales and to maximize the repeatability score. Our Key.Net model is trained on data synthetically created from ImageNet and evaluated on HPatches benchmark. Results show that our approach outperforms state-of-the-art detectors in terms of repeatability, matching performance and complexity.
1. Introduction
Key.Net addresses limited gains from fully learned keypoint detectors by combining handcrafted and learned filters with multiscale processing. Its objective targets stable keypoints across scales and transformations, and the paper evaluates the approach on HPatches.
- Motivation: Fully CNN-based detectors had limited improvements over handcrafted methods on repeatability, especially because scale robustness and affine-parameter estimation remained challenging.Orientation estimation was reported as more successfully regressed by CNNs than scale-related properties.
- Approach: The model processes multiscale representations and returns a response map containing a keypoint score for every pixel.Its multiscale input is intended to propose stable keypoints across scales without relying on a single image scale.
- Training objective: The proposed loss extends covariant-constraint training by combining local and global information through a differentiable Multi-scale Index Proposal operator.M-SIP proposes keypoints in multiscale regions, while the loss is evaluated across those regions.
- Evaluation: The paper evaluates Key.Net on the HPatches benchmark for accuracy and repeatability under the benchmark protocol.The introduction identifies HPatches as the evaluation benchmark for the method.
- Approach: Key.Net combines handcrafted and learned CNN features in a shallow architecture to detect and rank stable keypoints across multiple scales.The stated contributions include hybrid features, a multiscale loss and operator, and shallow multiscale detection.
2. Related Work
Prior feature detectors span handcrafted derivative-based methods and learned detectors built with machine learning or CNNs. Learned approaches expanded beyond detection into descriptors, matching, orientation, and affine-parameter prediction, but the section frames these as related developments rather than a settled replacement for handcrafted detection.
- Organization: The related work is organized into handcrafted and learned approaches for feature detection.The paper presents these as its two main categories of prior work.
- Handcrafted methods: Handcrafted detectors such as Harris, Hessian, DoG, SURF, KAZE, and A-KAZE use derivatives, approximations, or multiscale processing to localize image structures.These methods target corners, blobs, or related local structures using engineered operations.
- Learned detectors: Machine-learning detectors began with FAST and later added optimization, descriptors, or orientation estimation.The cited progression extends a learned corner detector with additional functionality.
- CNN-based detectors: CNN-based detectors addressed robustness to weather, illumination, covariant transformations, salient-point extraction, and parameterized image transformations.The related methods include TILDE, covariant-constraint training, predefined anchors, and MagicPoint/MagicWarp.
- Beyond detection: CNN architectures were also applied to predict matchability, find wide-baseline correspondences, estimate orientations, and predict affine parameters.These works extend learned local-feature processing beyond point detection itself.
3. Key.Net Architecture
Key.Net combines derivative-inspired handcrafted filters with learned convolutional blocks in a shallow, shared-weight multiscale architecture. The resulting feature maps are merged into a response map for keypoint detection.
- Architecture: Key.Net combines gradient-based extraction, learned combinations of low-level features, and a multiscale pyramid representation.These components integrate ideas from handcrafted and learned feature-detection methods.
- Handcrafted filters: The handcrafted design follows Harris and Hessian detectors by using first- and second-order image derivatives to compute salient responses.The filter construction is inspired by classical corner and blob detectors.
- Handcrafted filters: Only derivatives and combinations up to second order are included because higher-order derivatives are noise-sensitive and require large kernels.The paper explicitly limits the handcrafted basis to second order for this reason.
- Learned filters: The learned block consists of a convolutional layer, batch normalization, and ReLU activation.This defines the basic learned processing unit used in the architecture.
- Hybrid design: Hardcoded filters reduce the number of learnable parameters and improve training stability and convergence during backpropagation.The paper attributes both lower parameter count and optimization benefits to the fixed filters.
- Multiscale processing: Three shared-weight streams process blurred and downsampled input levels, then upsampled feature maps are concatenated and passed to a final convolutional filter.The final filter produces the response map used for keypoint scoring.
4. Loss Functions
Key.Net formulates differentiable, covariant keypoint detection around local maxima and extends it across multiple window scales. Handcrafted filters provide weak constraints while the multi-scale loss favors features that remain stable and dominant across scales.
- Loss Functions: Key.Net uses handcrafted filters as weak constraints, retaining anchor-based training benefits while allowing new stable keypoints without nearby anchors.Only geometric transformations between image pairs are required to guide the loss.
- Index Proposal Layer: Unlike prior grid- or feature-map-limited approaches, the Index Proposal layer returns multiple keypoint coordinates centered on local maxima from one image.The number of returned points is not constrained by feature-map depth or grid size.
- Index Proposal Layer: The differentiable Index Proposal layer uses spatial softmax to estimate one keypoint coordinate per response-map window.Exponential scaling makes the maximum dominate the weighted-average coordinate estimate.
- Loss Functions: The covariant constraint loss compares corresponding keypoint locations under a ground-truth homography and weights contributions by feature scores.The images are switched and both losses combined to enforce consistency because NMS is non-differentiable.
- Multi-scale Index Proposal Layer: M-SIP repeatedly partitions response maps into grids at multiple window sizes and averages covariant losses across scale levels.Its scale weights decrease with window area, while simultaneous scoring and ranking favors features dominant across scales.
5. Experimental Settings
The experiments train Key.Net on synthetically transformed ImageNet regions and evaluate repeatability under controlled correspondence and matching protocols. Design studies examine transformation filtering, scale settings, and handcrafted-filter configurations.
- Dataset and Training: The synthetic training set uses ImageNet images transformed with scale [0.5, 3.5], skew [−0.8, 0.8], and rotation [−60◦, 60◦].Textureless regions are discarded using handcrafted-filter responses, and HSV contrast, brightness, and hue are modified for robustness.
- Evaluation Metrics: Repeatability is computed as corresponding keypoints divided by the lower detected-keypoint count, with a fixed number of extracted points across methods.Each keypoint may match only once.
- Evaluation Metrics: The benchmark uses the top 1,000 interest points in the common image region, counting matches when ϵIoU < 0.4.This corresponds to more than 60% overlap between corresponding regions.
- Implementation Notes: Training uses a weight-sharing siamese pipeline with M = 8 filters per 5 × 5 convolutional layer and five M-SIP window sizes: Ns ∈[8, 16, 24, 32, 40].The corresponding loss terms are λs ∈[256, 64, 16, 4, 1], selected by validation-set hyperparameter search.
- Implementation Notes: Figure 5 reports repeatability on a synthetic ImageNet validation set while varying M-SIP levels, context losses, handcrafted filters, and learnable layers.The caption states that five window sizes from 8 × 8 to 40 × 40 perform best and that more layers improve results.
6. Results
Experiments show that Key.Net and its compact variant achieve strong HPatches detection and matching results, while handcrafted filters reduce the learnable complexity needed for competitive performance.
- Preliminary Analysis: Increasing M-SIP scale levels improves repeatability, with the best result obtained when all levels are combined.Smaller loss-window sizes also improve repeatability.
- Preliminary Analysis: Handcrafted filters are essential when few learnable layers are used, while deeper fully learnable and combined networks achieve comparable repeatability.The hybrid design uses engineered features as soft constraints and reduces the number of learnable layers.
- Keypoint Detection: Key.Net and Tiny-Key.Net are the best algorithms on HPatches viewpoint sequences for both location-only and scale-location evaluation.On illumination sequences, translation-invariant Key.Net-TI obtains the best accuracy; among scale-invariant detectors, TCDET leads in location and LF-Net in scale-location evaluation.
- Keypoint Detection: Tiny-Key.Net uses all handcrafted filters, one learnable layer with one filter, and a single scale input to reduce detector complexity.It performs below Key.Net but remains within the top three viewpoint repeatability scores.
- Keypoint Matching: Key.Net obtains the best matching score on viewpoint sequences, while LF-Net+HardNet performs best on illumination sequences.Matching scores use the same HardNet descriptor for detector comparisons, with orientation discarded where applicable.
- Keypoint Detection: Handcrafted detectors achieve the lowest average overlap error, while MSER detects a wide range of scales.The reported scale range is the ratio between maximum and minimum extracted interest-point scales.
7. Conclusions
The proposed detector combines handcrafted and learned CNN filters with multi-scale analysis to improve repeatability while controlling complexity. It also achieves state-of-the-art matching performance when paired with a descriptor.
- Combining handcrafted and learned features with multi-scale analysis improves repeatability compared with other state-of-the-art keypoint detectors.
- 280 learnable parameters and 175 frames per second demonstrate a low-complexity, fast detector enabled by handcrafted filters.
- The proposed detectors achieve state-of-the-art matching performance with a descriptor on viewpoint.
- Excessively increasing network complexity does not improve results.