Source-linked AI summary
R2D2: Repeatable and Reliable Detector and Descriptor
Jerome Revaud, Philippe Weinzaepfel, César De Souza, Noe Pion, Gabriela Csurka, Yohann Cabon, Martin Humenberger
TL;DR
Local feature pipelines can detect salient points that are repeatable but difficult to match, leaving descriptor learning vulnerable to ambiguous regions. R2D2 jointly learns detection, description, and descriptor reliability with self-supervision, and reports state-of-the-art performance across HPatches and visual localization.
Problem
Repeatability alone does not ensure discriminative descriptors, so learned descriptors may be trained on repeatable regions where accurate matching is unreliable.
Method
R2D2 jointly learns sparse keypoint detection, local description, and a reliability predictor using self-supervised training on synthetic transformations and real correspondences.
Results
The combined pipeline selects repeatable and reliable keypoints and achieves state-of-the-art results on HPatches and visual localization.
Takeaways & Limitations
Explicit reliability estimation avoids ambiguous regions and yields sparse, repeatable, and reliable keypoints for local feature matching.
Takeaways & Limitations
Repeatability training assumes transformations without occlusions, warp artifacts, or border effects, which can strongly impact performance in practice.
Abstract
from arXiv · showhide
Interest point detection and local feature description are fundamental steps in many computer vision applications. Classical methods for these tasks are based on a detect-then-describe paradigm where separate handcrafted methods are used to first identify repeatable keypoints and then represent them with a local descriptor. Neural networks trained with metric learning losses have recently caught up with these techniques, focusing on learning repeatable saliency maps for keypoint detection and learning descriptors at the detected keypoint locations. In this work, we argue that salient regions are not necessarily discriminative, and therefore can harm the performance of the description. Furthermore, we claim that descriptors should be learned only in regions for which matching can be performed with high confidence. We thus propose to jointly learn keypoint detection and description together with a predictor of the local descriptor discriminativeness. This allows us to avoid ambiguous areas and leads to reliable keypoint detections and descriptions. Our detection-and-description approach, trained with self-supervision, can simultaneously output sparse, repeatable and reliable keypoints that outperforms state-of-the-art detectors and descriptors on the HPatches dataset. It also establishes a record on the recently released Aachen Day-Night localization dataset.
1. Introduction
R2D2 addresses the mismatch between repeatability and discriminativeness by jointly learning sparse keypoint detection, local description, and descriptor reliability. Its self-supervised pipeline selects keypoints that are both repeatable and reliable, achieving state-of-the-art results.
- Keypoints must be sparse, repeatable, and discriminative to reduce memory use while improving matching accuracy in visual localization, detection, pose estimation, SfM, and reconstruction.
- Learning detectors has focused on repeatability, but repeatable regions such as checkerboards may remain ambiguous because self-similar patterns cannot be accurately matched.
- R2D2 jointly outputs dense descriptors and repeatability and reliability maps, then selects locations maximizing both confidence estimates.
- The method introduces an unsupervised detector loss encouraging repeatability, sparsity, and uniform image coverage.
- A new descriptor loss explicitly estimates reliability while learning descriptors, targeting discriminative regions where matching can be performed confidently.
- The combined pipeline selects repeatable and reliable keypoints and achieves state-of-the-art results.
2. Related work
Prior learned local-feature methods separately or jointly address detection and description, but differ in supervision, handcrafted bias, and how keypoints are selected. R2D2 is positioned as an unsupervised approach that explicitly models repeatability and reliability.
- Learned descriptors: Learned descriptor methods commonly use triplet or contrastive metric-learning losses on sparse or dense features, while R2D2 jointly trains detection and description.
- Learned detectors: Earlier learned detectors either retain handcrafted detectors or filters, imitate handcrafted outputs, or use unsupervised saliency-ranking and peakiness objectives.
- Jointly learned descriptor and detector: LIFT, SuperPoint, DELF, D2-Net, and related systems jointly or sequentially learn detection and description, but use differing supervision, keypoint definitions, and optimization schemes.
- Jointly learned descriptor and detector: R2D2's distinction is explicit estimation of keypoint repeatability and reliability rather than defining detections as descriptor-space local maxima.
3. Joint learning reliable and repeatable detectors and descriptors
R2D2 jointly learns repeatability, descriptor discriminativeness, and dense local features so keypoints are selected where matching is both stable and reliable. Its self-supervised losses and confidence-based scoring address ambiguity in repeatable but nondiscriminative regions.
- Joint architecture: R2D2 predicts dense descriptors alongside separate repeatability and reliability maps, selecting locations that score highly on both.The reliability map estimates whether each descriptor is discriminative and can be matched accurately with high confidence.
- Learning repeatability: Repeatability is trained self-supervised from correspondences between image pairs, encouraging covariant local maxima under viewpoint or illumination changes.Correspondences can come from optical flow, stereo matching, or known synthetic transformations such as homographies.
- Learning repeatability: The repeatability loss averages cosine similarity over overlapping local patches to reduce sensitivity to occlusions, warp artifacts, and border effects.A peakiness term prevents the similarity objective from being minimized trivially by constant heatmaps, while patch size controls local-maximum frequency.
- Learning reliability: Descriptor training uses listwise ranking based on approximated Average Precision, while reliability weighting reduces pressure to optimize descriptors in nondiscriminative regions.The confidence value can be low where descriptors cannot be made sufficiently discriminative, including uniform, elongated, or semantically unreliable textures.
- Inference and training details: At test time, R2D2 extracts local maxima across multiple scales and ranks candidates by the product SijRij, retaining the best K descriptors.Training uses transformed or matched image pairs, with positives within 4 pixels of correspondence and negatives beyond 8 pixels.
4. Experimental results
Experiments on HPatches and visual localization evaluate R2D2’s repeatability, matching accuracy, and localization performance across keypoint counts, patch sizes, and ablations. The method generally benefits from jointly modeling repeatability and reliability, while revealing a trade-off between retaining more keypoints and matching quality.
- 4.1. Dataset and metrics: HPatches evaluation uses repeatability, M-score, and MMA, including MMA at multiple pixel-error thresholds.Repeatability measures correspondences normalized by the smaller detection count; M-score measures recovered ground-truth correspondences among estimated features; MMA averages correct-match percentages across thresholds.
- 4.2. Impact of repeatability patch size: Larger repeatability-loss patches produce fewer, more repeatable keypoints, whereas smaller patches produce more keypoints.The patch size N encourages approximately one local maximum per N × N window.
- 4.2. Impact of repeatability patch size: At low keypoint counts, larger N strongly outperforms smaller N on HPatches matching accuracy; retaining more keypoints introduces a quality trade-off.With many retained points, poor local maxima can be selected, while larger keypoint sets can help visual localization under occlusion or strong viewpoint and illumination changes.
- 4.3. Impact of separate reliability and repeatability: Removing repeatability significantly reduces both MMA@3 and M-score, showing that repeatability is not well correlated with descriptor reliability.The ablation compares the full method with variants removing either the repeatability or reliability map.
- 4.3. Impact of separate reliability and repeatability: Without descriptor-reliability estimation, M-score decreases by 3% and MMA@3 by 0.6%.Reliability scoring rejects ambiguous regions such as sky, one-dimensional patterns, and grid-like building windows from the top-K output.
- 4.4. Comparison with the state of the art: On HPatches, R2D2 significantly outperforms the state of the art at middle-range MMA thresholds and reaches an M-score of 0.425 at 3px.The 3px M-score compares with 0.335 for LF-Net and 0.288 for SIFT; on Aachen localization, the augmented K = 10000, N = 8 model outperforms competing methods at all error thresholds.
5. Conclusion
R2D2 jointly learns keypoint detection and description, including repeatability and reliability, using self-supervision. It reports superior results to state-of-the-art methods on HPatches and visual localization.
- R2D2 jointly detects and describes keypoints with a learning-based feature extraction method.
- The method learns keypoint repeatability and reliability confidence from relevant training data.
- Self-supervision combines synthetic images with known transformations and real data containing point correspondences.
- Style transfer is used to increase robustness against drastic illumination changes such as day-night transitions.
- Experiments on HPatches and visual localization report superior results compared with state-of-the-art methods.