Source-linked AI summary
HPatches: A benchmark and evaluation of handcrafted and learned local descriptors
Vassileios Balntas, Karel Lenc, Andrea Vedaldi, Krystian Mikolajczyk
TL;DR
Existing descriptor benchmarks leave evaluation details ambiguous, produce inconsistent comparisons, and often lack sufficient diversity for modern descriptors. The paper introduces HPatches, a large patch-based benchmark with defined protocols across complementary tasks, and finds that descriptor performance and rankings vary by task and can differ from prior literature.
Problem
Existing benchmarks lack unambiguous evaluation protocols and sufficient diversity, making descriptor comparisons unreliable for modern local-descriptor research.
Method
HPatches provides a large patch-based dataset with ground-truth transformations and clearly defined protocols for patch verification, image matching, and patch retrieval.
Results
Descriptor performance and rankings vary across tasks and differ from results reported in the literature; normalized SIFT reaches top-level performance in matching and retrieval.
Takeaways & Limitations
Reliable descriptor assessment requires a large, varied, reproducible benchmark that controls evaluation ambiguities and compares methods across tasks.
Abstract
from arXiv · showhide
In this paper, we propose a novel benchmark for evaluating local image descriptors. We demonstrate that the existing datasets and evaluation protocols do not specify unambiguously all aspects of evaluation, leading to ambiguities and inconsistencies in results reported in the literature. Furthermore, these datasets are nearly saturated due to the recent improvements in local descriptors obtained by learning them from large annotated datasets. Therefore, we introduce a new large dataset suitable for training and testing modern descriptors, together with strictly defined evaluation protocols in several tasks such as matching, retrieval and classification. This allows for more realistic, and thus more reliable comparisons in different application scenarios. We evaluate the performance of several state-of-the-art descriptors and analyse their properties. We show that a simple normalisation of traditional hand-crafted descriptors can boost their performance to the level of deep learning based descriptors within a realistic benchmarks evaluation.
1. Introduction
Existing local-descriptor benchmarks produce unreliable comparisons because their evaluation details are ambiguous and their datasets lack sufficient diversity. HPatches addresses this gap with a larger, reproducible, patch-based benchmark spanning multiple tasks and real capture conditions.
- Local descriptors remain essential to image matching and retrieval, while learned representations have accelerated descriptor research.
- Existing benchmarks can yield inconsistent descriptor rankings because detector choices, measurement regions, and other evaluation parameters vary across studies.
- The proposed HPatches benchmark is significantly larger, uses clearly defined protocols, and is designed to generalise better than existing datasets.
- Its data comprise diverse multi-image sequences captured under varying viewpoint, illumination, and temporal conditions, with ground-truth transformations for correspondence assessment.
- Patch-based evaluation removes detector-related influences, simplifies reproducibility, and avoids biases that make image-based descriptor results difficult to compare.
- The benchmark defines patch verification, image matching, and patch retrieval, and experiments show descriptor performance and rankings vary across tasks and differ from literature reports.
2. Review of existing benchmarks
Existing image- and patch-based benchmarks leave important sources of variation, limited diversity, or metric shortcomings that undermine reliable descriptor comparison. These limitations motivate a standardized patch-based benchmark with broader data and task coverage.
- Existing image datasets trade off realism, diversity, resolution, or application coverage, with examples including synthetic transformations, controlled laboratory capture, and very few scenes.
- Image-based benchmarks depend strongly on detector methods, implementations, and parameters because they lack a predefined set of regions to match.
- Measurement-region scaling is often arbitrary or unreported, and changing it can alter Oxford matching scores by more than 50% mAP.
- Patch-based evaluation standardizes patch extraction, normalization, and correspondence labeling, reducing variation that remains in image-based evaluations.
- PhotoTourism introduced patch verification as binary classification of matching patch pairs, but it contains only three scenes and one task.
- Evaluation metrics must account for severe class imbalance in patch matching, for which ROC curves may be less representative.
3. Benchmark design
HPatches is designed around reproducibility, diversity, realism, and scale. It combines varied real image sequences with standardized patch-based evaluation to support more reliable cross-descriptor comparisons.
- The benchmark requirements are reproducibility, patch-based evaluation, diverse scenes and capture conditions, real data, and sufficient scale.
- Figure 1 illustrates sequences spanning viewpoint, illumination, focus, reflections, and other nuisance factors.
- HPatches introduces a large-scale homography-annotated image-sequence dataset used to generate a patch-based local-descriptor benchmark suite.
4. Images and patches
HPatches builds its image and patch data from diverse real sequences with controlled geometric and photometric variation. Ground-truth homographies and simulated detector noise produce normalized 65 × 65 patches spanning EASY, HARD, and TOUGH conditions.
- 51 sequences were captured by camera, while 33, 12, 5, 4, 2, and 1 scenes came from existing datasets.
- The collection covers 57 photometric-change scenes and 59 sequences with significant viewpoint-induced geometric deformation.
- Each sequence contains one reference image and five target images whose transformations are approximated by homographies.
- Stable regions are extracted with scale-invariant detectors, deduplicated at 0.5 IoU, and sampled at approximately 1,300 regions per image.DoG, Hessian-Hessian, and Harris-Laplace detectors are used for scales above 1.6px.
- Reference-image regions are projected to target images using ground-truth homographies, avoiding detector failures under strong viewpoint or illumination changes.This also enables extracting more corresponding patches for descriptor evaluation.
- EASY, HARD, and TOUGH perturbations vary rotation, anisotropic scaling, and scale-proportional translation to simulate detector noise.The groups correspond to regions from progressively more transformed image pairs in Oxford-style viewpoint sequences.
5. Benchmark tasks
HPatches evaluates descriptors through three complementary tasks—patch verification, image matching, and patch retrieval—using normalized average precision over controlled correspondence and distractor settings. The protocols define labels, ranking scores, and task-specific aggregation while accommodating descriptors with custom similarity metrics.
- The benchmark defines patch verification, image matching, and patch retrieval to represent classification, correspondence, and indexing use cases.The tasks differ in how correspondence candidates are constructed and evaluated.
- HPatches computes precision and recall for ranked labels in {−1, 0, +1}, where 0 marks entries ignored during retrieval evaluation.The retrieval formulation allows non-corresponding patches from matching images to be treated as innocuous.
- Patch verification ranks confidence scores for positive and negative patch pairs and measures average precision over the resulting ordering.The method supplies scores for patch correspondence, and the benchmark averages AP across six patch sets.
- The verification benchmark includes EASY, HARD, and TOUGH projection-noise sets plus within-sequence and cross-sequence negative pairs.Each set contains 2 × 10^5 positive pairs and 1 × 10^6 negative pairs.
- The protocols require only confidence scores or selected indexes, thereby evaluating descriptors together with their supplied similarity metrics.This includes custom and learned similarity functions.
- Image matching selects one target patch and confidence score for each reference patch, then computes AP after labeling correct assignments as positive.Performance is averaged across viewpoint or illumination groups and their EASY, HARD, and TOUGH variants.
- Patch retrieval ranks a query against matching-sequence patches and distractors, with exactly K positives and same-sequence nonmatches ignored.The benchmark averages AP over 10^4 collections, each containing five corresponding patches and 2 × 10^4 distractors across three noise variants.
6. Experimental results
The benchmark evaluates handcrafted, binary, and learned descriptors across verification, matching, and retrieval, revealing task-dependent rankings and substantial benefits from normalization. Results also expose speed–accuracy trade-offs and show that verification alone is insufficient for descriptor evaluation.
- Benchmark tasks: Descriptors are evaluated on patch verification, image matching, and patch retrieval, with results varying across task and geometric-noise settings.Figure 4 summarizes six experimental variants per task, with marker colors indicating EASY, HARD, and TOUGH noise.
- Descriptor performance: Normalized +DDESC and +RSIFT achieve strong results, while descriptor rankings change between verification and matching.+RSIFT leads matching, whereas normalized +DDESC remains near the top across categories.
- Robustness: Up to 30% separates performance on EASY versus TOUGH geometric distortions and illumination changes, indicating remaining robustness challenges.The reported gap applies across the evaluated conditions rather than to a single descriptor.
- Efficiency and trade-offs: Binary descriptors trail SIFT in matching and retrieval but remain competitive for verification, while their compactness and speed suit accuracy-constrained applications.BRIEF and ORB are reported as four times faster than the most efficient CNN-based features, whereas +DDESC is close to top scores but slowest to calculate.
- Normalization: ZCA projection and normalization substantially improve most descriptors, bringing SIFT to top-level performance in matching and retrieval.The paper reports especially significant gains for DC- and SIFT-based descriptors, with smaller improvements for TF-based features.
- Evaluation scope: Patch verification obtains much higher average mAP than the other tasks, confirming that it is insufficient as the sole descriptor evaluation.The strong verification performance of the trivial MSTD descriptor motivates using complementary tasks.
7. Conclusions
The paper concludes that commonly used local-descriptor benchmarks are inadequate for reliable comparison. HPatches addresses this with a large, diverse, patch-based, multitask benchmark and open-source protocols for reproducible evaluation.
- Conclusions: Existing benchmarks make descriptor comparisons unreliable because their evaluation protocols are ambiguous and their data and task diversity are limited.The paper links these shortcomings to the rapid development of learned descriptors and the need for more rigorous evaluation.
- Conclusions: HPatches uses patch-based evaluation, diverse real capturing conditions, and three application-oriented tasks to improve rigor, reproducibility, and scale.The benchmark is designed to remove detector-related ambiguities while expanding scene, visual-effect, and task coverage.
- Conclusions: Open-source implementations make the multitask protocols usable with minimal effort and position HPatches as a successor to older descriptor datasets.The paper specifically names PhotoTourism and the Oxford matching dataset as datasets whose shortcomings HPatches addresses.