Source-linked AI summary
Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Ondřej Chum
TL;DR
The paper addresses unreliable comparisons on Oxford and Paris caused by annotation problems, limited scale, and insufficient difficulty. It revises the datasets and protocols, adds challenging distractors, and evaluates diverse retrieval methods. The resulting benchmark shows that image retrieval remains an open problem, especially at large scale and under difficult viewing conditions.
Problem
Oxford and Paris have unreliable annotation, limited size, and insufficient difficulty for modern image-retrieval benchmarking.
Method
The paper creates new annotations, updated evaluation protocols, more difficult queries, and a semi-automatically cleaned set of one million hard distractors.
Results
Image retrieval remains an open problem, especially at large scale and under difficult viewing conditions.
Takeaways & Limitations
Combining local features with CNN-based methods can improve performance by combining their respective benefits.
Takeaways & Limitations
Queries must use only cropped regions, and queries without positives in a setting are excluded from evaluation.
Abstract
from arXiv · showhide
In this paper we address issues with image retrieval benchmarking on standard and popular Oxford 5k and Paris 6k datasets. In particular, annotation errors, the size of the dataset, and the level of challenge are addressed: new annotation for both datasets is created with an extra attention to the reliability of the ground truth. Three new protocols of varying difficulty are introduced. The protocols allow fair comparison between different methods, including those using a dataset pre-processing stage. For each dataset, 15 new challenging queries are introduced. Finally, a new set of 1M hard, semi-automatically cleaned distractors is selected. An extensive comparison of the state-of-the-art methods is performed on the new benchmark. Different types of methods are evaluated, ranging from local-feature-based to modern CNN based methods. The best results are achieved by taking the best of the two worlds. Most importantly, image retrieval appears far from being solved.
1. Introduction
The paper revisits Oxford and Paris, popular image-retrieval benchmarks whose annotation, scale, and difficulty limit reliable comparisons. It introduces revised datasets and evaluates retrieval methods across these benchmark concerns.
- Benchmark quality depends on reliable annotation, sufficient dataset size, and an appropriate challenge level.These properties determine whether standardized benchmarks can measure progress and simulate real-world applications.
- Oxford and Paris contain annotation errors, are relatively small at 5,062 and 6,392 images, and yield near-perfect results with current methods.The errors include false positives, false negatives, and shared annotations for visually distinguishable landmark views.
- Outdated annotation labels many viewpoint changes as Junk, reducing the measured difficulty of instances that are now expected to be retrieved.The annotation was introduced about ten years ago, under different perceptions of image-retrieval limits.
- The Oxford 100k distractor set adds scale but includes false negatives and non-challenging images, while state-of-the-art methods remain near-perfect.Consequently, the added computational effort provides little benefit for drawing conclusions.
- The paper creates reliable new annotations, updated protocols, more difficult queries, and challenging distractors for revised Oxford and Paris benchmarks.The resulting resources are named Revisited Oxford, Revisited Paris, and R1M, a one-million-image distractor set.
- Its evaluation covers image-retrieval methods from local-feature-based approaches through CNN-descriptor-based methods, including re-ranking.
2. Revisiting the datasets
The paper revisits Oxford and Paris by correcting annotation problems, expanding queries, and introducing harder evaluation data. It also replaces the noisy Oxford100k-style distractors with a cleaned, challenging million-image set.
- Original datasets: Oxford and Paris originally use manually labeled landmark images, with mAP computed over 55 queries while junk images are ignored.The datasets contain 5,063 and 6,392 images, respectively, with positive, junk, and negative labels.
- Additional queries: 15 new queries per dataset expand the benchmark from 55 to 70 queries, with query groups defined by visual similarity.The added queries come from five landmarks, with three queries per landmark, and include background clutter and large-occlusion cases.
- Revisiting the annotation: The revision uses five annotators, potential-positive retrieval, Easy/Hard/Unclear/Negative labels, and majority voting to refine ground truth.The new annotation addresses false positives, false negatives, viewpoint differences, and other labeling errors.
- Evaluation protocol: The original protocol is closest to Easy, whereas Medium and Hard remain challenging because even the best methods achieve relatively low scores.Easy remains useful for near-duplicate detection or retrieval with ultra-short codes.
- Distractor limitations: Oxford100k contains false negatives and non-challenging distractors, so adding it produces little performance degradation while increasing evaluation noise and computational effort.Larger existing distractor sets are neither standardized for direct comparison nor manually cleaned.
- Distractor set R1M: R1M contains exactly 1,001,001 high-resolution images selected through semi-automatic cleaning and hard-image mining across queries and state-of-the-art methods.The selected set is much harder than a random 1M subset and nearly as hard as all 4M candidate distractors.
3. Extensive evaluation
The evaluation covers classical local-feature pipelines and CNN-based global descriptors, with varied architectures, pooling, fine-tuning, whitening, query expansion, and large-scale distractor testing.
- The benchmark evaluates classical local-feature methods and CNN-based methods producing global image descriptors.
- The evaluation uses Medium and Hard protocols on Oxford, Paris, ROxford, and RParis, with and without large distractor sets, alongside time and memory measurements.Table 2 compares several methods across all distractors, selected hard subsets, and a random 1M sample.
- Local-feature-based methods: DELF replaces Hessian-affine and RootSIFT within the BoW pipeline, creating a bridge between classical retrieval and deep learning.The default extraction uses at most 1000 features per image, with descriptors reduced to 128 dimensions.
- Local-feature-based methods: Local-feature pipelines include VLAD, SMK⋆, ASMK⋆, spatial verification, and Hamming query expansion.ASMK⋆ aggregates local descriptors assigned to the same visual word, while spatial verification uses RANSAC on top-ranked images.
- CNN-based global descriptor methods: CNN baselines vary across AlexNet, VGG-16, and ResNet101 architectures, with pooling methods including MAC, SPoC, CroW, R-MAC, GeM, and NetVLAD.The architectures produce descriptors of 256, 512, and 2048 dimensions, respectively.
- CNN-based global descriptor methods: CNN descriptors are tested with multi-scale processing, retrieval fine-tuning, descriptor whitening, and query expansion using αQE or diffusion.Fine-tuned and off-the-shelf networks use different whitening procedures, while multi-scale descriptors combine three image scales.
4. Results
Results show that the revisited Medium and Hard protocols are more challenging than the original evaluation, and no method dominates every dataset and protocol. Local features lead on ROxford, CNN methods on RParis, while combining both improves performance.
- Revisited vs. original: The original setup is close to the new Easy setup, whereas Medium and Hard are more challenging and Easy is nearly saturated.Subsequent experiments therefore use only Medium and Hard.
- State of the art evaluation: No single method achieves the highest score on every protocol and dataset.The comparison spans revisited Oxford and Paris with and without the R1M distractors.
- State of the art evaluation: Local-feature methods perform especially well on ROxford at large scale, while CNN-based methods dominate on RParis.
- State of the art evaluation: CNN fine-tuning consistently improves performance over off-the-shelf networks, while BoW-based approaches remain competitive at additional computational cost.
- State of the art evaluation: Query expansion can degrade performance for queries with few relevant images, particularly on the revisited datasets.The effect is attributed to similarity propagation beginning from CNN nearest neighbors that may include false positives.
- The best of both worlds: Starting diffusion from geometrically verified local-feature matches combines CNN similarity propagation with verified relevant images and improves performance.The experiment is expensive and is presented as a possible direction for improving CNN global descriptors.
5. Conclusions
The paper revisits Oxford and Paris with more reliable annotations and a hard 1M distractor set, then evaluates retrieval methods on the resulting benchmark. The experiments conclude that image retrieval remains open, especially at large scale and under difficult viewing conditions.
- New annotations and a 1M hard distractor set address weaknesses in the original Oxford and Paris benchmarks.
- Extensive evaluation establishes a testbed for future comparisons and finds image retrieval remains an open problem at large scale and under difficult viewing conditions.