Source-linked AI summary
Deep Image Retrieval: Learning global representations for image search
Albert Gordo, Jon Almazan, Jerome Revaud, Diane Larlus
TL;DR
Instance-level retrieval needs representations that distinguish particular objects rather than only semantic categories. The paper learns a compact global descriptor with retrieval-specific ranking, content-dependent region pooling, and cleaned landmark data, achieving stronger retrieval performance than global and many costly local methods.
Problem
ImageNet-pretrained features are optimized for semantic categories and can be too robust to intra-class variation for distinguishing particular instances, while local matching remains state of the art.
Method
A Siamese triplet-ranking network learns R-MAC convolutional and projection weights, while an RPN learns which image regions to pool; noisy landmark data are automatically cleaned for training.
Results
The compact fixed-length representation significantly outperforms previous global-descriptor approaches and surpasses most methods requiring local descriptor indexing and spatial verification.
Takeaways & Limitations
A retrieval-specific, single-pass global signature can match or exceed more complex retrieval systems on standard datasets without costly spatial verification at test time.
Takeaways & Limitations
The RPN is trained separately after the R-MAC representation because jointly weighting the ranking and proposal losses produced unstable results.
Abstract
from arXiv · showhide
We propose a novel approach for instance-level image retrieval. It produces a global and compact fixed-length representation for each image by aggregating many region-wise descriptors. In contrast to previous works employing pre-trained deep networks as a black box to produce features, our method leverages a deep architecture trained for the specific task of image retrieval. Our contribution is twofold: (i) we leverage a ranking framework to learn convolution and projection weights that are used to build the region features; and (ii) we employ a region proposal network to learn which regions should be pooled to form the final global descriptor. We show that using clean training data is key to the success of our approach. To that aim, we use a large scale but noisy landmark dataset and develop an automatic cleaning approach. The proposed architecture produces a global image representation in a single forward pass. Our approach significantly outperforms previous approaches based on global descriptors on standard datasets. It even surpasses most prior works based on costly local descriptor indexing and spatial verification. Additional material is available at www.xrce.xerox.com/Deep-Image-Retrieval.
1 Introduction
The paper addresses weak deep-learning performance in instance-level image retrieval by learning retrieval-specific representations, region pooling, and cleaner training data. Its architecture combines ranking-based representation learning with region proposals to produce compact descriptors that outperform global and more complex retrieval approaches.
- Instance-level retrieval remained dominated by local descriptor matching and spatial verification despite deep learning’s success in other vision tasks.
- ImageNet classification features are poorly matched to instance retrieval because their robustness to intra-class variation can hinder distinguishing particular objects within the same category.
- A three-stream Siamese network uses triplet ranking loss to optimize the differentiable R-MAC representation specifically for image retrieval.
- An automatic cleaning process improves learning from the noisy Landmarks dataset, which contains many mislabeled and false-positive images.
- The learned pooling mechanism replaces R-MAC’s rigid grid with region proposals predicted from image content, with quantitative and qualitative evidence favoring proposals.
- The resulting architecture produces compact fixed-length vectors in one forward pass and outperforms global-descriptor methods as well as more complex methods using keypoint matching and spatial verification.
2 Related Work
Prior retrieval systems use local matching or CNN features adapted with handcrafted pooling and re-ranking. This paper extends global R-MAC representations by learning their parameters and pooling regions for retrieval.
- Conventional image retrieval: Conventional instance retrieval uses bag-of-features representations, inverted files, descriptor-matching approximations, and spatial verification for re-ranking.
- CNN-based retrieval: Off-the-shelf CNN features improve on standard global descriptors but remain below the retrieval state of the art, motivating methods for scale, cropping, and clutter robustness.
- CNN-based retrieval: R-MAC aggregates CNN activations from a fixed spatial grid into a fixed-length global vector, while this work learns the representation parameters and improves region pooling.
3 Method
The method learns a retrieval-specific global descriptor end to end using a Siamese ranking network and replaces fixed region layouts with content-dependent proposals. Its differentiable pipeline encodes images efficiently in a single forward pass.
- Learning the representation: R-MAC’s differentiable components allow its representation weights to be learned end to end with a three-stream Siamese network and triplet ranking loss.
- R-MAC revisited: A pre-trained CNN extracts local activation features, which are max-pooled over overlapping cells in a multi-scale rigid grid.
- Learning the representation: The triplet loss back-propagates through the three network streams to update convolutional and projection layers, directly optimizing retrieval ranking.
- Beyond fixed regions: proposal pooling: The rigid grid can misalign with the object of interest and include background, weakening the many-to-many region matching used by R-MAC signatures.
- Beyond fixed regions: proposal pooling: A fully convolutional RPN predicts and refines candidate regions, then non-maximum suppression selects final proposals that replace the rigid grid.
- Beyond fixed regions: proposal pooling: The RPN is trained with a multi-task objective combining object-versus-background classification and smooth ℓ1 regression losses.
- Beyond fixed regions: proposal pooling: The RPN is trained after the R-MAC representation with fixed convolutional layers because jointly weighting ranking and proposal losses produced unstable results.
- Descriptor extraction: At test time, proposal generation, regional pooling, embedding, aggregation, and normalization occur in one forward pass for high-resolution images.
4 Leveraging large-scale noisy data
The paper cleans a noisy landmark dataset by retaining geometrically consistent image groups, then estimates object boxes from verified matches to support learned region selection.
- Dataset and noise: 214K images from 672 landmarks contain varied views, distractors, and unrelated images because they were collected through unverified image-search queries.The processed Landmarks-full set contains about 192,000 images across 586 landmarks after download, class, and benchmark-overlap filtering.
- Dataset cleaning: The cleaning pipeline matches image pairs with invariant keypoints and affine spatial verification, builds a graph, and removes low-scoring edges.Connected components represent different landmark profiles, such as outside and inside views of St Paul’s Cathedral.
- Dataset cleaning: 49,000 images remain in Landmarks-clean after retaining only the largest connected component for each landmark.The retained data include 42,410 training images and 6,382 validation images across 586 landmarks.
- Bounding box estimation: Verified keypoint locations provide object cues because the landmark remains consistently matched while distractor backgrounds and foreground objects vary.These matches are used to automatically estimate bounding boxes for the images.
- Bounding box estimation: Initial boxes enclose matched keypoints, and multiple boxes for one image are combined using a geometric median.The boxes are then propagated across graph edges through affine transformations.
- Bounding box estimation: A diffusion process updates boxes using affine transformations until convergence, improving their locations and consistency across images.The update uses α = 0.1, and repeated updates are merged robustly with a geometric median.
5 Experiments
Experiments evaluate learned retrieval representations, proposal-based pooling, and comparisons with prior methods across standard datasets. Clean training data, ranking-based fine-tuning, and learned proposals improve retrieval accuracy while retaining a compact global representation.
- Datasets: Experiments cover five standard retrieval datasets, including Oxford, Paris, their distractor-augmented versions, and Holidays.Oxford 5k and Paris 6k contain 5,062 and 6,412 images, respectively; larger variants add 100k distractors.
- Representation fine-tuning: Clean-data fine-tuning and ranking-based learning improve the R-MAC baseline, while a universal Landmarks-trained PCA model causes a slight performance decrease.The universal PCA avoids generating dataset-specific models, and classification fine-tuning already yields large improvements over the original results.
- Proposal pooling: 32–64 learned proposals already outperform rigid regions, and proposal recall is significantly higher than the rigid grid even with few regions.The learned proposals are centered on objects of interest and align with entire landmarks on Oxford 5k images.
- State-of-the-art comparison: The method significantly outperforms global-descriptor methods on all datasets, exceeding one comparison by more than 15 mAP points.The comparison excludes spatial verification and query expansion at test time.
- State-of-the-art comparison: 89.1 mAP is reached on Oxford 5k without costly spatial verification, while the approach is on equal footing with or improves most more complex methods.Several competing methods require larger memory footprints, spatial verification, or query expansion.
6 Conclusions
The paper presents a scalable image-retrieval method that learns both the representation and the pooled regions for the retrieval task. It substantially outperforms global-signature methods and matches or exceeds more complex approaches without costly pre- or post-processing.
- Conclusions: The method encodes images as compact global signatures compared with a dot-product, using retrieval-specific training and predicted regions of interest.A Siamese ranking-loss architecture learns the representation, while a region proposal network learns pooling locations.
- Conclusions: Retrieval-specific training and learned region pooling are the two central contributions beyond prior fixed-grid representations.The approach combines a three-stream Siamese architecture with a triplet ranking loss and region proposal networks.
- Conclusions: The approach very significantly outperforms state-of-the-art methods using global signatures and is on par with or better than more complex methods.It avoids complex pre-processing and post-processing.
A Qualitative results
Qualitative examples show that retrieval-specific training changes both ranking behavior and the image content that activates deep features. The trained method improves retrieval accuracy over the R-MAC baseline, while many neurons shift toward instance-relevant activations.
- Retrieval examples: The trained model consistently produces more accurate Oxford 5k rankings than the untrained VGG16 R-MAC baseline.Several correctly retrieved images rank far below the baseline’s results.
- Feature activations: Before training, many conv5_3 neurons activate on semantic patches such as shoulders, bow ties, waists, or sunglasses across different instances.These activations are undesirable for instance-level retrieval because they do not identify the same instance.
- Feature activations: After training, many neurons are repurposed toward patches belonging to the same instance, such as dome-like structures replacing shoulder activations.The qualitative change indicates more retrieval-relevant activation patterns, although the sunglasses neuron remains incorrectly repurposed.
- Retrieval examples: Figure 5 encodes the R-MAC baseline in red and the ranking-loss-plus-proposals method in blue, with borders marking positive, null, and negative images.The figure combines ranked retrieval examples with average-precision curves for selected Oxford queries.
B Comparing architectures: VGG16 vs ResNet-50
The comparison evaluates VGG16 and ResNet-50 under direct, classification-trained, and retrieval-trained settings, with and without learned region proposals. ResNet-50 generally performs better, especially beyond Oxford 5k, while both architectures outperform fixed-length representation methods and match or exceed more costly approaches.
- Training and architecture comparison: Table 4 compares VGG16 and ResNet-50 without specific training, with classification training, and with retrieval ranking training.All settings use the initial regular grid without a region proposal network.
- Training and architecture comparison: ResNet-50 has a noticeable lead over VGG16, particularly for the baseline approach, and retains a lead after retrieval training.The comparison also considers two image resolutions.
- Region proposals: With proposals replacing the rigid grid, proposals have a smaller impact on ResNet-50 than on VGG16.The authors suggest ResNet-50 already leverages rigid-grid information more effectively and may need less proposal granularity.
- Comparison with prior work: On Oxford 5k, VGG16 and ResNet-50 obtain similar results, while ResNet-50 performs significantly better on the remaining datasets.Both architectures outperform fixed-length representation methods and obtain comparable or better results than more complex, costly methods.