Source-linked AI summary
End-to-end Learning of Deep Visual Representations for Image Retrieval
Albert Gordo, Jon Almazan, Jerome Revaud, Diane Larlus
TL;DR
Instance-level retrieval needs representations that distinguish particular objects, but prior deep methods were limited by training data, architecture, and training procedure. The paper addresses these issues with cleaned landmark data, an end-to-end R-MAC-based network trained using siamese triplet ranking. Its compact representations outperform prior methods, including costly matching-and-verification systems, while supporting compression.
Problem
Deep learning had been less successful for instance-level image retrieval, motivating representations learned specifically to distinguish particular object instances.
Method
The method automatically cleans landmark data and trains a differentiable R-MAC-based architecture with a three-stream siamese network and triplet loss.
Results
The representation significantly outperforms state-of-the-art global-signature and costly matching-and-verification methods across several retrieval benchmarks, including with 64- or 128-byte codes.
Takeaways & Limitations
Compact global signatures can support accurate instance-level retrieval while being faster and more memory-efficient than methods based on costly matching and verification.
Takeaways & Limitations
Multi-resolution descriptors add approximately three times the feature-extraction cost when computed on the query and database sides.
Abstract
from arXiv · showhide
While deep learning has become a key ingredient in the top performing methods for many computer vision tasks, it has failed so far to bring similar improvements to instance-level image retrieval. In this article, we argue that reasons for the underwhelming results of deep methods on image retrieval are threefold: i) noisy training data, ii) inappropriate deep architecture, and iii) suboptimal training procedure. We address all three issues. First, we leverage a large-scale but noisy landmark dataset and develop an automatic cleaning method that produces a suitable training set for deep retrieval. Second, we build on the recent R-MAC descriptor, show that it can be interpreted as a deep and differentiable architecture, and present improvements to enhance it. Last, we train this network with a siamese architecture that combines three streams with a triplet loss. At the end of the training process, the proposed architecture produces a global image representation in a single forward pass that is well suited for image retrieval. Extensive experiments show that our approach significantly outperforms previous retrieval approaches, including state-of-the-art methods based on costly local descriptor indexing and spatial verification. On Oxford 5k, Paris 6k and Holidays, we respectively report 94.7, 96.6, and 94.8 mean average precision. Our representations can also be heavily compressed using product quantization with little loss in accuracy. For additional material, please see www.xrce.xerox.com/Deep-Image-Retrieval.
1 Introduction
The paper targets instance-level retrieval with representations and training explicitly optimized to distinguish particular object instances. It combines cleaned landmark data, an end-to-end learned R-MAC-based architecture, and ranking-based training to improve retrieval.
- Instance-level retrieval seeks all database images containing the same object instance as a query image, supporting applications such as reverse image search and annotation transfer.
- Deep learning has improved many vision tasks but has been less successful for instance-level image retrieval.
- The proposed method learns retrieval-specific representations that distinguish particular objects, including objects from the same semantic class.
- The architecture builds on R-MAC, using multi-scale regional CNN descriptors aggregated into a compact fixed-length vector that is moderately robust to scale and translation.
- Training uses a three-stream siamese network with triplet loss and learns both network weights and image-content-dependent pooling regions.
- The resulting network produces a compact fixed-length image vector in one forward pass, while multi-resolution representations improve results for scale changes.
- The method automatically cleans a large noisy Landmarks dataset to remove mislabeled images and estimate landmark locations without manual intervention.
- Extensive experiments report results that largely outperform prior methods across four standard retrieval benchmarks, including costly approaches requiring subsequent matching.
2 Related work on image retrieval
Prior retrieval systems used local-feature matching, aggregated global descriptors, or CNN features, while later work increasingly emphasized task-specific finetuning and ranking losses. R-MAC provides the immediate foundation for this paper’s end-to-end approach.
- Early retrieval methods relied on bag-of-features representations, large vocabularies, and inverted files, often supplemented by descriptor matching and spatial verification.
- Fisher Vector and VLAD methods aggregate local patches into global image representations and can be combined with postprocessing such as query expansion.
- CNN-based retrieval progressed from off-the-shelf ImageNet activations toward methods addressing robustness to scaling, cropping, and image clutter.
- R-MAC aggregates CNN activation features from fixed spatial regions into a fixed-length vector without distorting image aspect ratio, achieving results close to the state of the art with query expansion.
- Finetuning classification models on landmark images improved retrieval, but subsequent work highlighted the importance of combining a suitable representation with a ranking loss.
3 Leveraging large-scale noisy data
The paper converts a noisy landmark dataset into training data by matching images within classes, retaining coherent graph components, and propagating automatically estimated bounding boxes. This supplies data for retrieval-specific representation learning.
- The Landmarks dataset contains approximately 214k images from 672 landmark sites collected through image-search queries, including unrelated images and diverse visual profiles.
- After broken URLs, small classes, and benchmark overlaps were removed, Landmarks-full contained about 192,000 images across 586 landmarks.
- The cleaning procedure aims to produce a dataset containing images of the same particular object or scene while retaining variation in viewpoint, scale, lighting, and clutter.
- Pairwise invariant keypoint matching and spatial verification provide scores used to construct an image graph for each landmark class.
- The procedure retains only the largest connected component of each landmark graph and discards other components to improve profile consistency.
- Verified keypoint matches identify the consistently visible landmark, while varying distractors tend to remain unmatched.
- Bounding boxes are initialized from matched keypoints and refined by diffusion across graph edges using affine transformations until convergence.
- The cleaned dataset is then used to learn image representations tailored for retrieval.
4 Learning to rank: an end-to-end approach
The paper turns R-MAC into a differentiable end-to-end architecture and trains it for retrieval with a three-stream siamese network and triplet ranking loss. Experiments show that retrieval-specific training, careful triplet sampling, and suitable initialization improve accuracy while retaining efficient global descriptors.
- 4.1 The R-MAC baseline: R-MAC integrates multi-scale regional pooling, normalization, whitening, and aggregation into a compact global image representation.Its fixed-length vector is independent of image size and region count, and image representations can be compared with a dot-product.
- 4.1 The R-MAC baseline: Because the R-MAC operations are differentiable, convolutional and fully connected replacement layers can be optimized end to end.The multi-scale pooling corresponds to differentiable ROI pooling, while learned layers replace PCA.
- 4.2 Learning to retrieve: The proposed three-stream siamese network jointly processes query, relevant, and irrelevant images with shared weights and a triplet ranking loss.The loss enforces that the relevant image is closer to the query than the irrelevant image; shared weights support varying image sizes and aspect ratios.
- 4.3 Practical considerations: Useful triplet sampling is essential because random sampling usually produces zero-loss triplets that do not improve the model.The procedure mines losses after extracting features from randomly selected training samples.
- 4.4.2 Results: Ranking-loss finetuning is the best-performing strategy, while training on Landmarks-full significantly worsens accuracy compared with Landmarks-clean.Classification finetuning already improves over ImageNet pretraining, but ranking loss with cleaned data performs better.
- 4.4.2 Results: The trained descriptor remains efficient at inference: one image takes approximately 150 ms to encode, and similarities use a dot-product.The reported throughput is about 7 images per second on a single GPU, while millions of comparisons per second are possible on a standard processor.
5 Improving the R-MAC representation
The paper improves R-MAC by replacing rigid region pooling with learned proposals and by combining descriptors across image resolutions. These changes target better object coverage and scale matching while preserving compact representations and search-time efficiency.
- Region proposal network: Rigid grids can misalign with the object and include background, weakening many-to-many region matching.These problems are especially pronounced when the retrieved object occupies a small portion of the image.
- Region proposal network: A region proposal network replaces R-MAC’s rigid multi-scale grid to focus pooling on relevant image regions.The RPN uses approximate landmark bounding boxes obtained during dataset cleaning.
- Region proposal network: Proposal regions typically cover objects more tightly, and increasing proposals improves coverage and many-to-many matching.The benefit follows from greater overlap with the region of interest, even when proposals are not exactly aligned.
- Multi-resolution: Multi-resolution descriptors combine representations from resized images to improve matching between objects appearing at different scales.The scheme can be applied on the query side, database side, or both; query-side and database-side processing are complementary.
- Multi-resolution: Multi-resolution increases feature-extraction cost to approximately three times, while search-time and storage costs remain unchanged.The descriptor can be computed for both query and database images before comparison.
- Experiments: With VGG16, the proposal network significantly improves accuracy over the rigid grid when using 128 regions or more.Table 2 compares both fixed-grid and proposal pooling before and after ranking-loss finetuning across datasets and training settings.
6 Evaluation of the complete approach
The complete system combines the learned R-MAC representation with query expansion and database-side augmentation, achieving strong accuracy and scalability across retrieval benchmarks. Performance depends on choosing neighborhood sizes appropriately, while compression preserves a favorable performance–storage trade-off.
- The final method integrates the learned representation with query expansion and database-side feature augmentation.These are the two additional improvements used in the complete evaluation system.
- Large QE neighborhoods can degrade Oxford 5k accuracy, whereas Paris is less affected because its queries have many relevant items.The effect is observed independently of whether DBA is used.
- Weighted DBA generally helps, but inappropriate QE neighborhoods can worsen results; the authors suggest k′ = 20 with k = 1 or k = 2.QE and DBA can reinforce each other when their neighborhood sizes are selected appropriately.
- The method consistently outperforms global-representation competitors across datasets and exceeds the best competitor by more than 14 mAP points on Paris 106k.The comparison includes Oxford 5k, Paris 6k, Oxford 105k, and Paris 106k.
- On Holidays, accuracy falls from 94.8 to 90.3 without manual rotation, while three automatically rotated queries raise it to 92.9.The automatic approach is three times slower at query time but avoids oracle intervention.
- Training improves Oxford accuracy from 69.4 without training to 84.1 or 86.1 with single-resolution or multi-resolution testing.The result supports the importance of architecture and end-to-end training tailored to retrieval.
- Product-quantized 64-byte codes outperform most state-of-the-art 2,048-byte codes, enabling hundreds of millions of images on one 64 Gb machine.PQ provides the strongest performance-versus-size trade-off among the compression methods evaluated.
7 Conclusions
The paper presents a compact, scalable retrieval system built from cleaned landmark data, triplet-based training, and a differentiable R-MAC architecture. Across benchmarks, it outperforms both global-signature and more costly matching-based methods, including with short codes.
- The method encodes images as compact global signatures compared using the dot-product.
- Its three key ingredients are automatically cleaned training data, a triplet-based ranking loss, and a differentiable R-MAC network with region proposals.
- The representation significantly outperforms state-of-the-art methods using global signatures and costly matching or verification, while being faster and more memory-efficient.The advantage also holds for short 64- or 128-byte codes.