Source-linked AI summary
Training Vision Transformers for Image Retrieval
Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, Hervé Jégou
TL;DR
Image retrieval has traditionally relied on convolutional encoders, leaving the suitability of vision transformers for metric learning underexplored. The paper introduces a Siamese transformer retrieval model trained with contrastive loss and differential entropy regularization. It establishes state-of-the-art category-level results and competitive particular-object retrieval, especially with short descriptors and comparable settings.
Problem
Plain transformers had not been studied for image-retrieval metric learning, despite their success in NLP and image classification.
Method
The paper trains Siamese vision transformers for image retrieval with contrastive loss and differential entropy regularization.
Results
The method establishes state-of-the-art category-level retrieval and is competitive with convolutional models for particular-object retrieval, especially with short representations.
Takeaways & Limitations
Transformers are an attractive alternative to convolutional backbones for image retrieval in comparable settings.
Takeaways & Limitations
The representations are assumed to be l2-normalized, making their inner product equivalent to cosine similarity.
Abstract
from arXiv · showhide
Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-based approach for image retrieval: we adopt vision transformers for generating image descriptors and train the resulting model with a metric learning objective, which combines a contrastive loss with a differential entropy regularizer. Our results show consistent and significant improvements of transformers over convolution-based approaches. In particular, our method outperforms the state of the art on several public benchmarks for category-level retrieval, namely Stanford Online Product, In-Shop and CUB-200. Furthermore, our experiments on ROxford and RParis also show that, in comparable settings, transformers are competitive for particular object retrieval, especially in the regime of short vector representations and low-resolution images.
1. Introduction
This work introduces Image Retrieval Transformers, adapting Siamese vision transformers and metric-learning objectives to image retrieval. The approach achieves state-of-the-art category-level results and competitive particular-object retrieval, while entropy regularization improves performance.
- Motivation: Transformers offer alternative inductive biases to convolutional architectures, motivating their study for image retrieval.The paper notes that plain transformers had not previously been studied in this context.
- Contribution: Image Retrieval Transformers adapt a Siamese transformer backbone to metric learning for category-level and particular-object retrieval.The model maps image pairs into a common feature space and combines contrastive learning with differential entropy regularization.
- Results: The method establishes the new state of the art on three popular category-level retrieval benchmarks.The introduction reports this as a principal result of the study.
- Results: For particular-object retrieval, 128-component descriptors are comparable to convolutional models operating at much higher resolution and FLOPS.This comparison is reported for ROxford and RParis.
- Method: Differential entropy regularization enhances the contrastive loss and improves performance overall.The regularizer is intended to favor uniformity in the representation space.
2. Related Work
Prior image-retrieval research developed convolutional encoders, metric-learning objectives, sampling strategies, and descriptor-compression methods. Related work also established transformers as competitive alternatives to convolutional architectures and explored entropy-based representation spreading.
- Transformers: Transformer-based vision models have become competitive alternatives to convolutional architectures after succeeding in NLP and image classification.Vision Transformers demonstrated competitive or superior image-classification performance, while using convolution-free transformer layers.
- Metric Learning: Metric-learning methods use classification prototypes or pairwise similarity signals, with contrastive loss strengthened by careful implementation and hard-negative sampling.Recent work reported contrastive loss outperforming more sophisticated losses and achieving state-of-the-art performance with external memory.
- Particular Image Retrieval: Particular-image retrieval evolved from local descriptors to end-to-end convolutional encoders, with spatial-to-vector descriptor compression as a key design choice.Subsequent methods introduced varied forms of supervision and end-to-end training.
- Differential Entropy Regularization: Entropy-based approaches spread descriptors or class centers more uniformly on hyperspheres or manifolds.The paper positions its regularization approach alongside prior work on uniformity, entropy estimation, and feature binarization.
- Differential Entropy Regularization: This work focuses on pairwise contrastive learning to prevent dimensional collapse and spread representations more uniformly on the hypersphere.The related method uses differential entropy regularization for representation geometry and quantization properties.
3. Methods
The paper adapts vision transformers for image retrieval using a shared metric-learning approach that combines contrastive learning with differential entropy regularization. The method addresses representation collapse and training instability while evaluating transformer and convolutional architectures under comparable settings.
- The method uses one transformer-based approach for both category-level and particular object retrieval, rather than separate task-specific techniques.
- IRT comprises off-the-shelf feature extraction, contrastive-loss fine-tuning, and entropy regularization that encourages uniformity in the output feature space.
- DeiT-Small is the primary transformer because its parameter count and FLOPS are comparable to ResNet-50, supporting fair retrieval comparisons.
- The contrastive objective pulls same-label representations together and separates unmatched representations only when their similarity exceeds margin β.
- Very low or high margins produce noisier gradients, while entropy regularization lowers gradient-noise measures and transformer features show less collapse than convolutional features.
- The differential entropy term maximizes nearest-neighbor distances, reducing feature collapse and limiting noisy interactions with hard negatives.
4. Experiments & Ablation Studies
The experiments evaluate transformer-based retrieval across category-level and particular-object benchmarks, then ablate supervision, entropy regularization, feature aggregation, and objective functions. Transformers achieve strong retrieval performance, while entropy regularization consistently improves descriptor learning across tasks and architectures.
- Evaluation: The evaluation covers category-level retrieval on SOP, CUB-200-2011, and In-Shop, plus particular-object retrieval on revisited Oxford and Paris benchmarks.Particular-object models are evaluated at 224×224 and 384×384 resolutions, with Medium and Hard splits reported for Oxford and Paris.
- Category-level retrieval: 2.6% absolute points at Recall@1: DeiT-S384 outperforms previous methods on SOP, while CUB-200-2011 improves by 2.5% points at Recall@1.The distilled DeiT-S variant performs best across reported K values on both SOP and CUB-200-2011.
- Category-level retrieval: 0.2% and 0.6% Recall@1 margins: DeiT-S384 and its distilled variant outperform previous In-Shop models.The compressed DeiT-S128 representation also outperforms most prior methods on SOP and CUB-200-2011, with exceptions noted in the passage.
- Particular-object retrieval: DeiT-S† at 224×224 outperforms equal-capacity ResNet-50 and higher-capacity ResNet-101 across Oxford and Paris benchmarks and descriptor sizes.The larger DeiT-B† is strongest among the reported models, and increasing resolution to 384×384 improves all models.
- Particular-object retrieval: 128-D descriptors close the gap with multiscale convolutional methods and achieve higher mAP for RParis, while using substantially fewer FLOPS.At 384×384, competing methods remain higher in mAP but are approximately 248% and 437% more expensive with respect to FLOPS.
- Ablations: Finetuning improves over off-the-shelf features, entropy regularization further improves performance across benchmarks, and the regularizer benefits both transformer and convolutional models.The ablation reports greater separation of positive and negative similarities after finetuning, with entropy spreading similarity values across a wider range.
- Ablations: The CLS token is strongest or at least competitive with average, max, and GeM pooling, suggesting learned feature aggregation by the transformer.DeiT-S also outperforms its convolutional counterpart across the tested classification-based and pairwise objective functions.
5. Conclusion
The paper adapts transformers to metric learning and image retrieval by combining contrastive learning with differential-entropy regularization. It establishes state-of-the-art category-level retrieval and finds transformers competitive with more complex convolutional models for particular-object retrieval, especially with short descriptors.
- Contribution: The paper adapts transformer architectures to metric learning and image retrieval using a differential-entropy regularizer that spreads vectors over the unit hypersphere.The regularizer improves performance for both transformer-based and convolutional models.
- Conclusion: The method establishes the new state of the art for category-level image retrieval.
- Conclusion: Transformers are competitive with convolutional backbones for particular-object retrieval in comparable settings, especially with short vector representations.The conclusion specifically contrasts them with convolutional models having much higher complexity.