PaperlayerView exact revision on arXiv

Source-linked AI summary

Instance-level Image Retrieval using Reranking Transformers

Fuwen Tan, Jiangbo Yuan, Vicente Ordonez

arXiv:2103.12236v3cs.CV

TL;DR

Instance retrieval must distinguish a particular object from its category, and commonly reranks global-descriptor search results using local-feature methods. RRTs learn image-pair similarity from global and local features, and outperform prior reranking approaches across reported settings while using fewer descriptors. The paper also reports further gains from joint optimization with the feature extractor.

  • Problem

    Instance retrieval identifies a particular object, and common reranking methods rely on local-feature techniques such as geometric verification, which is expensive and requires many descriptors.

    Source: §1 · p. 1

  • Method

    Reranking Transformers use global and local descriptors to learn image-pair similarity, replacing geometric verification or jointly training with the feature extractor.

    Source: §1 · p. 2

  • Results

    RRTs outperform prior reranking approaches across a variety of settings and use fewer descriptors than geometric verification and other local-feature methods.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    RRTs can be optimized jointly with the feature extractor, and the paper reports further gains from this joint optimization.

    Source: §6 · p. 8

    RRT memory use remains high for large-scale systems, despite using about half the DELG local descriptors in those experiments.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Instance-level image retrieval is the task of searching in a large database for images that match an object in a query image. To address this task, systems usually rely on a retrieval step that uses global image descriptors, and a subsequent step that performs domain-specific refinements or reranking by leveraging operations such as geometric verification based on local features. In this work, we propose Reranking Transformers (RRTs) as a general model to incorporate both local and global features to rerank the matching images in a supervised fashion and thus replace the relatively expensive process of geometric verification. RRTs are lightweight and can be easily parallelized so that reranking a set of top matching results can be performed in a single forward-pass. We perform extensive experiments on the Revisited Oxford and Paris datasets, and the Google Landmarks v2 dataset, showing that RRTs outperform previous reranking approaches while using much fewer local descriptors. Moreover, we demonstrate that, unlike existing approaches, RRTs can be optimized jointly with the feature extractor, which can lead to feature representations tailored to downstream tasks and further accuracy improvements. The code and trained models are publicly available at https://github.com/uvavision/RerankingTransformer.

1. Introduction

Instance-level retrieval seeks a particular object rather than its category, but reranking commonly depends on costly local-feature matching. The paper introduces Reranking Transformers (RRTs), which predict image-pair similarity and are presented as an efficient, learnable alternative.

1. Introduction

  • Instance retrieval identifies a specific object rather than its class, so systems commonly retrieve with global descriptors and rerank candidates using local features.Geometric verification is expensive and assumes rigid instances; geometric verification and ASMK require many local descriptors for performance.Source: §1 · pp. 1–2
  • RRTs directly predict image-pair similarity and are reported to outperform prior reranking methods across standard instance-recognition benchmarks.They are proposed as a drop-in replacement for geometric verification or for joint training with a global retrieval approach.Source: §1 · p. 2
  • 2.2 million parameters, versus 25 million in ResNet50, make RRTs lightweight; reranking the top 100 neighbors can run in a single pass.The method uses fewer local descriptors than existing approaches and is designed for easy parallelization.Source: §1 · p. 2

2. Related Work

  • CNN-derived local features support matching, while global descriptors compactly represent images; the paper follows retrieve-and-rerank rather than codebook-based matching.Prior work also learns local feature detection and representation using non-local suppression or attention.Source: §2 · p. 2
  • Prior reranking includes query expansion over local-neighbor graphs and diffusion over global affinity graphs, while RRTs learn image-pair similarity directly.Diffusion methods seek data-manifold structure through similarity propagation and are nontrivial to scale.Source: §2 · p. 2
  • Unlike single-image transformer predictions, RRTs model relations between image pairs; unlike SuperGlue, they use image-level rather than pixel-level supervision.SuperGlue learns local correspondences between images.Source: §2 · p. 2

3. Methodology

The method adapts transformer attention to pairwise image retrieval, processing global and local descriptors from both images to predict their similarity. It encodes descriptor type, location, and scale before producing a score trained against same-object labels.

3.1. Attention Modules in Transformers

  • Single-Head Attention forms each output as a weighted combination of value vectors, while Multi-Head Attention concatenates outputs from multiple attention heads.Both modules take query, key, and value sequences as input.Source: §3.1 · pp. 2–3

3.2. Model

  • Each image is represented by a global descriptor and local descriptors; local descriptors can also carry their pixel coordinates and extraction scale.The input sequence adds position, scale, and segment embeddings, with special tokens for summarizing both images and separating them.Source: §3.2 · p. 3
  • The model applies multiple transformer layers, using each layer's current feature sequence as the queries, keys, and values for attention.The layer design also includes a two-layer perceptron and layer normalization.Source: §3.2 · p. 3
  • A binary cross-entropy objective trains the final CLS-token representation, projected to a scalar, to distinguish same-object image pairs from different-object pairs.The target is one when the images depict the same object and zero otherwise.Source: §3.2 · p. 3

4. Experiments

Experiments evaluate RRT across landmark-retrieval and product-image benchmarks, including pretrained-descriptor and jointly optimized feature-learning settings. Implementation choices vary by dataset and descriptor source, and the authors report latency and memory costs.

4.1. Datasets

  • Experiments cover GLDv2, Revisited Oxford/Paris, and Stanford Online Products (SOP).GLDv2 testing uses 1,129 queries and 761,757 gallery images.Source: §4.1 · p. 4
  • GLDv2 training samples 322,008 images from 12,000 landmarks, representing 20% of its “v2-clean” split.Testing uses the standard retrieval test set.Source: §4.1 · p. 4
  • Revisited Oxford and Paris report mAP on Medium and Hard setups, while SOP reports R@K for metric learning.Oxford and Paris each include 70 queries; SOP contains 59,551 training and 60,502 test images.Source: §4.1 · p. 4

4.2. Implementation

  • DELG experiments use 500 local descriptors per image, where retrieval performance was observed to saturate, with global and local features projected to 128 dimensions.The RRT has 2,243,201 learnable parameters, about 9% of ResNet50’s parameter count.Source: §4.2 · p. 4
  • The reported tables compare geometric verification on Oxford/Paris using mAP for Medium and Hard (+R1M), and on GLDv2 using mAP@100 for public and private test sets.Source: Table 1 · p. 5, Table 2 · p. 5
  • Position embeddings are omitted for sparsely sampled DELG keypoints but used for SOP, where all positions are included.The authors posit that using all positions explains the SOP benefit.Source: §4.2 · pp. 4–5
  • On an NVIDIA P100, top-100 reranking takes 0.36 seconds in DELG experiments and 0.013 seconds in SOP experiments; the authors still consider memory costly at scale.DELG experiments use half as many local descriptors as DELG, and the authors identify quantization as a possible way to reduce memory footprint.Source: §4.2 · p. 5

5. Results

The experiments compare RRT with established reranking methods, test joint feature learning, and evaluate it across instance-retrieval benchmarks. Results show advantages over several reranking baselines, with performance depending on the benchmark and reranking setup.

5.1. Comparison with Geometric Verification

  • RRT consistently outperforms geometric verification on Revisited Paris and on the Hard setup of Revisited Oxford, while matching it on Oxford Medium with R1M.With v1 descriptors and top-200 reranking, the reported absolute improvements over GV are 3.4 on ROxf and 8.4 on RPar.Source: §5.1 · p. 5
  • On GLDv2, RRT beats both global-only retrieval and geometric verification, although reranking gains are smaller than on Revisited Oxford and Paris.With v2-clean descriptors on the private set, RRT improves over global-only retrieval by 2.8 and over GV by 2.7.Source: §5.1 · p. 5

5.2. Comparison with Query Expansion

  • RRT surpasses αQE in five of eight settings when reranking the top 100 neighbors, and in all settings when reranking the top 400.On the Hard setup with v1 descriptors, top-100 RRT trails αQE in three Revisited Paris settings; top-200 αQE reranking gains also exceed αQE alone across all four reported Hard settings.Source: §5.2 · p. 6, Table 3 · p. 6

5.3. Comparison with Aggregated Selective Match Kernel (ASMK)

  • RRT outperforms ASMK used for global retrieval or reranking in every reported setting on Revisited Oxford and Paris.ASMK reranking itself improves on DELG global retrieval, indicating complementarity between those approaches.Source: Table 4 · p. 6, §5.3 · p. 7

5.4. Feature Learning & RRT: Joint Optimization

  • Finetuning the feature extractor with RRT gives the best SOP reranking performance, improving global-only R@1 by 3.8.The frozen RRT also improves performance, while R@100 and R@1k remain unchanged because only the top 100 images are reranked.Source: §5.4 · p. 7, Table 6 · p. 7

5.5. Comparison with the State-of-the-Art

  • RRT performs favorably against most state-of-the-art methods, with exceptions on ROxf and ROxf+R1M.The compared methods differ in training data, CNN backbones, and number of local features, complicating direct comparisons.Source: Table 5 · p. 7, §5.5 · p. 8
  • Qualitative examples show that reranking with global and local descriptors captures finer image matches than global-only retrieval.The figure presents the top three neighbors before and after reranking, marking correct and incorrect neighbors with green and red borders.Source: §5.5 · p. 8, Figure 3 · p. 8
  • RRT gains larger improvements than SuperGlue on most settings, especially challenging cases, when both rerank the same initial retrieval lists.The comparison uses pretrained DELG descriptors for global retrieval; SuperGlue and RRT were trained on datasets with distributions different from the Revisited Oxford/Paris test sets.Source: §5.5 · p. 8, Table 7 · p. 8

6. Conclusion

The paper concludes that RRTs improve on prior reranking approaches across varied settings while using fewer descriptors. Their parallelizable single-pass reranking and joint optimization with the feature extractor are further contributions.

6. Conclusion

  • RRTs outperform prior reranking approaches across varied settings while using fewer descriptors than geometric verification and other local-feature methods.Reranking the top neighbors can be parallelized in a single forward pass, and joint optimization with the feature extractor can yield further gains.Source: §6 · p. 8

Supplementary Material

The supplementary material outlines further baseline, ablation, feature-extractor, and qualitative experiments, and identifies a separate document containing sampled GLDv2 training-image names.

Supplementary Material

  • The supplement discusses the choices of geometric verification, query expansion, and ASMK as baselines.Source: Supplementary Material · p. 9
  • Planned supplemental experiments include descriptor-count ablations, SuperPoint and ResNet101 feature-extractor studies, and keypoint-correspondence visualizations.The supplement also includes a discussion of the proposed method's limitations and additional qualitative examples.Source: Supplementary Material · p. 9
  • The names of sampled GLDv2 training images are provided in a separate document.Source: Supplementary Material · p. 9

7. Appropriate baselines

The section frames reranking as using test-time information, comparing RRT with geometric verification and αQE. Partial matches expose a limitation of global-descriptor cosine retrieval, while ASMK is included as a local-descriptor-based global retrieval approach.

7. Appropriate baselines

  • RRT, geometric verification, and αQE are treated as methods that leverage test-time information when comparing or reranking images.The authors describe incorporating test-time knowledge as the motivation for image reranking.Source: §7 · p. 9
  • For crop-like partial matches, irrelevant image content can hinder global-descriptor cosine similarity.The authors argue that this issue remains even with a large global descriptor.Source: §7 · p. 9
  • ASMK is tested as a global retrieval approach that clusters local descriptors into visual filters and aggregates them into global vectors.The section includes it because it also relies on local descriptors.Source: §7 · p. 9

8. Ablation on the number of local descriptors

The descriptor-count ablation compares geometric verification (GV) and RRT on Revisited Oxford. RRT’s performance saturates at 500 local descriptors, and it outperforms GV in most same-count comparisons.

8. Ablation on the number of local descriptors

  • DELG extracts 955/759/987 local descriptors per image on average for the corresponding ROxf query and gallery sets and sampled GLDv2 training set.The experiment varies the maximum descriptors per image using DELG pretrained on GLDv2 v2-clean and reports ROxf mAP.Source: §8 · p. 9
  • RRT performance saturates at 500 local descriptors, while both RRT and GV generally benefit from using more descriptors.The authors conjecture that descriptors from seven image scales can duplicate information at the same image locations.Source: §8 · p. 9
  • 585/465/655 distinct local descriptors per image are observed on average for the ROxf query and gallery sets and sampled GLDv2 training set.Descriptors are grouped by grid location to count distinct locations.Source: §8 · p. 9
  • RRT outperforms GV in four of six Medium experiments and consistently on Hard when using the same number of local descriptors.Table 8 reports mAP on Revisited Oxford for the descriptor-count ablation.Source: §8 · p. 9, Table 8 · p. 10

9. SuperPoint as the CNN backbone.

The section evaluates RRT with SuperPoint features to compare against SuperGlue using the same backbone, and separately reports ResNet101 results against GV and SuperGlue. RRT beats frozen-backbone SuperGlue on SOP, while its ResNet101 comparison varies by descriptor set and evaluation setting.

9. SuperPoint as the CNN backbone.

  • The SuperPoint comparison uses grayscale 320x320 inputs and the top 500 local descriptors; RRT’s global descriptor averages local responses.The setup also investigates position embeddings and does not finetune SuperGlue on SOP because pixel-level annotations are unavailable.Source: §9 · p. 10
  • With a frozen SuperPoint backbone, RRT outperforms SuperGlue on SOP, and finetuning SuperPoint yields the best performance.Reranking by either model significantly improves retrieval; RRT does not benefit from position embeddings in this task.Source: §9 · p. 10
  • With ResNet101 and v1 descriptors, RRT performs favorably to GV and SuperGlue on all settings; with v2-clean, it trails GV on ROxf-Medium but is favorable on the remaining settings.Table 10 reports mAP on the Medium and Hard Revisited Oxford/Paris setups.Source: §10 · p. 10, Table 10 · p. 10

11. Visualizing the correspondences

The correspondence visualization indicates that RRT is not good at learning pixel-wise keypoint correspondences. Instead, it learns distinct knowledge for computing image similarity.

11. Visualizing the correspondences

  • RRT’s attention-derived correspondences are not good pixel-wise keypoint matches; the model instead learns distinct knowledge for image similarity.The correspondences are derived from last-layer attention scores using linear sum assignment.Source: §11 · p. 11

12. Limitation

The paper identifies reduced interpretability of its predicted similarity score and difficulty with large domain shifts as limitations. Its transfer tests span different datasets and instance categories, but transfer is only to a certain extent.

12. Limitation

  • The predicted similarity score is less interpretable than homography, which explicitly models image-pair alignment.Source: §12 · p. 11
  • The DELG experiment trains on Google Landmarks v2 and tests on Revisited Oxford/Paris, while SOP training and test sets have no overlapping instance categories.Source: §12 · p. 11
  • RRT can transfer across instance categories to a certain extent, but large domain shifts may challenge both the reranker and feature extractor.Source: §12 · p. 11

13. More qualitative examples

Qualitative examples compare RRT with global-only retrieval on Stanford Online Products and with geometry verification on Revisited Oxford/Paris. The reported advantages include challenging object matches and large viewpoint changes, alongside RRT failure cases.

13. More qualitative examples

  • On Stanford Online Products, RRT outperforms global-only retrieval on partial matches, articulated objects, and irrelevant context.The examples include rigid and deformable objects.Source: §13 · p. 11, Figure 6 · p. 12
  • On Revisited Oxford/Paris, RRT performs favorably to geometry verification under large viewpoint variations, though examples (E) and (F) show RRT failures.For two queries of the same landmark with a large viewpoint change, RRT predicts the same top-ranked images while geometry verification predicts different sets.Source: §13 · p. 11, Figure 7 · p. 13

Related papers

Loading 2103.12236v3…