Source-linked AI summary
$R^{2}$Former: Unified $R$etrieval and $R$eranking Transformer for Place Recognition
Sijie Zhu, Linjie Yang, Chen Chen, Mubarak Shah, Xiaohui Shen, Heng Wang
TL;DR
Visual place recognition needs efficient matching and reranking beyond geometric verification alone. R2Former unifies transformer-based retrieval and reranking using local correlations, attention, and coordinates, and reports state-of-the-art performance with substantially lower computational cost.
Problem
Conventional VPR uses CNN aggregation for retrieval and RANSAC reranking, but geometric verification overlooks local correlations and attention while requiring substantial inference time and memory.
Method
R2Former unifies global retrieval and reranking with transformer modules using class and patch tokens, local correlations, attention values, and patch coordinates.
Results
R2Former outperforms retrieval-only and retrieval-plus-reranking state-of-the-art methods across major VPR datasets while achieving > 4.7× QPS and < 22% memory consumption versus previous methods.
Takeaways & Limitations
Vision-transformer tokens are comparable to or better than CNN local features for reranking, and the reranking module can be used with CNN or transformer backbones.
Abstract
from arXiv · showhide
Visual Place Recognition (VPR) estimates the location of query images by matching them with images in a reference database. Conventional methods generally adopt aggregated CNN features for global retrieval and RANSAC-based geometric verification for reranking. However, RANSAC only employs geometric information but ignores other possible information that could be useful for reranking, e.g. local feature correlations, and attention values. In this paper, we propose a unified place recognition framework that handles both retrieval and reranking with a novel transformer model, named $R^{2}$Former. The proposed reranking module takes feature correlation, attention value, and xy coordinates into account, and learns to determine whether the image pair is from the same location. The whole pipeline is end-to-end trainable and the reranking module alone can also be adopted on other CNN or transformer backbones as a generic component. Remarkably, $R^{2}$Former significantly outperforms state-of-the-art methods on major VPR datasets with much less inference time and memory consumption. It also achieves the state-of-the-art on the hold-out MSLS challenge set and could serve as a simple yet strong solution for real-world large-scale applications. Experiments also show vision transformer tokens are comparable and sometimes better than CNN local features on local matching. The code is released at https://github.com/Jeff-Zilence/R2Former.
1 ByteDance 2Center for Research in Computer Vision, University of Central Florida
The listed authors are affiliated with ByteDance and the University of Central Florida’s Center for Research in Computer Vision.
- The contact addresses use ByteDance and UCF domains.
- The passage provides author contact information rather than research findings.
1. Introduction
VPR commonly separates global retrieval from reranking, but conventional reranking relies heavily on geometric verification and can be costly. R2Former unifies both stages with transformers, incorporating richer local-pair information while improving efficiency and benchmark performance.
- VPR localizes query images by matching them against reference images for applications including robotics, navigation, autonomous driving, and augmented reality.
- Conventional VPR pipelines use aggregated CNN features for global retrieval and RANSAC-based geometric verification for reranking.
- The introduction identifies uncertainty about whether vision-transformer tokens can match CNN local features for local matching.
- R2Former integrates retrieval and reranking with transformer layers using token correlations, attention values, and patch coordinates.
- > 4.7× QPS and < 22% memory consumption are reported versus previous methods.
- The method reports state-of-the-art performance across a wide range of VPR datasets with lower inference latency and memory consumption.
2. Related Work
Related VPR methods typically rely on CNN-based global descriptors and increasingly add local-feature reranking. The proposed framework instead uses transformer-derived class and patch tokens throughout its retrieval and reranking pipeline.
- Predominant VPR methods use CNN feature extractors with trainable aggregation or lightweight pooling layers for global representation.
- Recent state-of-the-art systems rerank retrieved candidates using RANSAC verification or attention-selected local features.
- Vision transformers represent image patches as tokens and can provide competitive global retrieval performance.
- The framework extracts global and local features as class and patch tokens, respectively, then selects important local features using attention.
3. R2Former
R2Former trains global retrieval and reranking within a transformer-only framework. It selects informative tokens, represents candidate pairs with correlation, attention, and position data, and aggregates them into match scores.
- 3.1. Problem Formulation and Training Objective: The framework has global-retrieval and reranking stages that learn embeddings for positive and negative query-reference pairs.
- 3.1. Problem Formulation and Training Objective: Global retrieval uses a margin triplet loss over query, positive, and negative global embeddings.
- 3.1. Problem Formulation and Training Objective: The reranking module classifies query-reference pairs using two-logit outputs trained with cross-entropy loss.
- 3.2. Global Retrieval Module: The shared vision-transformer encoder produces class and patch tokens without separate aggregation or key-point extraction modules.
- 3.2. Global Retrieval Module: Attention from the final transformer layer estimates patch importance through each patch’s contribution to the class-token global feature.
- 3.3. Reranking Transformer Module: The reranking correlation matrix stores patch coordinates, attention values, and cosine similarities across 500 × 500 local-feature pairs.
- 3.3. Reranking Transformer Module: Selecting five nearest neighbors per token reduces computation before transformer blocks aggregate pair information into a binary match score.
4. Experiment
Experiments evaluate R2Former across major VPR datasets, comparison protocols, computational costs, reranking methods, backbones, components, and training strategies. The method achieves strong retrieval performance while reducing reranking cost and memory, with separate training and global hard-negative mining providing the best reported strategy.
- Datasets and evaluation: R2Former is trained on MSLS and evaluated on MSLS Val, MSLS Challenge, Pitts30k, and Tokyo24/7 using standard dataset splits.Urban scenarios can additionally use finetuning on Pitts30k.
- Comparison with state-of-the-art: 2.9% and 9.1% absolute R@1 improvements are reported on MSLS Val and MSLS Challenge, respectively, over state-of-the-art methods.The comparison includes major retrieval and retrieval-plus-reranking baselines.
- Comparison with state-of-the-art: R2Former generalizes well from MSLS to other datasets and significantly outperforms the best VG benchmark models with comparable computational cost.Table 2 follows the protocol of training on MSLS and testing directly without finetuning.
- Computational cost: 47.17 ms per query on four GPUs delivers a 20.2× speedup over previous reranking methods, while MSLS validation requires 4.79 GB rather than 908.3 GB for Patch-NetVLAD-p.The reranking module uses one GPU network forward pass and stores 500 local features per image with dimensions 128 + 3.
- Computational cost: 244.01 GB stores features for datasets exceeding 1M reference images, reducible to 122 GB with float16 without observed performance drop on R-SF.Other reranking methods typically require more than four times the memory in this setting.
- Reranking and backbone ablations: The reranking module significantly outperforms RANSAC and functions as a generic component across CNN and transformer backbones.ViT-Small tokens perform slightly better than ResNet50+GeM local features under comparable global retrieval and reranking settings.
- Training strategy: Global hard-negative mining improves reranking training, while separate training of retrieval and reranking achieves the best performance; end-to-end training offers a simplicity–performance trade-off.All reranking strategies outperform previous state-of-the-art methods on R@5.
5. Conclusion
R2Former unifies retrieval and reranking for place recognition using transformers alone. It achieves strong performance with lower inference time and memory consumption, while leaving room for future geometric modeling.
- R2Former unifies global retrieval and reranking within a transformer-only place-recognition framework.
- Vision transformer tokens are comparable to or better than CNN local features for reranking.
- The method outperforms previous methods on major datasets with lower inference time and memory consumption.