$R^{2}$Former: Unified $R$etrieval and $R$eranking Transformer for Place Recognition

• Conventional VPR retrieves candidates with aggregated CNN features and often reranks top-k matches using RANSAC, but this geometric-only process is costly and may not scale to large databases or high QPS.The passage reports typical RANSAC reranking costs of over 1 second and 1 MB per image, with scalability concerns for databases exceeding 1 million images.Source: §1 · p. 1
• R2Former uses the class token for global retrieval and other image tokens as local features, reranking pairs from token correlations, xy coordinates, and attention information.Its retrieval and reranking modules can be trained end-to-end or tuned alternately, and the reranker can also be used with CNN or transformer backbones.Source: §1 · p. 2
• The paper reports that vision-transformer tokens are comparable to CNN local features for reranking and can sometimes perform better in local matching.This addresses uncertainty about whether transformer tokens retain useful local information despite global token correlations.Source: §1 · p. 2
• R2Former achieves state-of-the-art VPR performance with less than 22% of previous methods’ inference latency and memory consumption, over 4.7× QPS, and over 20× multi-GPU reranking speedup.The speed and memory comparisons are against previous methods [26] [53].Source: §1 · p. 2

Visual Place Recognition

• Predominant VPR methods use CNN feature extractors with trainable aggregation or pooling, while CosPlace formulates the task as classification using orientation information [3] [4] [5] [6] [26] [53] [24].Earlier approaches used nearest-neighbor search on aggregated hand-crafted features; later work further improved global representations following NetVLAD [3].Source: §2 · p. 2
• Leading VPR methods pair global retrieval with RANSAC reranking, while related-task learned matchers either generalize poorly to VPR or use a different problem setting [26] [53] [31] [48] [29].R²Former integrates retrieval and reranking in a unified VPR framework using pure transformers.Source: §2 · p. 2

Vision Transformer

• Vision transformers treat image patches as tokens; prior VPR work used vanilla ViT for global retrieval or transformers atop CNN backbones, leaving transformer tokens’ local-matching performance unexplored.The passage notes vanilla ViT may need large-scale training data for CNN-comparable results, while DeiT’s data-efficient strategy outperforms CNN on ImageNet-1k [17] [27] [51].Source: §2 · pp. 2–3

3.1. Problem Formulation and Training Objective

• Global retrieval learns embeddings with margin triplet loss from positives within 10 meters and negatives beyond 25 meters; reranking uses binary cross-entropy.Training first freezes retrieval while reranking learns from randomly selected negatives among the database’s top-k hardest candidates, then finetunes both modules together with partial negative mining.Source: §3.1 · p. 3

3.2. Global Retrieval Module

• A shared ViT encoder produces global features from its class token and local features from penultimate-layer patch tokens, without separate aggregation or keypoint modules.Linear heads reduce the global and local features to 256 and 128 dimensions, respectively, and positional embeddings are interpolated to support arbitrary input resolutions.Source: §3.2 · p. 4

3.3. Reranking Transformer Module

• The reranker selects the 500 patches with highest class-token attention and stores each local feature with its attention value and x,y coordinates.This attention-based selection reduces the number of local features saved for large-scale use; each stored feature has 131 dimensions.Source: §3.3 · p. 4
• Reranking encodes every selected feature pair’s coordinates, attention values, and cosine similarity, then uses two transformer blocks to classify image pairs as matches or non-matches.The pairwise correlation matrix has shape 500 × 500 × 7; the model retains each token’s five nearest neighbors to reduce computation before aggregation.Source: §3.3 · p. 5

4.1. Datasets and Evaluation Metrics

• Experiments train on MSLS, finetune on Pitts30k for urban datasets, evaluate recall@k at a 25-meter threshold, and rerank top-100 ViT-S candidates.Evaluation includes MSLS Val and the hold-out MSLS Challenge set, Pitts30k, Tokyo 24/7, R-SF, and St Lucia; the default ViT-S backbone has 12 layers and 384 dimensions.Source: §4.1 · p. 5, §4.2 · p. 5

4.3. Comparison with State-of-the-art

• R@1 improves by 2.9% on MSLS Val and 9.1% on MSLS Challenge; the model also reaches state of the art on Pitts30k and Tokyo24/7 and beats VG’s best models at comparable cost.The model is trained on MSLS and finetuned on Pitts30k for the urban datasets.Source: §4.3 · p. 6, Table 1 · p. 6, Table 2 · p. 6
• 47.17 ms/query on 4 GPUs gives a 20.2× speedup; MSLS Val storage is 4.79 GB, 189× below Patch-NetVLAD-p’s 908.3 GB.For the R-SF dataset with over 1M reference images, memory is 244.01 GB, or 122 GB with float16 and no observed performance drop; other reranking methods typically need over 4× more memory.Source: §4.3 · pp. 6–7, Table 3 · p. 6

4.4. Ablation Study

• R²Former reranking outperforms RANSAC using less than 22% of its inference time and also surpasses two recent landmark-retrieval reranking methods.The comparison uses the same global and local features from the default backbone; RRT is trained on that backbone, while CVNet uses its pretrained Google Landmark model.Source: §4.4 · p. 7, Table 4 · p. 7
• Separate training with global hard-negative mining achieves the best performance; end-to-end training offers a simpler trade-off, and all reranking strategies outperform prior state-of-the-art methods on R@5.The authors attribute the benefit to global hard-negative mining for reranker training.Source: §4.4 · p. 7, Table 5 · p. 7
• ViT-Small tokens are a good alternative to CNN local features, while ViT-Base improves global retrieval and RANSAC reranking but adds little overall improvement beyond R@10.ResNet50+GeM also achieves competitive retrieval and reranking results and significantly outperforms prior works, supporting the reranking module’s use with CNN and transformer backbones.Source: §4.4 · pp. 7–8, Table 6 · p. 7
• Transformer Block-2 and Correlation Value matter most; random feature selection moderately hurts performance, while removing other listed components causes slight drops.Removing positional embeddings or replacing top-5 pair selection with top-1 has negligible effect; the authors say the top-1 pair contains most reranking-relevant information.Source: Table 7 · p. 7, §4.4 · p. 8

4.5. Visualization

• In a case where global retrieval fails on the top three results, R²Former finds more correct references in the top three after reranking.Its highly attended patch pairs are mostly correct matches, though they may lack RANSAC’s tight geometric consistency and provide task-relevant reranking information.Source: §4.5 · p. 8, Figure 4 · p. 8

• The framework significantly outperforms previous methods on major datasets while using much less inference time and memory.Source: §5 · p. 8
• The proposed framework unifies place-recognition retrieval and reranking using only transformers.Source: §5 · p. 8
• Vision-transformer tokens are comparable to or better than CNN local features for reranking.Source: §5 · p. 8
• The generic reranking module can be adopted on CNN or transformer backbones.Source: §5 · p. 8
• The method can provide possible matched local pairs like RANSAC, and future work could improve it with geometric modeling.Source: §5 · p. 8