Source-linked AI summary
Optimal Transport Aggregation for Visual Place Recognition
Sergio Izquierdo, Javier Civera
TL;DR
Visual place recognition requires robust image descriptors for matching query images against large geo-localized databases under changing conditions. The paper introduces DINOv2 SALAD, which combines fine-tuned DINOv2 features with Sinkhorn optimal-transport aggregation and a dustbin for uninformative features. It reports state-of-the-art single-stage VPR results, including 75.0% Recall@1 on MSLS Challenge and 76.0% on Nordland, while outperforming two-stage baselines with re-ranking.
Problem
VPR needs discriminative image descriptors that remain robust under illumination, structural, temporal, weather, and seasonal changes.
Method
DINOv2 SALAD fine-tunes DINOv2 and reformulates local feature assignment as Sinkhorn optimal transport with a dustbin for uninformative features.
Results
75.0% Recall@1 in MSLS Challenge and 76.0% in Nordland are reported for the single-stage DINOv2 SALAD pipeline.
Takeaways & Limitations
The single-stage model surpasses single-stage baselines and two-stage re-ranking methods while requiring less than 3 ms inference per image.
Takeaways & Limitations
DINOv2 processes more slowly than ResNet backbones, and SALAD is less suited to coarser features despite being a general aggregation module.
Abstract
from arXiv · showhide
The task of Visual Place Recognition (VPR) aims to match a query image against references from an extensive database of images from different places, relying solely on visual cues. State-of-the-art pipelines focus on the aggregation of features extracted from a deep backbone, in order to form a global descriptor for each image. In this context, we introduce SALAD (Sinkhorn Algorithm for Locally Aggregated Descriptors), which reformulates NetVLAD's soft-assignment of local features to clusters as an optimal transport problem. In SALAD, we consider both feature-to-cluster and cluster-to-feature relations and we also introduce a 'dustbin' cluster, designed to selectively discard features deemed non-informative, enhancing the overall descriptor quality. Additionally, we leverage and fine-tune DINOv2 as a backbone, which provides enhanced description power for the local features, and dramatically reduces the required training time. As a result, our single-stage method not only surpasses single-stage baselines in public VPR datasets, but also surpasses two-stage methods that add a re-ranking with significantly higher cost. Code and models are available at https://github.com/serizba/salad.
1. Introduction
Visual place recognition retrieves geographically matching images despite substantial appearance and structural changes. The paper introduces a single-stage pipeline combining fine-tuned DINOv2 features with SALAD aggregation to achieve state-of-the-art benchmark results.
- Task: VPR retrieves an ordered list of top-k database images matching a query image.The references are geo-localized images from pre-existing databases.
- Challenge: Robust, discriminative image descriptors are needed across illumination, structural, temporal, weather, and seasonal changes.Recent VPR research therefore emphasizes feature-extraction backbones and feature-aggregation methods.
- Contributions: DINOv2 is fine-tuned for VPR rather than used in its pre-trained form, unlike the cited AnyLoc approach.The authors state that task-specific fine-tuning substantially improves performance.
- Contributions: SALAD reformulates feature-to-cluster assignment as optimal transport and adds a dustbin mechanism for discarding uninformative features.The method is designed to distribute local features more effectively into global descriptor bins.
- Results: 75.0% Recall@1 in MSLS Challenge and 76.0% in Nordland are reported for the single-stage DINOv2 SALAD pipeline.The model trains in less than one hour and runs at less than 3 ms per image.
2. Related Work
Related VPR work spans handcrafted and learned descriptors, feature aggregation, and costly two-stage re-ranking. This paper positions optimal transport as a new perspective for local feature aggregation within a DINOv2-based retrieval pipeline.
- Scope: VPR research has explored new losses, image sequences, extreme viewpoint changes, and text features, alongside feature extraction and aggregation.The paper focuses its related-work discussion on the latter two areas.
- Feature extraction: Task-specific training or fine-tuning with contrastive or triplet losses improves over recognition-pretrained features without fine-tuning.This follows earlier use of handcrafted local features and global descriptors.
- Aggregation: NetVLAD, GeM, region aggregation, and MixVPR represent prominent learned or pooled alternatives for constructing VPR descriptors.NetVLAD jointly learns convolutional features and cluster centroids, while MixVPR combines deep features with an MLP layer.
- Re-ranking: Two-stage VPR methods re-rank top retrieved candidates using unaggregated local features, adding considerable computational overhead.The re-ranking can assess geometric consistency or predict similarity.
- Optimal transport: Optimal transport has been used in image retrieval, image matching, and feature matching, but this paper is the first cited work to apply it to local feature aggregation.A prior work used optimal transport at the re-ranking stage instead.
3. Method
DINOv2 SALAD extracts local and global representations, computes feature-to-cluster and dustbin assignments with Sinkhorn optimal transport, and aggregates reduced features into a normalized descriptor. Its design removes some NetVLAD priors, discards uninformative features, and incorporates global scene information.
- 3.1. Local Feature Extraction: DINOv2 supplies patch tokens and a global token, with VPR gains from selectively fine-tuning the encoder’s final blocks.The backbone divides images into 14 × 14 patches and produces local tokens plus a class-information token.
- 3.2. Assignment: SALAD learns each feature’s score row from scratch using two fully connected layers instead of NetVLAD’s k-means-derived initialization.The score projection uses weights, biases, and a nonlinear activation.
- 3.2. Assignment: SALAD augments the score matrix with a learnable dustbin relation so non-informative features can be assigned away from descriptor clusters.The dustbin relation is represented by a single learnable parameter z.
- 3.2. Assignment: Sinkhorn optimal transport jointly enforces feature-to-cluster and cluster-to-feature mass relations when producing the assignment.Feature mass is distributed among the clusters and dustbin, after which the dustbin column is removed.
- 3.3. Aggregation: SALAD reduces token dimensionality before aggregating assigned features directly by summation without subtracting cluster centroids.This produces a matrix-valued VLAD representation from the assignment matrix.
- 3.3. Aggregation: A transformed DINOv2 global token is concatenated with the flattened aggregated features and L2-normalized to form the final descriptor.The global token supplies scene information not easily incorporated into local features.
4. Experiments
Experiments evaluate DINOv2 SALAD against single-stage and re-ranking baselines, then analyze backbone, aggregation, hyperparameter, component, and feature-selection effects. The method achieves strong benchmark recall with efficient training and inference, while ablations support the roles of fine-tuned DINOv2, SALAD, and its components.
- Benchmark results: DINOv2 SALAD outperforms previous methods on all datasets and metrics, with the largest recall gains on MSLS Challenge and NordLand.Improvements over the second-best method are +7.6% and +11.7% for R@1 and R@5 on MSLS Challenge, and +17.6% and +14.6% on NordLand.
- Benchmark results: DINOv2 SALAD significantly outperforms leading two-stage re-ranking methods on all benchmarks while being orders of magnitude faster and more memory-efficient.The comparison uses a single retrieval stage for DINOv2 SALAD and re-ranking methods including R2Former, TransVPR, and Patch-NetVLAD.
- Ablation studies: End-to-end fine-tuning makes DINOv2-based aggregators outperform their baselines, including AnyLoc, which uses pretrained DINOv2 without task-specific fine-tuning.The experiments integrate DINOv2 with several aggregation modules and report improvements over ResNet NetVLAD and AnyLoc.
- Ablation studies: At comparable descriptor sizes, DINOv2 SALAD clearly outperforms DINOv2 NetVLAD, while SALAD with ResNet or ConvNext is worse than with DINOv2.DINOv2 NetVLAD can be competitive with a descriptor almost three times as large, but its generalization is limited on NordLand.
- Ablation studies: The global token and dustbin both contribute to SALAD performance, while fine-tuning two or four backbone blocks gives the best results without significant computation overhead.The Sinkhorn assignment is only slightly better than dual-softmax in the reported component comparison, although the authors select Sinkhorn for theoretical soundness.
5. Conclusions and Limitations
DINOv2 SALAD combines a fine-tuned DINOv2 backbone with the SALAD aggregation module and outperforms previous baselines. Its single-stage design also offers fast training and inference, while performance has limitations tied to backbone choice and processing speed.
- DINOv2 SALAD combines a fine-tuned DINOv2 backbone for feature extraction with SALAD for feature aggregation.
- DINOv2 slows processing compared with ResNet-based methods, and SALAD is less suited to coarser backbone features.SALAD excels with DINOv2's high spatial resolution features; more sophisticated optimal-transport constraints remain future work.