Source-linked AI summary
MegaLoc: One Retrieval to Place Them All
Gabriele Berton, Carlo Masone
TL;DR
Existing retrieval systems are commonly specialized for Landmark Retrieval, Visual Place Recognition, or Visual Localization, despite retrieval supporting all three and related 3D vision tasks. MegaLoc combines datasets, samplers, and training practices from these areas into one model. It performs strongly across diverse tasks, including state-of-the-art results on many VPR datasets and LaMAR visual localization, while remaining suboptimal in several documented settings.
Problem
Image retrieval supports Landmark Retrieval, Visual Place Recognition, Visual Localization, and 3D vision, but differing definitions and requirements have led to task-specific solutions.
Method
MegaLoc trains a single retrieval model by combining diverse datasets, task-specific sampling methods, and training practices from Landmark Retrieval, Visual Place Recognition, and Visual Localization.
Results
MegaLoc achieves strong results across diverse tasks, including performance comparable to leading models on most evaluated VPR datasets and a reported state-of-the-art result on LaMAR visual localization.
Takeaways & Limitations
A single retrieval model can work across Visual Place Recognition, Visual Localization, Landmark Retrieval, and retrieval needs in 3D reconstruction.
Takeaways & Limitations
MegaLoc is outperformed by CliqueMining on MSLS, may be suboptimal in unusual natural environments, and is less suitable for embedded systems than lightweight alternatives.
Abstract
from arXiv · showhide
Retrieving images from the same location as a given query is an important component of multiple computer vision tasks, like Visual Place Recognition, Landmark Retrieval, Visual Localization, 3D reconstruction, and SLAM. However, existing solutions are built to specifically work for one of these tasks, and are known to fail when the requirements slightly change or when they meet out-of-distribution data. In this paper we combine a variety of existing methods, training techniques, and datasets to train a retrieval model, called MegaLoc, that is performant on multiple tasks. We find that MegaLoc (1) achieves state of the art on a large number of Visual Place Recognition datasets, (2) impressive results on common Landmark Retrieval datasets, and (3) sets a new state of the art for Visual Localization on the LaMAR datasets, where we only changed the retrieval method to the existing localization pipeline. The code for MegaLoc is available at https://github.com/gmberton/MegaLoc
1. Introduction
Image retrieval is used across Landmark Retrieval, Visual Place Recognition, Visual Localization, and 3D vision, but each task defines “same place” differently. MegaLoc addresses this fragmentation by training one model from combined task-specific lessons, datasets, and training practices.
- Landmark Retrieval matches images depicting the same landmark, Visual Place Recognition uses a 25-meter pose-distance threshold, and Visual Localization requires poses to be as close as possible.
- Task-specific definitions and requirements have produced ad-hoc retrieval solutions that are rarely evaluated across more than one task.
- MegaLoc combines lessons from Landmark Retrieval, Visual Place Recognition, and Visual Localization without proposing technical novelty, using samplers, datasets, and general training techniques.
- 3D reconstruction may require nearby-image retrieval for small scenes or images hundreds of meters apart for large landmarks, exposing different strengths of VPR and LR models.
- The paper demonstrates that diverse data sources and best practices can produce one retrieval model that works across Visual Place Recognition, Visual Localization, and Landmark Retrieval.
2. Method
MegaLoc trains one retrieval model by fusing five datasets covering indoor and outdoor imagery and multiple localization tasks. Training uses dataset-specific sampling or mining strategies and multi-similarity loss over six sub-batches.
- MegaLoc fuses GSV-Cities, MSLS, MegaScenes, ScanNet, and SF-XL, extracting six sub-batches per iteration and applying multi-similarity loss to each.
- SF-XL contains 41M images spanning San Francisco across 12 years, and EigenPlaces sampling forms classes with diverse perspectives while separating different places visually.
- GSV-Cities provides 530k images in 62k classes across 40 cities, which are directly fed to multi-similarity loss because its classes are already non-overlapping.
- CliqueMining sampling for MSLS selects visually similar but geographically different places as hard negatives to teach the model to disambiguate similar-looking locations.
3. Experiments
MegaLoc is evaluated across Visual Place Recognition, Visual Localization, and Landmark Retrieval, with strong results across these settings and identified failure boundaries.
- Evaluation scope: MegaLoc is tested on Visual Place Recognition, Visual Localization, and Landmark Retrieval tasks.The evaluation covers retrieval within 25 meters, retrieval within a localization pipeline, and same-landmark retrieval.
- Visual Place Recognition: MegaLoc vastly outperforms every other model on the indoor-only Baidu Visual Place Recognition dataset.Other high-performing VPR models are comparable to MegaLoc on most evaluated datasets.
- Visual Localization: MegaLoc consistently achieves high results across LaMAR datasets, whereas competing models perform well only on some datasets.The comparison uses both HoloLens and Phone queries across LaMAR locations, with recalls at (1°, 10cm) and (5°, 1m).
- Landmark Retrieval: MegaLoc creates a large gap over previous Visual Place Recognition models on Landmark Retrieval datasets.The evaluation uses Revisited Paris 6k and Revisited Oxford 5k.
- Failure Cases: Database images facing only one direction can limit retrieval when queries face the opposite direction, as commonly occurs in MSLS.The paper notes that collecting images in multiple directions can address this real-world coverage issue.
- Failure Cases: Four failure categories include very difficult cases, difficult cases, incorrect GPS labels, and predictions just outside the 25-meter threshold.The figure shows five examples per category, each containing a query and MegaLoc’s top-two predictions.
4. Conclusion and limitations
MegaLoc performs well across diverse retrieval tasks and domains, but it is not universally optimal. The paper identifies specific settings where competing or lighter models are preferable.
- MegaLoc achieves good results across a variety of diverse tasks and domains, though some datasets still leave room for improvement.The paper attributes remaining gaps to arguably unsolvable cases, wrong labels, and relatively few cases solvable by better models.
- CliqueMining outperforms MegaLoc on MSLS, which consists almost entirely of forward-facing street images.This setting differs from imagery facing sideways toward the street side.
- MegaLoc may be suboptimal in unusual natural environments such as forests or caves, where AnyLoc has performed well.
- Embedded systems may favor ResNet-18 versions of CosPlace, which use 11M parameters instead of MegaLoc’s 228M.