Source-linked AI summary
Deep Visual Geo-localization Benchmark
Gabriele Berton, Riccardo Mereu, Gabriele Trivigno, Carlo Masone, Gabriela Csurka, Torsten Sattler, Barbara Caputo
TL;DR
Visual geo-localization benchmarks have often emphasized recall or compared methods with differing setups, while giving limited attention to system requirements and individual component effects. This paper introduces a modular benchmark for controlled evaluation of VG architectures and pipeline choices. Experiments identify practical trade-offs among architectures, mining, resolution, performance, and resource use.
Problem
VG comparisons often focus on recall alone or use differing setups, limiting assessment of system requirements and the effects of individual component changes.
Source: §1 · p. 1Method
Results
ResNet-50 provides a good accuracy, FLOPs, and model-size trade-off, while partial negative mining and reduced resolution can reduce computation without significant performance loss or can yield gains.
Source: §1 · p. 2Takeaways & Limitations
The experiments offer criteria for choosing backbones, aggregation, and negative mining according to use case and resource requirements.
Source: §1 · p. 2, §5 · p. 8The framework focuses on outdoor urban, single-image VG and does not analyze viewpoint or luminosity invariance; some recent methods and losses are not yet implemented or compared.
Source: §5 · p. 8
Abstract
from arXiv · showhide
In this paper, we propose a new open-source benchmarking framework for Visual Geo-localization (VG) that allows to build, train, and test a wide range of commonly used architectures, with the flexibility to change individual components of a geo-localization pipeline. The purpose of this framework is twofold: i) gaining insights into how different components and design choices in a VG pipeline impact the final results, both in terms of performance (recall@N metric) and system requirements (such as execution time and memory consumption); ii) establish a systematic evaluation protocol for comparing different methods. Using the proposed framework, we perform a large suite of experiments which provide criteria for choosing backbone, aggregation and negative mining depending on the use-case and requirements. We also assess the impact of engineering techniques like pre/post-processing, data augmentation and image resizing, showing that better performance can be obtained through somewhat simple procedures: for example, downscaling the images' resolution to 80% can lead to similar results with a 36% savings in extraction time and dataset storage requirement. Code and trained models are available at https://deep-vg-bench.herokuapp.com/.
1. Introduction
Visual geo-localization research lacked a standardized way to compare methods while accounting for system requirements as well as recall. The proposed open-source benchmark enables controlled component comparisons and reports practical trade-offs across configurations.
1. Introduction
- Table 1 illustrates recall@1 for ResNet18 with NetVLAD trained on Pitts30k and tested on Tokyo24/7 under pipeline changes.The caption describes the example's setup but does not state the table's result values.Source: Table 1 · p. 1
- Prior comparisons often optimize recall alone or compare off-the-shelf methods with differing setups, making system requirements and the effect of individual algorithmic changes difficult to assess.A 5% accuracy drop may be acceptable if descriptor size falls by 90%, reducing memory needs and enabling greater scalability.Source: §1 · p. 1
- The benchmark provides an open-source framework for building, training, and testing common VG architectures while changing individual pipeline components.It is intended to measure how components affect final results.Source: §1 · p. 1
- The framework computes parameters, FLOPs, and descriptor dimensionality on the fly, alongside performance information.Source: §1 · p. 2
- ResNet-50 offers a good accuracy, FLOPs, and model-size trade-off; Transformers can outperform CNNs when trained on larger datasets, while partial negative mining and reduced resolution can cut computation without significant performance loss.These choices sometimes yield performance gains.Source: §1 · p. 2
2. Related Work
Visual geo-localization overlaps with visual localization and landmark retrieval but has a distinct goal and evaluation scope. The benchmark compares retrieval architectures and system requirements under matched conditions, addressing gaps in prior benchmarks.
2. Related Work
- VG estimates a query image’s geographic location, VL estimates its camera pose in a known scene, and landmark retrieval identifies images of the same landmark regardless of visual overlap.VG often uses retrieval, with the top retrieved image’s GPS tag supplying the location estimate.Source: §2 · p. 2
- VG representations include CNN features aggregated by methods such as NetVLAD, GeM, and R-MAC, as well as attention-based and Transformer approaches.NetVLAD’s high-dimensional descriptors can impose steep memory requirements; Transformers had been used in image retrieval but not yet in VG, according to the passage.Source: §2 · p. 2
- The benchmark examines how backbones, pooling, augmentation, and engineering choices affect geo-localization performance, memory, and computation.Source: §2 · p. 2
- Unlike VPR-Bench [85], which compares off-the-shelf models for practical use, this benchmark holds other factors constant to measure individual algorithmic changes.It also reports hardware-agnostic measures including FLOPs, model size, training complexity, and storage requirements.Source: §2 · p. 2
3. Methodology
The methodology treats visual geo-localization as a modular retrieval pipeline and tests how its components affect performance and resource demands. Experiments use heterogeneous datasets and a repeated recall-based protocol with specified training and mining practices.
3.1. Visual Geo-localization System
- The benchmark models VG as offline database and online query descriptor extraction followed by nearest-neighbor search, with configurable architectures, mining, image sizes, and augmentation.Its modular framework supports reproducing CNN-based methods and developing Transformer-based models.Source: §3.1 · p. 3
- The codebase provides dataset download and formatting scripts plus training and testing scripts to support consistent, reproducible experiments.It reproduces architectures and training protocols used in prior work.Source: §3.1 · p. 3
3.2. Datasets
- Six heterogeneous datasets cover varied scales, image variability, and camera types; Pitts30k and MSLS provide smaller and larger training sets, and four other datasets test inter-dataset robustness.The four test datasets are Tokyo 24/7, Revisited San Francisco, Eynsham, and St Lucia; MSLS validation recalls are reported because its test set lacks labels.Source: §3.2 · p. 3
3.3. Benchmark Protocol
- Results use recall@N, mostly R@1 with a 25-meter threshold, and average three experiment repetitions; training uses triplets and full mining on Pitts30k versus partial mining on MSLS.An epoch is a pass over 5,000 queries; training stops when validation recall@5 fails to improve for three epochs.Source: §3.3 · pp. 3–4
4. Results
The experiments examine architecture, training choices, image resolution, and search efficiency, showing that VG results involve trade-offs between recall and system resources.
4.1. CNN Backbones
- ResNet-50 achieves recalls on par with ResNet-101 using less than half its FLOPs and model size, while ResNet-18 is the lightest, fastest option.Training-data choice also matters: the same network yields a 30% gap when tested on St. Lucia after training on Pitts30k versus MSLS.Source: §4.1 · p. 4
4.2. Aggregation and Descriptor Dimensionality
- CRN performs best overall when trained on Pitts30k, but its advantage is reduced with larger MSLS; it also requires nearly twice NetVLAD’s training time and three times its hyperparameters.Training may not converge depending on modulation-layer initialization.Source: §4.2 · pp. 4–5
4.3. Visual Transformers
- Visual Transformers are viable CNN alternatives: CCT often outperforms ResNet-50 despite roughly half its computational cost, while ViT is competitive with larger NetVLAD descriptors at higher computational cost.The reported architectures lack an all-around best configuration and require use-case-specific tuning of where to truncate or freeze the network.Source: Table 5 · p. 5, §4.3 · pp. 5–6
4.4. Negative Mining
- Partial negative mining generally balances cost and accuracy: full mining gains about 1% on average, while random sampling loses 5% on Pitts30k and 10% or more on MSLS.Full mining is not feasible in a reasonable time on large datasets such as MSLS.Source: §4.4 · p. 6
4.5. Data Augmentation
- Augmentation effects vary by test dataset: contrast3 up to 2 improves recall@1 by more than 3% on MSLS and 5% on Tokyo 24/7 and St Lucia, with less than a 1% drop on Pitts30k and Eynsham.Random horizontal flipping at 50% probability and random resized cropping to as small as 50% are notable exceptions to the generally inconsistent improvements.Source: §4.5 · p. 6, Figure 2 · p. 6
4.6. Resize
- 40% resolution gives the best R@1 on St Lucia, while 60% is generally a good compromise and higher resolution is often unnecessary or detrimental.At 40% resolution, images are 192x256 and FLOPs fall to 16% of full-resolution levels; NetVLAD handles higher resolutions better than GeM on average.Source: Figure 3 · p. 7, §4.6 · p. 7
4.7. Nearest Neighbor Search and Inference Time
- 98.5% reductions in matching time and memory with IVFPQ accompany accuracy falling from 45.4% to 41.4%; MultiIndex saves 80% matching time with a 0.9% recall loss.As databases grow, matching time can dominate inference; the paper argues that exact search is rarely justified for practical applications.Source: Figure 4 · p. 7, §4.7 · pp. 7–8
- Matching time grows linearly with database size and descriptor dimensionality; kNN is the bottleneck when their product exceeds 200M.The figure contrasts matching time with ResNet-101 + GeM image extraction time.Source: Figure 4 · p. 7, §4.7 · p. 7
5. Discussions and Findings
The framework makes VG pipelines modular so their design choices can be evaluated for both performance and resource requirements. Its findings cover architecture, training, image processing, and search, while noting limits to the framework’s scope.
Architecture
- CCT with NetVLAD achieves the best overall results; CCT costs about as much as ResNet-18 while outperforming ResNet-50.Source: §5 · p. 8
Negative mining
- Partial negative mining can match or sometimes exceed full mining performance at a fraction of the computational cost.Source: §5 · p. 8
Training dataset
- Large-scale training data spanning diverse conditions and cities significantly improves results; comparisons across models trained on different datasets are unfair and should be avoided if possible.Source: §5 · p. 8
Image size and data augmentation
- Scaling images to 60% of full resolution reduces FLOPs while yielding comparable, and sometimes better, results on average.Horizontal flipping and resized cropping give a slight but consistent boost; color jittering’s effectiveness depends strongly on the dataset.Source: §5 · p. 8
Inference time and kNN search
- Neighbor-search choice can substantially affect time and memory with little performance impact, and advanced methods may narrow resource gaps between descriptor sizes.Source: §5 · p. 8
Final remarks
- Carefully optimizing the same simple architecture can yield results comparable to much more complex, unoptimized methods.Source: §5 · p. 8
Limitations
- The framework focuses on outdoor urban, single-image VG and does not analyze viewpoint or luminosity invariance.Source: §5 · p. 8
A. Datasets
The datasets vary in geography, viewpoints, cameras, and acquisition conditions, making generalization between them difficult. They also differ in how database and query images are collected and split.
Pitts30k
- Pitts30k uses Google Street View imagery from Pittsburgh, with database and query images collected two years apart and no noticeable weather variation.Source: A. Datasets · p. 12
MSLS
- MSLS spans cities across six continents and diverse domains, cameras, and seasons; its test ground truths are unreleased, so validation recalls are reported.Only Pitts30k and MSLS provide training sets with temporal variability, which is necessary for training a VG model.Source: A. Datasets · p. 12
Tokyo 24/7 and San Francisco
- Tokyo 24/7 pairs a large Google Street View database with fewer phone queries split across day, sunset, and night; San Francisco likewise pairs a large car-camera database with far fewer phone queries.Source: A. Datasets · p. 12
Eynsham and St Lucia
- Eynsham uses grayscale car-camera imagery from two loops, with the first as database and the second as queries; St Lucia uses the first and last of nine drives.St Lucia frames are selected every 5 meters because the video-derived imagery is dense.Source: A. Datasets · p. 12
Dataset comparison
- Examples across datasets illustrate variation in views, environments, and acquisition conditions; the paper summarizes their image counts, covered areas, perimeters, and geographic image density.Source: A. Datasets · p. 12
B. Software
The software offers a modular, reproducible framework for training, testing, and comparing visual geo-localization components. Its experiments compare backbone, aggregation, and negative-mining choices, while the appendix includes data-augmentation plots.
B. Software
- The framework is designed as a modular, reproducible playground for training, testing, and fairly comparing VG components across datasets of different scales.Its three modules cover VG model training and evaluation, dataset downloading and formatting, and backbone pretraining.Source: B. Software · p. 14
C.1. CNN Backbones
- Conv4 x cropping gives better average results than conv5 x cropping while making ResNet-18 and ResNet-50 somewhat lighter.The comparison covers both ResNet backbones.Source: §C.1 · p. 15
C.2. Aggregation and Descriptors Dimensionality
- GeM pooling generally outperforms SPOC, MAC, R-MAC, and RRM across the reported backbone and descriptor configurations.Table 10 groups aggregation results by backbone and feature dimension.Source: §C.2 · p. 15
C.3. Visual Transformers: full table
- On Pitts30k, Transformer backbones generalize better than ResNet-18 and ResNet-50, while CLS-token aggregation performs worse than SeqPool, GeM, or NetVLAD.The authors suggest CLS may be less robust on small-scale training data because it does not use the whole token set.Source: §C.3 · p. 15
C.4. Negative Mining
- On Pitts30k, random negatives lose only 5% recall@1 versus partial mining, while MSLS favors partial mining and full mining was stopped after five days.The authors relate the difference to MSLS's greater domain variety and describe full database mining as impractical at large scale.Source: §C.4 · p. 15
C.5. Data Augmentation
- Figure 7 reproduces the data-augmentation results plots from the main paper at a larger, more readable size.The supplied passage does not state the plotted outcomes.Source: §C.5 · p. 15
C.6. Query pre/post-processing and Predictions Refinement
The experiments compare query resizing, cropping, and prediction-refinement methods for differing query and database resolutions. For mixed-resolution datasets, crop-based methods can improve results while enabling batched inference.
C.6. Query pre/post-processing and Predictions Refinement
- The methods address production queries whose resolutions may differ from database images; single-query inference can work well but is slower than batched processing.Source: §C.6 · pp. 15–16
- Hard Resize, Single Query, and Central Crop resize or crop queries, while Five Crops generates five square crops based on the database images’ shortest side.Single Query preserves aspect ratio and does not pad images; varying query resolutions cannot be stacked in a batch.Source: §C.6 · pp. 18–19
- Mean averages the five crop descriptors, Nearest Crop selects a prediction using crop-to-descriptor distance, and Majority Voting aggregates distances from each crop’s top 20 predictions.Source: §C.6 · p. 19
- On Tokyo 24/7 and R-SF, Nearest Crop and Majority Voting yield better average results; Nearest Crop gives the best R@1 and permits batch parallelization.When query and database image sizes match, Hard Resize, Single Query, and Central Crop produce identical results.Source: §C.6 · p. 19
C.7. Nearest Neighbor Search and Inference Time
The appendix evaluates faster nearest-neighbor indexing and additional experiments on training data, database distractors, fine-tuning, and evaluation metrics. Approximate indexes trade matching time and memory against recall, with the best choice depending on memory needs.
C.7. Nearest Neighbor Search and Inference Time
- IVF reduces matching time by roughly 20 times with less than a 2% average recall@1 drop; HNSW and MultiIndex are slightly slower but achieve higher recall.Figure 8 plots matching time against recall@1 and marks RAM requirements for the indexing methods.Source: §C.7 · p. 19, Figure 8 · p. 20
- IVFPQ cuts memory footprint by a factor of 64; on R-SF, matching time falls 98.5% while accuracy drops from 45.4% to 41.4%.The authors identify IVFPQ as Pareto optimal for recall@1 and matching time when memory efficiency matters.Source: Figure 8 · p. 20, §C.7 · p. 20
D. Additional Experiments
- The additional experiments examine landmark-retrieval pretraining, landmark images as database distractors, VG fine-tuning, and the relationship between evaluation metrics and results.Source: D. Additional Experiments · p. 20
D.1. The role of the training dataset
The experiments compare landmark-retrieval pretrained models with models trained on VG datasets. Their findings link generalization to training-data variety, viewpoint similarity, and model size relative to dataset size.
D.1. The role of the training dataset
- Large-scale landmark-retrieval models, especially GLDv1-trained models, offer decent off-the-shelf performance on many VG datasets.Table 14 reports results for models trained on large-scale landmark-retrieval datasets.Source: §D.1 · p. 21, Table 14 · p. 21
- MSLS-trained models generalize better than Pitts30k-trained models on all datasets except R-SF and Tokyo 24/7, whose images include 360° views unlike MSLS training images.Source: §D.1 · p. 21
- The authors associate Pitts30k models’ poor generalization with overfitting from a large FC layer, while NetVLAD + PCA improves generalization by reducing parameters.Source: §D.1 · p. 21
- The authors recommend training on viewpoint-similar data when known, otherwise using a more general training set, and matching model parameter count to dataset size.Source: §D.1 · p. 21
D.2. Scaling datasets with distractors
The benchmark scales databases with distractors up to 8 million images; recall declines as database size grows, and the authors conclude the task remains unsolved.
D.2. Scaling datasets with distractors
- The San Francisco database contains 1 million images but covers only 9% of the city.Source: §D.2 · p. 21
- Recall steadily decreases as the database grows, showing that visual geo-localization remains far from solved.The experiment scales the database up to 8 million images.Source: Figure 9 · p. 21, §D.2 · p. 21
- The distractor databases grow from Tokyo 24/7 queries and positives to 8 million images, adding San Francisco, Google Landmark v2, Places 365, and MSLS training images.Source: §D.2 · p. 21
D.3. Testing on an ensemble of datasets
The ensemble benchmark tests queries from six datasets against their respective databases or a merged database. Matching against the merged database reduces recall relative to separate-dataset results.
D.3. Testing on an ensemble of datasets
- The preceding large-scale experiment used 8 million distractors with queries from a single dataset; the ensemble instead tests queries across different distributions and geographical areas.Source: §D.3 · pp. 21–22
- Multi DB results show lower recall than averaging Single DB results; models trained on MSLS perform better overall, and GeM has a larger percentile R@1 drop than NetVLAD.Single DB matches each query only to its respective database; Multi DB matches against all six merged databases.Source: Table 15 · p. 22, §D.3 · p. 22
- The ensemble combines queries and databases from Pitts30k, MSLS, Tokyo 24/7, R-SF, Eynsham, and St Lucia.The experiment compares matching to each query’s respective database with matching to all six databases merged.Source: §D.3 · p. 22
D.4. Pretraining the backbone on other datasets
Across these sections, the paper evaluates alternative backbone pretraining and metric choices, including heading, distance thresholds, and recall cutoffs. It also discusses ethical boundaries of using the benchmark.
D.5.1 Taking heading/yaw/compass into account
- 99.8% of correct predictions are within 90° of the query heading, 98% within 60°, 90% within 45°, 57% within 30°, and about 14% within 10°; results are stable across models.The experiment uses R-SF heading labels and a fixed 25-meter distance threshold; the authors say post-processing should be considered when accurate pose estimation is needed.Source: Figure 10 · p. 22, §D.5.1 · pp. 22–23
D.5. Metrics
- The metric experiments use Pitts30k-trained models and change only the test-time metric, varying heading, positive-distance threshold, and recall N.Source: §D.5 · p. 22
D.4. Pretraining the backbone on other datasets
- ImageNet pretraining is advantageous in most cases; alternatives usually yield close scores, but reduce performance on R-SF and Tokyo 24/7 on average.Even where GLDv2 or Places365 scores highest, the gap is negligible, and ImageNet backbones are more readily available.Source: §D.4 · p. 22, Table 16 · p. 23
D.5.2 Changing the positives’ threshold distance
- Recall@1 rises rapidly as the positive-distance threshold grows, then slows between 10 and 25 meters; it remains below 90% for every dataset even at 50 meters.A 25-meter threshold gives close to 100% query coverage across the considered datasets, while random chance yields recalls close to zero.Source: §D.5.2 · p. 23, Figure 11 · p. 24
D.5.3 Other values of recall
- If a location is absent from the top five predictions, its chance of appearing within the top 20 averages 35%; the plots also suggest top-20/30 reranking may perform similarly at lower cost.For MSLS and San Francisco, the top-five-to-top-20 chances are 19% and 26%, respectively; reranking top 100 cannot exceed the initial recall@100.Source: §D.5.3 · pp. 23, 25, Figure 12 · p. 24
D.5.4 The relation between threshold distances, values of the recall and heading distance
- The interaction plots vary recall N, distance threshold, and heading difference for ResNet-50 + GeM trained on Pitts30k, using R-SF because it has heading labels.Source: Figure 13 · p. 25, §D.5.4 · p. 25
E. Ethical implications
- The benchmark cannot guarantee how evaluated algorithms will be used; its data come from public, place-focused datasets considered safe, and researchers are urged to avoid misuse.Source: E. Ethical implications · p. 25