PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Visual Geo-localization for Large-Scale Applications

Gabriele Berton, Carlo Masone, Barbara Caputo

arXiv:2204.02287v2cs.CV

TL;DR

Visual geo-localization methods must support city-scale databases, but existing datasets and contrastive-learning training are poorly suited to that setting. The paper introduces the city-wide SF-XL dataset and CosPlace, a classification-based training procedure that avoids mining. CosPlace achieves strong results across datasets while using substantially less training memory and smaller descriptors.

  • Problem

    Existing VG datasets do not represent large, dense environments, while contrastive-learning training depends on expensive negative mining that becomes prohibitive for massive databases.

  • Method

    CosPlace trains retrieval descriptors through a classification proxy over geographically partitioned groups, then uses those descriptors for classic image retrieval without mining.

  • Results

    CosPlace achieves the best average results across five datasets, outperforming the second-best method by 8.5% of R@1, while requiring roughly 80% less training GPU memory and using 8x smaller embeddings.

  • Takeaways & Limitations

    SF-XL and CosPlace support evaluation and training for large-scale VG while retaining retrieval-based localization and generalizing to other domains.

  • Takeaways & Limitations

    CosPlace requires heading labels and many training images, so it cannot train on datasets without heading labels and is not suited to small training datasets.

Abstract

from arXiv · show

Visual Geo-localization (VG) is the task of estimating the position where a given photo was taken by comparing it with a large database of images of known locations. To investigate how existing techniques would perform on a real-world city-wide VG application, we build San Francisco eXtra Large, a new dataset covering a whole city and providing a wide range of challenging cases, with a size 30x bigger than the previous largest dataset for visual geo-localization. We find that current methods fail to scale to such large datasets, therefore we design a new highly scalable training technique, called CosPlace, which casts the training as a classification problem avoiding the expensive mining needed by the commonly used contrastive learning. We achieve state-of-the-art performance on a wide range of datasets and find that CosPlace is robust to heavy domain changes. Moreover, we show that, compared to the previous state-of-the-art, CosPlace requires roughly 80% less GPU memory at train time, and it achieves better results with 8x smaller descriptors, paving the way for city-wide real-world visual geo-localization. Dataset, code and trained models are available for research purposes at https://github.com/gmberton/CosPlace.

1. Introduction

Visual geo-localization estimates where photos were taken through image retrieval, but existing datasets and contrastive-learning training do not reflect or scale to city-wide applications. The paper introduces SF-XL and CosPlace to address these limitations.

  • Task: Visual geo-localization estimates a photo’s geographical location by retrieving similar images from a database of geo-tagged images.Neural networks project images into an embedding space whose descriptors support retrieval.
  • Motivation: Real-world applications such as autonomous driving and assistive devices require VG systems to operate over city- or metropolitan-scale databases.The paper argues that such databases should also support model training, not only inference.
  • Limitations: Current VG datasets are either geographically narrow or sparse, and geographically disjoint train–inference splits do not match likely real-world data use.The paper notes that using imagery from the target area for training would better reflect deployment and reduce collection costs.
  • Limitations: Contrastive-learning methods rely on mining negative examples across the training database, making training increasingly expensive and potentially prohibitive at massive scale.Lightweight mining reduces mining duration but remains slow as the database grows.
  • Contributions: SF-XL is a large-scale, dense dataset roughly 30x bigger than previous resources, with crowd-sourced multi-domain queries for a challenging problem.It is designed to address the lack of realistic city-wide evaluation settings.
  • Contributions: CosPlace uses classification as a proxy for training retrieval descriptors, avoids negative-example mining, and learns effectively from massive collections.The method is presented as a simple, scalable alternative to contrastive learning.
  • Results: 80% less GPU memory at train time and 8x smaller embeddings are reported for CosPlace relative to current state of the art.A simple model trained with CosPlace on SF-XL surpasses the state of the art while using smaller embeddings.

2. Related work

Prior VG work primarily treats localization as image retrieval with learned embeddings, while alternative classification formulations improve scalability but use coarse geographic classes. CosPlace combines retrieval’s fine tolerance with classification-based training without mining.

  • Retrieval: VG retrieval compares a query against geo-tagged database images and typically evaluates correctness within a predefined distance, often 25 meters.Learned embeddings are produced by a backbone and aggregation or pooling head, commonly NetVLAD.
  • Retrieval: Recent retrieval methods generally use contrastive learning, often triplet loss, whose negative-example mining causes poor training-time scalability.Mining can also involve cached or cropped negatives, but these variants retain scalability problems.
  • Classification: Classification-based VG partitions the geographic area into cells and groups images by cell, exploiting shared semantics among images from nearby regions.This formulation is intended to scale to larger geographic areas.
  • CosPlace: CosPlace trains with classification without mining, then extracts descriptors for classic retrieval at validation and test time.Its partitioning uses dense data so images in the same class visualize the same scene, unlike prior methods using cells up to hundreds of kilometers wide.

3. The San Francisco XL dataset

SF-XL is a city-wide, dense, temporally variable VG dataset designed for large environments and few-meter tolerance. It combines a massive StreetView database with whole-city training and diverse, cross-domain test queries.

  • Dataset motivation: SF-XL addresses the absence of datasets combining large geographic coverage, dense imagery, and few-meter localization tolerance.The paper describes it as the first city-wide, dense, and temporally variable dataset.
  • Database: The database contains 3.43M panoramas split into 41.2M crops, each labeled with GPS, heading, and other 6 DoF information.Imagery collected from 2009 to 2021 provides long-term temporal variation.
  • Database: The training set uses all 41.2M images across San Francisco rather than geographically disjoint database and query subsets.This design reflects applications that train on imagery from the target geographic area.
  • Evaluation: The test-time database uses 2.8M images from 2013, balancing feasible research-time testing with whole-area coverage.The selected set avoids making validation or hyperparameter tuning on the test set impractical.
  • Queries: SF-XL test set v1 contains 1,000 manually verified Flickr queries spanning viewpoint and illumination changes, including day and night images.The queries are intended to represent unseen domains relative to the StreetView database.
  • Queries: Test set v2 contains 598 San Francisco Landmark Dataset queries with 6 DoF coordinates, enabling large-scale pose estimation.The paper separately provides a validation set of 8k database images and 8k queries distributed across the city.

4. Method

CosPlace addresses the scalability limitations of classification and contrastive-learning approaches by partitioning geo-localization data into spatially and orientation-separated groups, then training sequentially over those groups.

  • Scalability limitations: Current state-of-the-art methods require O(n) feature caching during training and high-dimensional descriptors that make large-scale inference memory-intensive.A VGG-16 with NetVLAD produces 32k-dimensional vectors, requiring 1220GB for a 10M-image database.
  • 4.1. Splitting the dataset into classes: A naive classification scheme divides UTM-based square cells into heading-specific classes, but quantization can assign nearly identical images to different classes.Images only a few centimeters apart may cross cell boundaries despite depicting the same scene or building.
  • 4.1. Splitting the dataset into classes: CosPlace Groups contain nonadjacent classes, separating classes by at least M · (N − 1) meters or α · (L − 1) degrees within each group.N controls the minimum number of cells between same-group classes, while L controls the analogous orientation separation; there are N × N × L groups.
  • 4.2. Training the network: CosPlace applies Large Margin Cosine Loss sequentially to each group, using one fully connected layer per group and discarding those layers at validation and test.The model extracts descriptors for standard image retrieval rather than classifying queries, allowing use on unseen geographical areas.

5. Experiments

Experiments show that CosPlace combines competitive cross-dataset retrieval with substantially lower memory and descriptor costs, while its performance depends on large training sets and heading labels.

  • Comparison with other methods: CosPlace achieves the best average results, outperforming the second-best method by 8.5% of R@1 averaged over five datasets.
  • Comparison with other methods: CosPlace generalizes more robustly across datasets from different sources than competing methods.Other methods tend to perform well either on StreetView-sourced databases or on frontal-view imagery.
  • Computational efficiency: 80% less GPU memory is required at train time than current state of the art, using 7.5 GB on one GPU versus four 11GB GPUs for SFRS.
  • Computational efficiency: 512-D descriptors achieve state-of-the-art results, while inference kNN is 8 times faster than 4096-D SFRS and 64 times faster than 32k-D NetVLAD.The comparison uses the linear dependence of exhaustive kNN execution time on descriptor dimensionality.
  • Ablation: CosPlace Groups improve results because they avoid placing adjacent geographical classes in the same training group.
  • Ablation: CosPlace performs well with multiple backbones, with ResNets generally faster and less memory-intensive than VGG-16.

6. Conclusions

The paper studies large-scale visual geo-localization with SF-XL and proposes CosPlace to train efficiently on massive datasets. CosPlace surpasses state of the art with smaller descriptors, uses fewer training resources, and generalizes well, while having limitations on small datasets and datasets without orientation labels.

  • 6. Conclusions: CosPlace enables efficient training on large quantities of data by using classification as a proxy instead of contrastive learning with mining.The method extracts descriptors for retrieval at inference time and does not require mining negative examples during training.
  • 6. Conclusions: CosPlace-trained simple architectures surpass the current state of the art while using 8x smaller descriptors.The paper reports this result on the SF-XL dataset.
  • 6. Conclusions: CosPlace requires much less training resources and generalizes extremely well to other domains.The reported resource advantage includes roughly 80% less GPU memory at train time than current state of the art.
  • 6. Conclusions: CosPlace is not suited for training on small datasets or datasets without orientation labels.These are the limitations explicitly identified in the conclusion.

A. Further information on SF-XL

SF-XL combines city-wide StreetView coverage with temporally varied imagery and challenging crowd-sourced queries. Its two test-query sets differ in domain diversity and pose-label availability, with test v2 being more visually homogeneous.

  • Dataset characteristics: SF-XL’s training, validation, and test database images are 512 × 512 crops from 360° panoramas, with training density, temporal distribution, and query variability documented separately.Figure 6 uses a logarithmic y axis to show panoramas per cell, while Figures 7 and 8 show temporal distributions.
  • SF-XL test v1: SF-XL test v1 uses a homogeneous StreetView database but crowd-sourced queries with night scenes, grayscale images, viewpoint changes, occlusions, and varied cameras.The query images come from Flickr and therefore span multiple users and camera types.
  • SF-XL test v2: SF-XL test v2 shares test v1’s database and supports pose estimation through 6 DoF labels.Its queries are different from test v1’s queries.
  • SF-XL test v2: SF-XL test v2 queries are comparatively homogeneous, with mostly sunny-day images, clear views, and few heavy occlusions.This visual homogeneity is the main stated downside of the set.

B.1. Further ablations

The further ablations examine CosPlace hyperparameters, training-group count, and pooling choices using recall@1 on SF-XL validation data. Training with 8 of 50 groups is optimal in the reported group-count ablation, while standard pooling remains a comparison point for the architecture.

  • Hyperparameter ablations: The ablation varies M, α, N, L, and the number of training groups to study their relationships with final results.Figure 10 reports each hyperparameter’s recall@1 on the SF-XL validation set using ResNet-18.
  • Qualitative examples: Figure 9 provides additional SF-XL examples from the train set and the queries of test v1 and test v2.The rows are ordered as two train rows, two test v1 query rows, and two test v2 query rows.
  • Hyperparameter ablations: A single training group reduces recall@1 by just 1%, while the optimal result uses 8 of 50 groups.These results come from the rightmost plot of the group-count ablation.
  • Pooling ablation: CosPlace can outperform the previous state of the art even with a standard classification architecture using max pooling and a fully connected layer.The cited comparison concerns replacing GeM pooling in the CosPlace architecture.
  • Pooling ablation: The pooling ablation replaces GeM with max or average pooling and evaluates recall@1 with ResNet-18.The comparison is reported in Table 7.

B.2. Further implementation details

CosPlace training filters poorly represented cells and uses fixed hyperparameters to form groups and classes. The reported setup trains on 8 of 50 groups, covering roughly 13.6% of the SF-XL training images, leaving substantial room for improved scalability.

  • Training data: Cells with fewer than 10 panoramas are excluded from training, discarding about 15% of the images.The filtering ensures that each training class is well represented.
  • Training configuration: M = 10, α = 30, N = 5, and L = 2 create 50 groups with roughly 35k classes per group and 19.8 images per class on average.These are the hyperparameters used for the reported training setup.
  • Training configuration: Training uses 8 of the 50 groups, containing roughly 5.6M images out of the 41.2M-image SF-XL training set.Thus, the reported setup uses only 13.6% of the training images.
  • Optimization and augmentation: The implementation uses Adam with learning rate 0.00001, batch size 32, color jittering, random cropping, and cosFace margin 0.40.Random cropping is used for fairness with a prior method that employs smart region cropping.
  • Hyperparameters: CosPlace removes the need for several prior-method hyperparameters associated with negative sampling, caching, cropping, and NetVLAD clustering.The listed examples include negatives per query, cache refresh rate, random-negative pool size, potential-positive distance, and NetVLAD cluster count.

B.3.1 Further results on backbones and descriptors dimensionality.

The experiments examine how backbone choice and descriptor dimensionality affect CosPlace, emphasizing modern architectures and compact representations for practical visual geo-localization.

  • Backbones: Recent backbones are investigated to narrow the gap between research methods and real-world applications requiring strong results with low computational complexity.The evaluated backbones include VGG-16, ResNets, ViT, CCT224, and CCT384.
  • Backbones: CosPlace achieves encouraging results across backbone depths, with ResNets generally faster and lighter than VGG-16 while achieving comparable or better results.
  • Experimental caveat: Transformer results are lower in these experiments, but shared hyperparameters and smaller input resolutions may explain part of the difference.All experiments used the same learning rate and optimizer; CNNs used 512 × 512 resolution, while transformers used 224×224 or 384×384.

B.3.2 Comparison with other methods using same descriptors dimensionality.

Using a common 512-dimensional descriptor size makes CosPlace’s advantages over previous methods more noticeable, while comparisons with landmark-retrieval models are intended only as intuition.

  • Equal descriptor dimensionality: CosPlace’s advantages over previous methods become more noticeable when all methods use the same 512-dimensional descriptor size.These experiments reproduce the comparisons from the main paper using equal descriptor dimensionality.
  • Landmark-retrieval comparison: SF-XL models trained with CosPlace are compared against GLD and SfM120k models trained with triplet loss using equivalent 512-dimensional architectures.The compared architectures use ResNets, GeM pooling, and a fully connected layer.
  • Landmark-retrieval comparison: CosPlace cannot be used directly on GLD and SfM120k because those landmark-retrieval datasets lack GPS coordinates and heading labels.
  • Equal descriptor dimensionality: Tables 8 and 9 report equal-dimensionality comparisons on popular datasets and both SF-XL test versions.
  • Comparison scope: The SF-XL versus landmark-retrieval comparison is not rigorous because visual geo-localization and landmark retrieval are different underlying tasks.The authors present it only to provide intuition.

B.3.4 Comparison with other methods: qualitative results.

The paper provides qualitative comparisons of images retrieved by CosPlace and several previous state-of-the-art methods.

  • Qualitative comparisons: Figure 12 compares qualitative retrieval results from CosPlace, NetVLAD, CRN, SARE, and SFRS.
Loading 2204.02287v2…