PaperlayerView exact revision on arXiv

Source-linked AI summary

GSV-Cities: Toward Appropriate Supervised Visual Place Recognition

Amar Ali-bey, Brahim Chaib-draa, Philippe Giguère

arXiv:2210.10239v1cs.CV

TL;DR

Large-scale visual place recognition lacks datasets combining accurate ground truth, broad geographic coverage, and environmental diversity. The paper introduces GSV-CITIES, evaluates supervised metric-learning strategies, and proposes Conv-AP; these choices improve existing methods and establish state-of-the-art benchmark results. The main scope boundary is that prior geotagged datasets often provide noisy GPS labels without known positive matches.

  • Problem

    Large-scale visual place recognition lacks datasets combining accurate ground truth, broad geographic coverage, and sufficient perceptual diversity for fully supervised training.

  • Method

    The paper introduces GSV-CITIES and combines its accurate labels with online hard-sample mining, metric-learning losses, and the fully convolutional Conv-AP aggregation layer.

  • Results

    Existing place-recognition methods substantially improve when trained on GSV-CITIES, while Conv-AP significantly outperforms existing aggregation techniques.

  • Takeaways & Limitations

    Accurate labels enable supervised sampling and metric-learning approaches that improve visual place-recognition performance while reducing reliance on weak supervision.

  • Takeaways & Limitations

    Existing geotagged datasets often provide noisy GPS labels without prior knowledge of positive matches, requiring weakly supervised losses.

Abstract

from arXiv · show

This paper aims to investigate representation learning for large scale visual place recognition, which consists of determining the location depicted in a query image by referring to a database of reference images. This is a challenging task due to the large-scale environmental changes that can occur over time (i.e., weather, illumination, season, traffic, occlusion). Progress is currently challenged by the lack of large databases with accurate ground truth. To address this challenge, we introduce GSV-Cities, a new image dataset providing the widest geographic coverage to date with highly accurate ground truth, covering more than 40 cities across all continents over a 14-year period. We subsequently explore the full potential of recent advances in deep metric learning to train networks specifically for place recognition, and evaluate how different loss functions influence performance. In addition, we show that performance of existing methods substantially improves when trained on GSV-Cities. Finally, we introduce a new fully convolutional aggregation layer that outperforms existing techniques, including GeM, NetVLAD and CosPlace, and establish a new state-of-the-art on large-scale benchmarks, such as Pittsburgh, Mapillary-SLS, SPED and Nordland. The dataset and code are available for research purposes at https://github.com/amaralibey/gsv-cities.

1 Introduction

Large-scale supervised visual place recognition is limited by datasets that lack accurate ground truth, broad geographic coverage, and sufficient perceptual diversity. The paper addresses these gaps with GSV-CITIES, a fully supervised training setup, a new aggregation layer, and metric-learning strategies.

  • Dataset limitations: Existing place-recognition datasets often lack broad geographic coverage, accurate ground truth, or sufficient appearance variation for large-scale supervised learning.These limitations include restricted areas, noisy labels, and missing viewpoint or structural changes.
  • Supervision challenge: Accurate positive pairs are difficult to obtain from GPS alone because images may face different directions and depict different places.This motivates weak supervision in prior training procedures.
  • Contributions: GSV-CITIES covers 40 cities across all continents over 14 years and provides accurate ground truth for straightforward mini-batch sampling.The dataset is designed to remove the weak-supervision bottleneck in large-scale training.
  • Contributions: Conv-AP is a fully convolutional aggregation layer that generates efficient representations and outperforms GeM, NetVLAD, and CosPlace.The proposed layer is presented as the paper’s image-representation contribution.
  • Contributions: Online hard-sample mining combined with modern metric-learning losses, including Multi-Similarity, improves visual place-recognition performance.The approach relies on the accurate labels provided by GSV-CITIES.

2 Related Works

Prior place-recognition methods use image representations and datasets with varying coverage, appearance diversity, and label quality. The paper introduces Conv-AP while addressing weak supervision caused by noisy or incomplete geographic annotations.

  • 2.1 Place recognition: Place recognition evolved from hand-crafted features and vector aggregation toward CNN-based representations with trainable aggregation layers.Related methods include Fisher Vectors, Bag of Words, VLAD, R-MAC, GeM, NetVLAD, and CosPlace.
  • 2.1 Place recognition: Conv-AP performs channel-wise pooling followed by spatial adaptive pooling, producing configurable representations that achieve state-of-the-art results on five benchmarks.The method is also 16× more compact than NetVLAD.
  • 2.2 Place recognition datasets: Existing datasets trade off geographic coverage, viewpoint variation, seasonal change, and ground-truth accuracy.SPED and Nordland provide accurate ground truth but have limited geographic or visual scope.
  • 2.3 Weak supervision: Google Street View datasets provide accurate GPS coordinates and viewpoint variation but lack viewing directions, making positive-pair formation from GPS alone impossible.This limitation complicates fully supervised training.
  • 2.3 Weak supervision: Noisy GPS labels leave positive matches unknown, so prior methods commonly use weakly supervised losses based on potential positives and definite negatives.The paper identifies this as a limitation associated with current datasets.

3 Methodology

The methodology combines GSV-CITIES, a geographically broad dataset with accurate place identities and temporal diversity, with supervised representation learning and a fully convolutional aggregation layer. This setup enables online mining of informative pairs or triplets and avoids weak supervision and offline hard-example mining.

  • Data collection and overview: GSV-CITIES uses Google Street View panoramas and bearings to collect perspectives of the same place across time, retaining locations visited at least four times.The collection spans 40 cities across all continents and uses historical panoramas from 2007 to 2021.
  • Data collection and overview: Each GSV-CITIES place is represented by images sharing an ID, with four to 20 images depicting the same physical location.The shared identity supports place-level sampling for supervised training.
  • Data collection and overview: GSV-CITIES contains over 67,000 places and provides broader temporal and geographical diversity than MSLS.Images are captured at multiple dates, and the dataset includes substantially more places represented at four or six dates than MSLS.
  • Supervised training and mining: Accurate place labels allow batches of P places with K images each, enabling supervised pair or triplet losses and online informative-pair mining.The framework is intended to remove weak supervision and offline hard-example mining while accelerating training and improving existing techniques.
  • Representation learning: The representation-learning pipeline maps each image to a normalized embedding whose cosine similarity is higher for images of the same place and lower for different places.A backbone and aggregation layer produce representations, while pairwise similarities supervise the learned mapping.
  • Fully convolutional feature aggregation: Conv-AP projects backbone feature maps into a compact channel space, applies adaptive average pooling, and flattens and L2-normalizes the resulting representation.Adaptive pooling produces fixed spatial dimensions, with global average pooling as the special case s1 = s2 = 1.

4 Experiments

The experiments define the evaluation setup for training and testing place-recognition models, including four benchmarks, Recall@k retrieval criteria, and standardized backbone and optimization settings.

  • Evaluation uses Pitts250k-test, MSLS, SPED, and Nordland, containing 8k, 750, 607, and 1622 query images, respectively.
  • Recall@k counts a retrieval as successful on Pitts250k and MSLS when a top-k reference lies within 25 meters of the query.
  • Models use an ImageNet-pretrained ResNet50 backbone cropped at the last residual block and extended with an aggregation layer.
  • Training uses batches of 100 places with four images each, SGD with momentum 0.9, and a maximum of 30 epochs at 320×320 resolution.

4.2 Importance of training with GSV-Cities

Training on GSV-CITIES substantially improves existing aggregation methods across place-recognition benchmarks, while accurate ground truth enables faster online mining than MSLS training.

  • GSV-CITIES training drastically improves AVG, GeM, and NetVLAD performance across all evaluated benchmarks.
  • AVG trained on GSV-CITIES gains 15.7, 14.2, 4.1, and 10.9 recall@1 percentage points on Pitts250k, MSLS, SPED, and Nordland versus MSLS training.
  • NetVLAD rises from 86.0% to 90.5% on Pitts250k, 59.5% to 82.6% on MSLS-val, and 4.1% to 32.6% on Nordland when trained on GSV-CITIES.Training NetVLAD takes 8 hours on GSV-CITIES versus 55 days on MSLS, attributed to online rather than offline mining.

4.3 Comparing different loss functions

The study compares five metric-learning losses on GSV-CITIES and finds that performance depends strongly on the loss and mining strategy.

  • The comparison evaluates Contrastive, Triplet, FastAP, Circle, and Multi-Similarity losses using a 20k-place GSV-CITIES subset.
  • Multi-Similarity loss achieves the best results on both evaluated benchmarks.The authors associate this result with its pair-mining and weighting strategy.
  • Contrastive loss becomes second best when combined with a sophisticated mining strategy, highlighting the role of informative online sample mining.

4.4 Comparison to state-of-the-art

Conv-AP is evaluated against existing aggregation methods under matched training configurations and achieves state-of-the-art results across four large-scale benchmarks.

  • All compared methods are trained on GSV-CITIES using the same configurations and hyperparameters.
  • Conv-AP reaches 92.4% on Pitts250k, 83.4% on MSLS, 81.2% on SPED, and 38.2% on Nordland.
  • Conv-AP2×2 obtains the best results and beats NetVLAD and CosPlace on every benchmark.The 2×2 adaptive pooling preserves four spatial sub-regions, whereas Conv-AP1×1 shows a relative performance drop associated with collapsing spatial dimensions.
  • Without re-ranking, Conv-AP exceeds Patch-NetVLAD by 3.9 recall@1 points and SuperGlue by 5 points on MSLS-val.On the Mapillary Challenge, Conv-AP achieves 68.0% versus 57.6% recall@5 for Patch-NetVLAD.

4.5 Further analysis

The analysis examines how Conv-AP’s depth and spatial pooling size affect representation dimensions and performance.

  • Conv-AP representations have size d×s1×s2 before flattening and L2 normalization.The experiments vary channel-wise pooling depth d and adaptive average pooling size s.
  • The study evaluates Conv_d-AP_s×s variants to measure how depth and spatial size affect performance.ResNet50 is used as the backbone in the reported configurations.
  • Reducing the depth d does not necessarily produce low performance.

4.6 Backbone architectures

The paper compares Conv-AP with multiple backbone architectures and finds that compact backbones can perform strongly, while ResNet50 performs best overall.

  • Compact backbones such as ResNet18, MobileNet, and EfficientNet-B0 achieve top-notch performance on challenging benchmarks with small memory footprints.These results are shown for Conv-AP models trained on GSV-CITIES.
  • ResNet50 obtains the best overall performance, especially on MSLS.

4.7 Dimensionality reduction

The paper studies PCA and Whitening for compact descriptors while noting that Conv-AP can also produce compact representations directly.

  • Conv-AP can generate 512-D outputs directly by setting d = 128 and s = 2.The authors nevertheless apply PCA for fair comparison with other techniques.
  • PCA and Whitening are used by state-of-the-art techniques to reduce descriptor dimensionality for efficient storage.The comparison evaluates Recall@1 on Pitts250k-test after PCA dimensionality reduction.
  • 512-D Conv-AP performs better than 4× larger 2048-D NetVLAD on Pitts250k-test Recall@1.Conv-AP also convincingly outperforms the other techniques in the reported comparison.

5 Conclusion

The paper concludes that GSV-CITIES provides large-scale, accurately labeled training data and that Conv-AP substantially improves visual place recognition performance. It reports new state-of-the-art results across four challenging benchmarks.

  • GSV-CITIES contains 560k images from 67k locations for training visual place recognition methods.Its highly accurate ground truth addresses the weak-supervision bottleneck and reduces training time.
  • Accurate labels enable metric learning loss functions to improve visual place recognition techniques.The conclusion presents this as motivation for further research into place-recognition-specific architectures and loss functions.
  • Conv-AP significantly outperforms existing techniques as a fully convolutional aggregation method.
  • New state-of-the-art results are established on Pitts250k-test, MSLS-val, SPED, and Nordland.
Loading 2210.10239v1…