PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation

Fabian Deuser, Konrad Habel, Norbert Oswald

arXiv:2303.11851v2cs.CV

TL;DR

Cross-view geo-localisation must match images with different viewpoints, while conventional pipelines make hard-negative integration difficult and often rely on specialized processing. The paper uses a weight-sharing CNN with symmetric InfoNCE and staged GPS- and visual-similarity sampling. It reports strong performance and generalisation across CVUSA, CVACT, University-1652, and VIGOR without complex preprocessing or task-specific aggregation modules.

  • Problem

    Cross-view geo-localisation remains difficult because different image views have different geometries and standard losses make hard negatives difficult to include.

  • Method

    The paper combines a weight-sharing Siamese CNN, symmetric InfoNCE, GPS-Sampling, and Dynamic Similarity Sampling for hard-negative mining.

  • Results

    The approach shows strong performance and generalisation on VIGOR, CVUSA, CVACT, and University-1652 without complex preprocessing or task-specific aggregation modules.

  • Takeaways & Limitations

    A lightweight shared-encoder pipeline can achieve outstanding cross-area generalisation without the specialized processing used by many prior approaches.

  • Takeaways & Limitations

    Existing benchmarks are dominated by urban, road-based, aligned imagery, limiting their representation of varied scenes and less constrained viewpoints.

Abstract

from arXiv · show

Cross-View Geo-Localisation is still a challenging task where additional modules, specific pre-processing or zooming strategies are necessary to determine accurate positions of images. Since different views have different geometries, pre-processing like polar transformation helps to merge them. However, this results in distorted images which then have to be rectified. Adding hard negatives to the training batch could improve the overall performance but with the default loss functions in geo-localisation it is difficult to include them. In this article, we present a simplified but effective architecture based on contrastive learning with symmetric InfoNCE loss that outperforms current state-of-the-art results. Our framework consists of a narrow training pipeline that eliminates the need of using aggregation modules, avoids further pre-processing steps and even increases the generalisation capability of the model to unknown regions. We introduce two types of sampling strategies for hard negatives. The first explicitly exploits geographically neighboring locations to provide a good starting point. The second leverages the visual similarity between the image embeddings in order to mine hard negative samples. Our work shows excellent performance on common cross-view datasets like CVUSA, CVACT, University-1652 and VIGOR. A comparison between cross-area and same-area settings demonstrate the good generalisation capability of our model.

1 Introduction

Cross-view geo-localisation matches ground or street views with satellite imagery, but differing geometries and the difficulty of incorporating hard negatives complicate training. The paper proposes a simple symmetric-InfoNCE framework with GPS- and similarity-based sampling that avoids specialized modules and preprocessing.

  • Motivation: Cross-view geo-localisation compares ground images with satellite databases to determine geographic positions.The task is relevant to positioning when GPS is expensive, unavailable, or unreliable.
  • Motivation: Different view geometries motivate methods such as polar transformation, while Transformer or MLP-Mixer approaches may require separate view-specific encoders.Polar transformation can bridge geometry but introduces image distortion.
  • Contributions: The paper presents a weight-sharing Siamese CNN that learns class-agnostic embeddings with symmetric InfoNCE for both view directions.The approach uses contrastive learning rather than separate view-specific encoders.
  • Contributions: GPS-Sampling contrasts geographically close satellite locations during early training to initialize hard-negative mining.This strategy exploits neighboring locations as an initial source of difficult negatives.
  • Contributions: Dynamic Similarity Sampling selects hard negatives using cosine similarity between street-view and satellite-view embeddings during training.The method focuses batch construction on visually similar samples.
  • Contributions: The framework uses a simple training pipeline that eliminates special aggregation modules and complex preprocessing while improving performance and generalisation.The contribution passage characterizes the pipeline as outperforming current approaches in both performance and generalisation ability.

2 Related Work

Related work progresses from CNN and Siamese approaches toward Transformer-based models, orientation-aware objectives, geometric preprocessing, refinement, and stronger benchmarks. These methods address view differences and challenging localization settings through increasingly specialized architectures and operations.

  • Earlier CNN and orientation methods: Early cross-view methods used CNN features, Siamese networks, L2 targets, semantic labels, and known image orientation.CVUSA became a major benchmark, while later work introduced CVACT and orientation-related objectives.
  • Geometric alignment: Polar transformation was introduced to reduce the geometric gap between satellite and street views, often alongside Spatial-Aware Feature Aggregation.The transformation stretches satellite imagery toward the street-view domain.
  • Geometric alignment: Polar transformation can inject image distortion, prompting later work to learn disturbance removal with GANs.At inference, the generator and discriminator were discarded and only latent representations were used for similarity search.
  • Transformer-based models: Later approaches combined CNN backbones with Transformers and positional encodings to process feature maps.This reflects the field’s shift toward Transformer-based architectures.
  • Datasets and refinement: VIGOR was introduced as a more challenging benchmark, while University-1652 used UAV images from university buildings instead of street views.These datasets broadened evaluation beyond the earlier street-view-to-satellite setting.
  • Datasets and refinement: TransGeo and SIRNet added refinement mechanisms, including attention-based zooming, extra refinement modules, and sharpness-aware minimization.These methods address the limitation of single-step prediction and aim to improve generalisation.

3 Methodology

The methodology combines symmetric InfoNCE, a weight-shared ConvNeXt Siamese encoder, and staged hard-negative sampling. GPS-based sampling initializes training, after which Dynamic Similarity Sampling mines visually similar examples for batch construction.

  • Loss design: Triplet loss contrasts one positive and one negative with an anchor, making it difficult to leverage multiple hard negatives directly.The framework is designed to use multiple hard negatives instead.
  • Loss design: InfoNCE uses all available batch negatives, contrasting one positive with N-1 negatives and calculating similarity through dot products.The loss is paired with cross-entropy and includes a temperature hyperparameter.
  • Loss design: The symmetric formulation propagates information from satellite to street views and from street to satellite views.This bidirectional objective is used to bridge the two image modalities.
  • Model architecture: The model uses a weight-shared Siamese ConvNeXt encoder with mean-pooled feature vectors and no attention-pooling or refinement modules.ConvNeXt was selected over a Vision Transformer for this approach.
  • Hard-negative sampling: GPS-based sampling mines negatives from geographically related images during early epochs, except for University-1652, where shuffling initializes training.Nearby images share properties such as vegetation, street signs, or housing type.
  • Hard-negative sampling: Dynamic Similarity Sampling periodically computes cosine similarities, selects each query’s top K neighbors, and constructs batches from sorted and randomly chosen neighbors.The reported settings are k = 64, K = 128, and e = 4, with duplicate entries removed before batching.

4 Evaluation

The evaluation covers four cross-view geo-localisation benchmarks and compares the proposed architecture across datasets, architectures, sampling strategies, and generalisation settings. Results report strong benchmark performance, parameter efficiency, sampling gains, and transfer to unknown regions.

  • Benchmark setup: Experiments evaluate the approach on CVUSA, CVACT, University-1652, and VIGOR using comparisons with prior work.The evaluation includes standard benchmark tables and multiple dataset-specific settings.
  • CVUSA & CVACT: CVUSA uses 35,532 training pairs and 8,884 evaluation pairs, while CVACT adds a 92,802-image test split with higher-resolution urban imagery.Both datasets use aligned one-to-one street-view and satellite-view mappings.
  • CVUSA & CVACT: The approach outperforms previous work on CVUSA and CVACT in R@1 without using polar transformation.The CVACT test split is more difficult because its denser street-view sampling creates many semi-positive matches.
  • VIGOR: On VIGOR, the model outperforms current work in same-area evaluation and retains strong performance in the harder cross-area setting involving unknown cities.The cross-area split trains on New York and Seattle and evaluates on San Francisco and Chicago.
  • Architecture evaluation: Using a single shared encoder outperforms two separate encoders, while increasing parameter count provides only a marginal performance advantage.The architecture uses ConvNeXt-B with 88M parameters in the experiments.
  • Sampling strategies: Contrastive InfoNCE training is competitive without sampling, but the proposed sampling strategy substantially improves performance.GPS sampling remains consistent across same-area and cross-area experiments, supporting its use for generalisation.
  • Generalisation capabilities: Transfer experiments indicate good generalisation between CVUSA and CVACT, although unknown-region performance remains below same-area performance.Incorrectly retrieved satellite images are often visually similar in road course, and the model tends to encode road-course features.
  • Feature analysis: Activation visualisations show that streets are especially important features in CVUSA, whereas VIGOR is harder because multiple semi-positive images share features such as trees.The presence of several semi-positives makes clear prediction more difficult in VIGOR.

5 Conclusion

The paper presents a lightweight cross-view geo-localisation approach using one encoder and contrastive learning, achieving strong results without complex preprocessing or task-specific modules.

  • A single modern CNN encoder handles both satellite and ground views while training uses the InfoNCE loss.
  • The approach achieves superior results on VIGOR, CVUSA, CVACT, and University-1652 through effective hard-negative sampling.
  • The model achieves strong cross-area generalisation without complex preprocessing, task-specific aggregation modules, SAM, or ASAM.

6 Discussion

The discussion identifies important dataset limitations: common benchmarks simplify localisation through urban, aligned, and near-building imagery. More diverse, less aligned data are needed for broader applicability.

  • Existing datasets often simplify localisation because images are captured near buildings, while some benchmarks align street-view positions with satellite-image centres.
  • Future benchmarks should include unknown geographic orientations, non-centred positions, and fields of view below 120 degrees.
  • Most datasets focus on urban street scenes, limiting representation of scenes in the wild and encouraging matching through streets and intersections.
  • Future datasets should add ground views taken away from roads to increase diversity, usefulness, and practicability.

2 Hyperparameter

The analyses select the number of similarity-sampling neighbours and compare loss directions. DSS performs best with 64 neighbours, while symmetric loss benefits the model most.

  • k = 64 is selected from a nearest-neighbour pool of K = 128 for Dynamic Similarity Sampling on the VIGOR same split.
  • Sampling all K hard negatives does not collapse the method to a simple solution when the random factor is removed.
  • The VIGOR same-split comparison evaluates unidirectional and symmetric loss calculations, with symmetric loss providing the greatest benefit.

3 Symmetric InfoNCE Loss

The loss-direction analysis shows that symmetric InfoNCE is important because the sampling strategy benefits one query direction more than the other.

  • Performance declines when satellite views query street views, whereas street-view queries against satellite references perform almost as well as the symmetric loss.
  • DSS samples street-view images close to satellite images, so the reverse loss direction cannot benefit equally from the sampling.
  • The symmetric formulation enables both directions to benefit from hard-negative sampling and achieves the best performance.

4 Loss comparison

The comparison examines loss functions and hard-negative sampling. Triplet loss collapses with only hard negatives, while combining GPS sampling and DSS gives the best sampling results across datasets.

  • Loss functions: Triplet loss collapses when trained with only hard negatives, whereas the soft-margin extension avoids this failure in the comparison.The paper attributes the collapse to the unextended triplet loss when only hard negatives are used within a batch.
  • Experimental comparisons: Table 9 compares triplet, soft-margin triplet, and InfoNCE losses for CVUSA, while Table 10 compares sampling strategies for CVUSA/CVACT.The supplied table passages provide the comparison scope but no numerical values.
  • Sampling strategies: The combination of GPS sampling and DSS produces the best results regardless of whether the dataset is CVUSA, CVACT, or VIGOR.The two sampling strategies individually behave differently depending on the dataset, especially when sampled locations are geographically close.

6 Visualisation

The visualisations show how activation maps align the two views and which scene features guide predictions. Performance transfers to new regions but remains lower than in same-region evaluation.

  • Activation-map alignment: An inverse polar transformation of street-view activation maps makes correspondence between the two views easier to visualise.The transformation is applied only for visualisation of activation-map alignment.
  • Cross-region transfer: The model performs well on new regions, but transferability is lower than performance when training and evaluation use the same region.The visual comparison includes false predictions from CVACT evaluated on CVUSA using visual inspection and cosine similarity.
  • Activation patterns: In VIGOR, correct predictions attend particularly to vegetation and road markings.These visual cues are highlighted in a correct predicted sample.
  • Activation patterns: On CVUSA, correct in-domain predictions mainly focus on road features, especially road course and intersection positions.The activation maps are visualised for a model trained in-domain on CVUSA.
  • Dataset context: VIGOR’s more urban training data makes roads more dominant cues in rural settings.The paper contrasts this rural-setting behavior with VIGOR’s urban-heavy training distribution.
Loading 2303.11851v2…