PaperlayerView exact revision on arXiv

Source-linked AI summary

VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval

Sijie Zhu, Taojiannan Yang, Chen Chen

arXiv:2011.12172v2cs.CVcs.AI

TL;DR

Cross-view geo-localization commonly assumes an aerial reference centered exactly at each street-view query, which is unrealistic when queries are arbitrary and references are pre-captured. VIGOR introduces a benchmark with non-aligned, potentially multiple references and an end-to-end retrieval-plus-calibration framework. It evaluates both retrieval and GPS-based distance localization, while results show that the realistic setting remains challenging.

  • Problem

    Existing cross-view geo-localization datasets assume perfectly aligned one-to-one query-reference pairs, whereas practical queries can occur arbitrarily and references are captured beforehand.

  • Method

    VIGOR constructs a realistic benchmark and proposes an end-to-end framework combining aerial-image retrieval, within-image offset regression, and an IOU-based hybrid loss for semi-positive references.

  • Results

    The proposed framework provides coarse-to-fine localization and supports evaluation by actual GPS-based distance, while extensive results show strong potential in realistic settings.

  • Takeaways & Limitations

    VIGOR offers a testbed for studying cross-view geo-localization beyond one-to-one retrieval and for developing methods suited to realistic application settings.

  • Takeaways & Limitations

    Cross-view geo-localization remains challenging in the realistic setting introduced by VIGOR.

Abstract

from arXiv · show

Cross-view image geo-localization aims to determine the locations of street-view query images by matching with GPS-tagged reference images from aerial view. Recent works have achieved surprisingly high retrieval accuracy on city-scale datasets. However, these results rely on the assumption that there exists a reference image exactly centered at the location of any query image, which is not applicable for practical scenarios. In this paper, we redefine this problem with a more realistic assumption that the query image can be arbitrary in the area of interest and the reference images are captured before the queries emerge. This assumption breaks the one-to-one retrieval setting of existing datasets as the queries and reference images are not perfectly aligned pairs, and there may be multiple reference images covering one query location. To bridge the gap between this realistic setting and existing datasets, we propose a new large-scale benchmark -- VIGOR -- for cross-View Image Geo-localization beyond One-to-one Retrieval. We benchmark existing state-of-the-art methods and propose a novel end-to-end framework to localize the query in a coarse-to-fine manner. Apart from the image-level retrieval accuracy, we also evaluate the localization accuracy in terms of the actual distance (meters) using the raw GPS data. Extensive experiments are conducted under different application scenarios to validate the effectiveness of the proposed method. The results indicate that cross-view geo-localization in this realistic setting is still challenging, fostering new research in this direction. Our dataset and code will be released at \url{https://github.com/Jeff-Zilence/VIGOR}

1. Introduction

VIGOR reframes cross-view geo-localization for arbitrary query locations and non-aligned aerial references, addressing limitations of one-to-one retrieval. It introduces a benchmark and a coarse-to-fine framework that combines retrieval with within-image offset calibration.

  • Motivation: Existing datasets assume each street-view query has one aerial reference image centered exactly at the query location, an impractical real-world condition.Arbitrary query locations and pre-captured references break the one-to-one correspondence assumption.
  • Benchmark: VIGOR densely samples aerial references for seamless area coverage while capturing street-view queries at arbitrary locations.The benchmark contains 90,618 aerial images and 238,696 street panoramas from four major U.S. cities.
  • Beyond One-to-one: Multiple partially covering references can correspond to one query, so VIGOR introduces a hybrid loss that leverages these references during training.This setting replaces perfectly aligned pairs with positive and semi-positive reference relationships.
  • Beyond Retrieval: The end-to-end framework retrieves candidate images coarsely and predicts the query’s within-image offset for finer localization.Offset prediction shares feature descriptors with retrieval, enabling fast inference.
  • Evaluation: Raw GPS data enables evaluation by actual localization distance in meters rather than image-level retrieval alone.This provides a direct measure of geographic localization accuracy.
  • Contributions: The contributions include a realistic benchmark, joint retrieval-and-calibration, hybrid multi-reference supervision, and an assistive-navigation application scenario.The application scenario simulates noisy GPS for assistive navigation.

2. Related Work

Prior cross-view geo-localization datasets largely use aligned one-to-one image pairs and do not provide raw GPS for distance-based evaluation. Existing datasets vary in scale, city coverage, view information, and alignment construction.

  • Cross-view Datasets: Existing cross-view datasets generally enforce one-to-one correspondence between ground and aerial images.Some datasets densely sample aerial images but still force aligned pairings or generate aligned pairs by warping panoramas.
  • Cross-view Datasets: CVUSA includes more than 1 million ground-level and aerial images from multiple U.S. cities.A camera-parameter-based version provides 35,532 training pairs and 8,884 testing pairs.
  • Cross-view Datasets: Existing datasets do not provide raw GPS data for evaluating localization accuracy in meters.Their evaluation focuses on one-to-one retrieval rather than distance-based geographic error.

3. VIGOR Dataset

VIGOR constructs a realistic cross-view geo-localization benchmark by densely covering four urban areas with aerial references while allowing arbitrary street-view query locations. It provides raw GPS data and two evaluation settings to assess retrieval and meter-level localization beyond one-to-one alignment.

  • Sampling strategy: VIGOR densely samples aerial references with 50% overlap along latitude and longitude, ensuring arbitrary query locations are covered by four reference images.One reference is positive because it has the nearest GPS and the most shared objects; the other three are semi-positive.
  • Data collection: VIGOR retains aerial images covering no panoramas as distraction samples, with around 4% of aerial images falling into this category.Aerial images use zoom level 20 with approximately 0.114 m ground resolution, while ground-view images are 2048 × 1024.
  • Beyond one-to-one retrieval: Unlike one-to-one datasets, VIGOR permits reference images that are not centered at query locations and supports multiple references partially covering one query.This setting reflects arbitrary query locations and makes retrieval more realistic but challenging because even the ground-truth reference may have different GPS coordinates.
  • Evaluation protocol: The benchmark defines same-area and cross-area evaluation settings for applications that either train within the target area or transfer across held-out cities.In cross-area evaluation, New York and Seattle are used for training, while San Francisco and Chicago are held out.

4. Coarse-to-fine Cross-view Localization

The framework combines image retrieval for coarse localization with offset prediction for meter-level refinement. It uses semi-positive references and a hybrid training loss to support arbitrary query locations and multiple covering references.

  • Framework overview: The joint-retrieval-and-calibration framework localizes queries from coarse image retrieval to fine meter-level offset prediction.The Siamese network supplies retrieval embeddings, while an auxiliary predictor estimates the query location relative to the retrieved image center.
  • Baseline framework: The baseline uses SAFA feature aggregation and global hard-sample mining while omitting SAFA’s center-assuming polar transformation.The polar transformation is unsuitable because query and reference centers are not guaranteed to align.
  • Semi-positive assignment: Semi-positive aerial references are not treated as equally positive because they cover only part of the query scene.Their embedding similarity is constrained relative to positive samples rather than simply assigned the same label.
  • Semi-positive assignment: The IOU-based assignment loss matches the ratio of positive and semi-positive embedding similarities to their corresponding IOUs.The loss uses cosine similarities for positive and semi-positive pairs and IOU values to encode graded supervision.
  • Offset prediction: An MLP predicts the query’s latitude and longitude offset from the center of the top-1 retrieved aerial reference.The offset variables are converted to meters and normalized by the aerial-image width during training.

5. Experiments

Experiments evaluate retrieval and real-world meter-level localization under same-area and cross-area settings. The proposed method improves both evaluation types over prior approaches and the SAFA+Mining baseline.

  • Implementation and evaluation: The experiments resize panoramas and aerial images before training a VGG-16 backbone with eight SAFA blocks.The method uses Adam optimization and switches from triplet loss to hybrid loss after 30 same-area epochs or 10 cross-area epochs.
  • Implementation and evaluation: Evaluation includes top-k retrieval recall, top-1 hit rate when the reference covers the query, and real-world distance from the predicted location to the query GPS.The experiments report retrieval accuracy alongside meter-level localization accuracy.
  • Main results: The proposed method consistently outperforms previous approaches in both retrieval and meter-level evaluation.Comparisons are made under same-area and cross-area settings using Table 3 and Figure 5.
  • Main results: 124% relative improvement in 10-meter-level accuracy is reported over SAFA+Mining in the same-area setting, from 11.4% to 25.5%.This comparison uses the SAFA+Mining model as the baseline.
  • Main results: 121% relative improvement in 10-meter-level accuracy is reported over SAFA+Mining in the cross-area setting, from 2.8% to 6.2%.The result is reported for the cross-area evaluation shown in Figure 5.

6. Ablation Study

The ablation study evaluates semi-positive assignment, offset prediction, and sample balancing for retrieval and meter-level localization. IOU-based assignment and regression-based calibration provide the clearest benefits, while denser panorama sampling yields consistent modest gains.

  • Semi-positive Assignment: IOU-based semi-positive assignment improves performance over ignoring semi-positive samples, whereas treating semi-positive samples as positives hinders retrieval.The “Positive” and “Positive+IOU” strategies both reduce retrieval performance relative to the baseline.
  • Visualization: Visualization compares positive and semi-positive reference matching and illustrates predicted offsets against ground truth and retrieval-only centers.Figure 8 marks regression predictions, ground truth, and center predictions with distinct symbols.
  • Offset Prediction: Regression-based calibration almost doubles 10-meter-level localization accuracy, while classification performs poorly for calibration.The authors attribute classification’s weaker calibration to potentially ambiguous grid-based supervision.
  • Sample Balancing: More densely sampled panoramas bring slightly better performance across settings, with improvements consistent across balancing configurations.The balancing comparison evaluates “balancing-2” and “balancing-3” settings.

7. Application: Assistive Navigation

The assistive-navigation experiment tests cross-view geo-localization under noisy GPS signals. Smaller search scopes improve retrieval and meter-level evaluation, and the method substantially outperforms the noisy GPS baseline.

  • Noisy GPS Evaluation: Smaller search scopes generate better retrieval and meter-level results because they contain fewer negative reference samples.The same-area setting exceeds 70% accuracy for 30m-level localization.
  • Noisy GPS Evaluation: More than 70% same-area accuracy is achieved for 30m-level localization under the tested noisy-GPS application setting.The reported noisy GPS simulation adds random offsets in [−100m, 100m] to ground-truth GPS data.
  • Assistive Navigation: Cross-view geo-localization significantly improves localization accuracy compared with the original noisy GPS signal.The experiment is motivated by urban GPS errors that can reach up to 50 meters in Manhattan.

8. Conclusion

The paper introduces VIGOR as a realistic benchmark beyond one-to-one cross-view retrieval and pairs it with an end-to-end coarse-to-fine localization framework. Its experiments show potential in realistic settings and position the dataset as a testbed for further research.

  • Conclusion: VIGOR benchmarks cross-view image geolocalization beyond one-to-one retrieval in a more realistic setting for real-world applications.The conclusion presents the benchmark as a new testbed for cross-view geo-localization.
  • Conclusion: The end-to-end framework first coarsely localizes queries through retrieval and then refines locations by predicting offsets with regression.The conclusion describes this as a coarse-to-fine localization pipeline.
  • Conclusion: An IOU-based hybrid loss leverages supervision from semi-positive samples.This design addresses the beyond-one-to-one setting represented by VIGOR.
  • Conclusion: Extensive results show the proposed method’s potential in realistic settings and motivate further research in cross-view geo-localization.The dataset is intended to inspire novel research in the field.
Loading 2011.12172v2…