PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Feature Transport for Cross-View Image Geo-Localization

Yujiao Shi, Xin Yu, Liu Liu, Tong Zhang, Hongdong Li

arXiv:1907.05021v3cs.CV

TL;DR

Cross-view geo-localization must match a ground-level street image to a large aerial map despite missing spatial-layout cues and a substantial ground–aerial domain gap. The paper introduces differentiable Cross-View Feature Transport, which maps features across domains for alignment; experiments report large recall gains and state-of-the-art performance. Its current input assumes a full 360° panorama.

  • Problem

    Cross-view localization methods often miss local-feature spatial layout and pay limited attention to the domain gap between ground and aerial images.

  • Method

    The paper introduces differentiable CVFT layers that transport features between ground and aerial domains using an entropy-regularized optimal transport plan.

  • Results

    The method achieves state-of-the-art cross-view localization performance and outperforms baselines by over 15% in recall@1 on both CVUSA and CVACT val datasets.

  • Takeaways & Limitations

    Explicitly handling domain differences through feature transport is presented as valuable for large-scale cross-view localization and potentially other computer-vision and machine-learning problems.

  • Takeaways & Limitations

    The method currently assumes that the input query image is a full 360° panorama; relaxing this restriction is left as future work.

Abstract

from arXiv · show

This paper addresses the problem of cross-view image geo-localization, where the geographic location of a ground-level street-view query image is estimated by matching it against a large scale aerial map (e.g., a high-resolution satellite image). State-of-the-art deep-learning based methods tackle this problem as deep metric learning which aims to learn global feature representations of the scene seen by the two different views. Despite promising results are obtained by such deep metric learning methods, they, however, fail to exploit a crucial cue relevant for localization, namely, the spatial layout of local features. Moreover, little attention is paid to the obvious domain gap (between aerial view and ground view) in the context of cross-view localization. This paper proposes a novel Cross-View Feature Transport (CVFT) technique to explicitly establish cross-view domain transfer that facilitates feature alignment between ground and aerial images. Specifically, we implement the CVFT as network layers, which transports features from one domain to the other, leading to more meaningful feature similarity comparison. Our model is differentiable and can be learned end-to-end. Experiments on large-scale datasets have demonstrated that our method has remarkably boosted the state-of-the-art cross-view localization performance, e.g., on the CVUSA dataset, with significant improvements for top-1 recall from 40.79% to 61.43%, and for top-10 from 76.36% to 90.49%. We expect the key insight of the paper (i.e., explicitly handling domain difference via domain transport) will prove to be useful for other similar problems in computer vision as well.

1 Introduction

Cross-view geo-localization estimates a ground image’s location by matching it to a geo-tagged aerial map, despite viewpoint and appearance differences. The paper proposes CVFT to preserve spatial layout and explicitly transfer features between ground and aerial domains.

  • Problem: Ground-to-aerial localization estimates a street scene’s geo-location relative to a large geo-tagged satellite map.The task is practically relevant because high-resolution satellite imagery is widely accessible.
  • Problem: Viewpoint differences and substantial visual-appearance variation make cross-view localization challenging.The query is ground-level, whereas the reference image is overhead.
  • Approach: CVFT explicitly addresses both local-feature spatial layout and cross-domain feature differences.It is designed to retain positional information while accounting for the ground–aerial domain gap.
  • Approach: CVFT transports ground-domain feature maps into the aerial domain so corresponding features become closer in position and feature space.The transported representation embeds spatial layout information and supports more discriminative similarity comparison.
  • Contribution: The differentiable CVFT module is trained end-to-end, and the paper reports a new state-of-the-art cross-view image geolocalization method.The contribution is presented as a domain-transfer module that facilitates cross-domain feature similarity matching.

2 Related works

Prior cross-view geo-localization work largely uses two-branch CNNs and deep metric learning for ground–aerial matching. Related research also includes global aggregation, viewpoint-invariant descriptors, and efficient entropy-regularized optimal transport.

  • Cross-view geo-localization: Most existing cross-view matching methods use two-branch CNNs to learn ground and aerial representations as a deep metric-learning problem.Earlier work explored pretrained-network fine-tuning, Siamese and triplet architectures, and classification versus retrieval formulations.
  • Cross-view geo-localization: NetVLAD-based descriptors were developed to provide representations invariant to large viewpoint changes.This work is situated among two-branch CNN approaches for cross-view matching and localization.
  • Cross-view architectures: Figure 2 contrasts an existing network with global feature aggregation against the proposed network using Feature Transport.The comparison frames Feature Transport as an alternative to aggregation-based cross-view matching.
  • Optimal transport: Entropy-regularized optimal transport can be solved efficiently with the Sinkhorn-Knopp algorithm and has been applied to domain adaptation and other machine-learning problems.This provides the relevant optimal-transport background for feature transport.

3 Motivation: Cross-View Feature Transport

The motivation is that conventional global aggregation can discard spatial layout, while direct comparison of ground and aerial features can overlook their domain correspondence. The paper therefore uses optimal feature transport to align cross-domain features spatially.

  • Limitations of existing methods: Global pooling and VLAD-style aggregation can discard the relative positions of local deep features.Spatial layout is described as useful for geo-localization and navigation.
  • Cross-view domain gap: Ground and satellite images differ in viewpoint and appearance, so direct deep-feature distance may neglect important domain correspondences.The ground image is panoramic and ground-level, whereas the satellite image is an overhead view.
  • Feature Transport: The proposed Feature Transport block uses a transport matrix to map features from one domain, such as ground, to another, such as aerial.The design targets drastic appearance changes and geometric distortions.
  • Optimal transport formulation: Entropy-regularized optimal feature transport is solved with differentiable Sinkhorn operation for integration into a deep neural network.The resulting doubly stochastic transport plan encodes cross-domain feature relationships.
  • Feature alignment: The transport plan maps ground features into the aerial domain so matching features can be spatially aligned.This explicitly exposes relationships between ground and aerial features for cross-domain alignment.

4 The Proposed CVFT Framework

The CVFT framework extracts ground and aerial feature maps, formulates their alignment as an optimal-transport problem, and uses differentiable Sinkhorn optimization to transport features before metric learning.

  • Framework: A two-branch Siamese-like architecture separately learns feature maps for ground and aerial images.
  • Feature Extraction: The backbone preserves spatial layout by reducing channel dimension while retaining 8 × 8 feature-map resolution, producing 8 × 8 × 64 = 4096-dimensional features.
  • Feature Transport: Optimal transport generalizes from distributions to features to expose a transport plan for cross-domain feature alignment.
  • Feature Transport: A regression block generates the cost matrix from one domain, avoiding the expense of computing it for every gallery image.
  • Sinkhorn Solver: The entropy-regularized transport objective uses Sinkhorn normalization to obtain a doubly stochastic transport matrix, with λ controlling solution smoothness and sparsity.
  • Feature Alignment and Metric Learning: After convergence, the differentiable transport matrix moves feature maps between views, aligning ground features with overhead features before triplet-loss metric learning.

5 Experiments

Experiments evaluate CVFT on CVUSA and CVACT using recall-based benchmarks, ablations, and feature-map visualizations. Results indicate that preserving spatial layout and explicitly modeling cross-view correspondences substantially improves localization.

  • Datasets and Metrics: CVUSA and CVACT each provide 35,532 ground-and-aerial image pairs for training, while CVUSA has 8,884 test pairs and CVACT has 8,884 validation pairs.CVACT additionally provides 92,802 accurately geo-tagged test pairs.
  • Datasets and Metrics: r@K measures how many satellite images must be retrieved to find the true matching image.The experiments report recall at top-1, 5, and 10, among other settings.
  • Ablation Study: The network without CVFT achieves top-1 recalls of 41.68% on CVUSA and 45.28% on CVACT val, but struggles to establish spatial correspondences.This baseline preserves relative spatial information without explicitly constructing cross-view correspondences.
  • Ablation Study: Global max pooling produces the worst recall performance, indicating that local feature spatial layout is critical for cross-view geo-localization.The comparison removes spatial layout information while retaining the VGG conv5_3 representation.
  • Overall Results: Over 15% improvements in recall@1 on both CVUSA and CVACT val are obtained with CVFT compared with the baselines.The full network combines spatial-layout modeling with explicit domain correspondence construction.
  • Feature Alignment: CVFT transports ground-view feature maps toward corresponding aerial-view maps, producing better-aligned representations for similarity comparison.The transported ground features are visually closer to the directly extracted aerial features.

5.3 Comparisons with the State-of-the-Art

The paper compares CVFT with prior cross-view localization methods on CVUSA, CVACT validation, and the large-scale CVACT test set. CVFT achieves large improvements under both retrieval-based and distance-based evaluation protocols.

  • Benchmark Comparisons: The comparison includes Workman et al., Vo et al., Zhai et al., CVM-NET, and Liu and Li, with recall at top 1% reported in Table 2.The paper also reports recalls at top-1, 5, and 10 for more practical retrieval analysis.
  • Benchmark Comparisons: 20.64% and 14.09% improvements in top-1 recall over Liu and Li are reported on CVUSA and CVACT val, respectively.The paper identifies Liu and Li as the second-best method in this comparison.
  • CVACT Test Evaluation: The CVACT test evaluation counts a query as correctly localized when one of the top K satellite images lies within 25m of the ground-truth position.Recall is plotted for different values of K under this distance-based protocol.
  • Orientation Setting: CVUSA and CVACT ground panoramas and overhead aerial images are north aligned, while orientation perturbation is discussed when ground-view north direction is unavailable.An approximate north direction may be estimated from scene clues such as tree shadows.

6 Conclusion

The paper concludes that CVFT bridges cross-view domain differences and significantly improves top-K recall for large-scale cross-view localization. It currently assumes full 360° panorama queries, while relaxing this restriction is left for future work.

  • CVFT bridges the cross-view domain gap by transforming features between ground and aerial domains.
  • The method significantly improves large-scale cross-view localization performance in terms of top-K recall rate.
  • The method is illustrated on ACT with top-5 retrieval comparisons against Liu & Li and CVM-NET, using green and red borders for correct and incorrect results.
  • The current method assumes that each input query is a full 360° panorama.
  • Relaxing the full-panorama assumption within the CVFT framework is identified as a possible future extension.

A Additional Visualization of Cross-view Feature Alignment

Additional visualizations show that CVFT effectively aligns feature maps between cross-view images.

  • Figure 11 presents cross-view feature maps before and after CVFT.
  • The visualizations demonstrate that CVFT can effectively align feature maps of cross-view images.

B Comparison with STN

The STN comparison evaluates whether simple geometric transforms can model cross-view feature relationships. Affine and thin-plate-spline STNs perform worse than CVFT, suggesting that a few-parameter geometric transform is insufficient.

  • Ground panoramas and satellite images share scene contents but exhibit large appearance differences because they come from distinct viewing geometries.
  • Conventional STNs regress a small set of geometric-transform parameters, such as affine or thin-plate-spline transformations.
  • 2D-affine and thin-plate-spline STNs perform worse than the proposed CVFT network in the Table 3 comparison.
  • The comparison implies that cross-view feature relationships cannot be represented by simple geometric transforms with only a few parameters.

C The Impact of Dimensionality Reduction

This section studies how changing feature embedding dimensionality affects the proposed CVFT model on CVUSA and CVACT.

  • The proposed CVFT uses a 4096-dimensional feature embedding, matching CVM-net, while Liu and Li’s work uses 1536 dimensions.
  • Different embedding dimensions are produced by resizing the output feature map to different spatial sizes.
  • The resulting recall performance is evaluated for different CVFT embedding dimensions on CVUSA and CVACT.

D Visualization of Cross-view Embeddings

The experiments visualize how CVFT aligns cross-view feature maps and embeds matched ground-satellite image pairs nearby despite significant viewpoint changes.

  • Matched cross-view image pairs are embedded into nearby positions in the CVACT t-SNE space despite significant viewpoint changes.The embedding goal is to place features from the same scene close together.
  • Table 3 compares recall performance between the proposed framework and spatial transformer networks.
  • Table 4 reports CVFT recall performance under different feature embedding dimensions.
  • CVFT-transported ground-view feature maps are well-aligned with the corresponding satellite image feature maps.Figure 11 compares input ground-view, satellite, and transported ground-view feature maps.
  • Figure 12 visualizes cross-view feature embeddings on the CVACT dataset using t-SNE, with red identifiers marking cross-view image pairs.
Loading 1907.05021v3…