PaperlayerView exact revision on arXiv

Source-linked AI summary

TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization

Sijie Zhu, Mubarak Shah, Chen Chen

arXiv:2204.00097v1cs.CV

TL;DR

Cross-view geo-localization methods rely on polar transforms and CNNs that have limitations in geometric robustness and global correlation modeling. TransGeo uses a pure transformer with explicit position encoding, ASAM, and attention-guided non-uniform cropping. It achieves state-of-the-art results on aligned and unaligned urban and rural datasets with lower computational cost than CNN-based methods.

  • Problem

    Existing cross-view geo-localization methods rely on polar transforms and CNNs, which can depend on view geometry and provide limited global correlation modeling.

  • Method

    TransGeo is a pure transformer-based method using explicit position embeddings, ASAM regularization, and attention-guided cropping to zoom computation into informative aerial regions.

  • Results

    TransGeo achieves state-of-the-art results on aligned and unaligned datasets with less computational cost than CNN-based methods.

  • Takeaways & Limitations

    The method provides a generic and flexible cross-view geo-localization pipeline without polar transforms or data augmentation.

  • Takeaways & Limitations

    TransGeo uses a two-stage pipeline, and its patch selection relies on a non-learnable attention map.

Abstract

from arXiv · show

The dominant CNN-based methods for cross-view image geo-localization rely on polar transform and fail to model global correlation. We propose a pure transformer-based approach (TransGeo) to address these limitations from a different perspective. TransGeo takes full advantage of the strengths of transformer related to global information modeling and explicit position information encoding. We further leverage the flexibility of transformer input and propose an attention-guided non-uniform cropping method, so that uninformative image patches are removed with negligible drop on performance to reduce computation cost. The saved computation can be reallocated to increase resolution only for informative patches, resulting in performance improvement with no additional computation cost. This "attend and zoom-in" strategy is highly similar to human behavior when observing images. Remarkably, TransGeo achieves state-of-the-art results on both urban and rural datasets, with significantly less computation cost than CNN-based methods. It does not rely on polar transform and infers faster than CNN-based methods. Code is available at https://github.com/Jeff-Zilence/TransGeo2022.

1. Introduction

Cross-view geo-localization methods face geometric domain gaps and limited global modeling, motivating TransGeo, a pure transformer approach with attention-guided computation allocation.

  • Polar transforms improve geometric alignment but rely on prior view geometry and may fail when street queries are spatially misaligned.
  • Vision transformers explicitly encode position information and model long-range correlations from the first layer, addressing key cross-view correspondence challenges.
  • Vanilla vision transformers require very large training datasets and substantial memory, while lacking CNN-like inductive biases on medium-scale data.
  • TransGeo combines a pure transformer architecture with ASAM and attention-guided non-uniform cropping to improve generalization and focus computation on informative aerial regions.
  • TransGeo reports state-of-the-art performance on urban and rural datasets with lower computation, GPU memory, and inference-time costs than CNN-based methods.

2. Related Work

Prior work mainly uses CNNs, polar transforms, or generative models to bridge cross-view appearance and geometry gaps, whereas TransGeo pursues a fully transformer-based alternative.

  • Two-stream CNN methods learn a shared embedding space but struggle with the substantial appearance gap between street-view and aerial images.
  • Polar-transform and generative approaches reduce domain differences by reshaping or synthesizing views, but depend heavily on geometric correspondence.
  • Recent methods address imperfect orientation or spatial alignment through dynamic similarity matching, activation maps, or datasets designed for unaligned views.
  • L2LTR combines CNN features with a transformer, so self-attention and position embeddings operate only on high-level CNN features rather than from the first layer.
  • Stage 2 of the proposed pipeline uses attention-guided cropping to remove uninformative aerial patches and reallocate computation to higher resolution in important regions.
  • TransGeo uses ASAM as a regularizer to reduce overfitting when training vision transformers without augmentation.

3. Method

TransGeo combines a two-stream vision transformer with attention-guided token selection and ASAM regularization for cross-view geo-localization. Its cropping strategy removes uninformative aerial patches and reallocates computation to higher resolution on informative regions.

  • Problem formulation: TransGeo trains street-view and aerial-view encoders with soft-margin triplet loss over positive and negative image pairs.The method samples exhaustive triplets from each mini-batch and L2-normalizes output embeddings.
  • Vision Transformer: Vision-transformer patch, class-token, and learnable position-embedding components produce embedding features while preserving token positional information.Images are divided into patches, projected into tokens, and combined with a learnable class token and positional embeddings.
  • Attention-guided cropping: The method uses the aerial branch’s class-to-patch attention map to identify informative regions and remove scattered, low-contribution patches.The attention map is reshaped to image space and used to guide non-uniform cropping rather than rectangular cropping.
  • Attention-guided cropping: Increasing image resolution by √γ creates γ times as many patches, while thresholding retains γβN patches after cropping.This reallocates computation toward informative regions while maintaining the patch size.
  • Attention-guided cropping: When β × γ = 1, cropping and resolution increase preserve the stage-1 baseline token count; γ = 1 instead reduces tokens for greater efficiency.Attention maps are computed once during stage-1 training, so the strategy adds no computation cost, and query inference remains unchanged.
  • Model optimization: ASAM jointly minimizes training loss and adaptive sharpness to encourage smooth loss curvature and improve generalization without data augmentation.The adaptive formulation addresses weight-scale dependence in transformer encoders.

4. Experiment

Experiments evaluate TransGeo on aligned rural and unaligned urban cross-view localization settings, reporting retrieval, meter-level accuracy, computation, and ablations. TransGeo performs strongly across datasets while reducing computational cost and avoiding polar transforms.

  • Datasets and metrics: TransGeo is evaluated on CVUSA and VIGOR, representing spatially aligned rural and spatially unaligned urban scenarios.The experiments use retrieval accuracy and, on VIGOR, meter-level GPS distance evaluation.
  • Comparison with state of the art: 49.7% and 72.6% relative R@1 improvements are reported over VIGOR for same-area and cross-area protocols, respectively.The cross-area protocol trains and tests on different cities, measuring robustness to distribution shift.
  • Comparison with state of the art: TransGeo achieves state-of-the-art CVUSA performance and outperforms methods without polar transform by a large margin.The method also uses less GPU memory and a smaller pre-training setup than L2LTR, while achieving better performance.
  • Computational cost: 26.8% of SAFA’s GFLOPs are required by TransGeo, while it also uses less training GPU memory and runs faster during inference.The comparison uses the same GTX 1080 Ti and batch size of 32.
  • Ablation studies: Polar transform provides only marginal improvement for TransGeo, while applying it to SAFA on VIGOR causes a performance drop in spatially unaligned settings.Learnable position embedding explicitly encodes geometric information, reducing the need for a predefined polar transform.
  • Computational cost: Attention-guided non-uniform cropping removes uninformative aerial-image patches with negligible performance drop and reallocates computation to higher resolution in important regions.The two-stage pipeline uses aerial attention maps to guide cropping and resolution allocation.

5. Conclusion and Discussion

TransGeo is a pure transformer method for cross-view geo-localization that achieves state-of-the-art results on aligned and unaligned datasets with lower computational cost than CNN-based methods. The authors identify a two-stage pipeline and non-learnable patch selection as limitations for future work.

  • TransGeo achieves state-of-the-art results on both aligned and unaligned datasets with less computational cost than CNN-based methods.
  • The method does not rely on polar transform or data augmentation, making it generic and flexible.
  • TransGeo uses a two-stage pipeline, which the authors identify as a limitation.
  • The patch-selection procedure relies on an attention map that is not learnable with parameters.

Supplementary Material

The supplementary material provides additional comparisons, evaluations, examples, qualitative results, and implementation details for TransGeo.

  • The supplementary material includes a head-to-head comparison with L2LTR and performance results on CVACT.
  • It also reports limited-FoV results on CVUSA and unknown-orientation results on VIGOR.
  • Additional materials cover polar-transform and non-uniform-crop examples, qualitative results, and implementation details.

A. Head-to-head Comparison with L2LTR

The supplementary material compares TransGeo with L2LTR and reports additional CVACT results, including effects of patch removal under different image resolutions.

  • TransGeo has clear superiority over L2LTR in both performance and computational efficiency.L2LTR uses vanilla ViT on top of ResNet, whereas TransGeo is purely transformer-based.
  • TransGeo achieves state-of-the-art results on CVACT.
  • Removing patches causes more performance drop on CVACT than on CVUSA.The authors suggest differing satellite-image resolutions as one possible explanation.
  • Table 10 compares previous methods on CVACT-val using R@k (%), with a dagger marking methods that use polar transform.

C. Unknown Orientation Results on VIGOR

Under unknown orientation, TransGeo outperforms VIGOR by a large margin, showing that its superiority does not depend on orientation alignment between the views.

  • TransGeo outperforms VIGOR by a large margin when panorama orientation is randomly shifted.
  • The result indicates that TransGeo’s superiority does not rely on orientation alignment between the two views.

D. Limited FoV results on CVUSA

On CVUSA with limited field of view and unknown orientation, TransGeo significantly outperforms DSM, with a larger gap at narrower views. Its attention-guided cropping also focuses on discriminative objects in rural scenes, despite more scattered attention.

  • D. Limited FoV results on CVUSA: TransGeo significantly outperforms DSM on CVUSA at FoV = 180° and FoV = 90° with unknown orientation.The performance gap becomes more significant as the field of view decreases.
  • D. Limited FoV results on CVUSA: The limited-FoV evaluation randomly crops panoramas with random shifts, while orientation remains unknown.
  • D. Limited FoV results on CVUSA: Polar transformation can distort unaligned views and poorly model correspondence when nearby tall buildings cause strong occlusion.Occluded roofs and other invisible regions occupy substantial transformed-image area without helping cross-view matching.
  • D. Limited FoV results on CVUSA: Rural attention maps appear more scattering and uniform than city maps but still focus more on discriminative objects such as roads.

G. Qualitative Results

Qualitative retrievals on CVUSA and VIGOR include both successful and failed cases. TransGeo retrieves visually similar ground truths effectively, while failures arise from ambiguous or degraded visual evidence and VIGOR’s semi-positive samples.

  • G. Qualitative Results: The qualitative examples deliberately cover ground-truth ranks 1, [2], [6, 100], and > 100 to represent success and failure cases.Ground-truth images are marked with red boxes in the retrieved results.
  • G. Qualitative Results: Ground-truth retrieval at rank 1 on both datasets can remain difficult because the second retrieved image is very similar.The authors interpret this as evidence of TransGeo’s strong discriminative ability.
  • G. Qualitative Results: CVUSA failures include extreme darkness, unrecognizable scenes dominated by road and grass, similar-colored negatives, and different capture seasons.
  • G. Qualitative Results: On VIGOR, semi-positive samples may rank first despite being GPS-close to the ground truth, while tree-and-road-only queries lack enough information for distinction.Semi-positive samples are not counted as correct top-1 predictions but support good meter-level evaluation.

H. Implementation Details

The implementation uses ASAM with specified optimizer regularization and follows the sampling strategy of prior work through a PyTorch reimplementation.

  • H. Implementation Details: ASAM uses ρ = 2.5, while AdamW uses weight decay 0.03 with default epsilon and other PyTorch parameters.
  • H. Implementation Details: The sampling strategy follows prior work [36] and is reimplemented in PyTorch.Further implementation details are provided in the code.
Loading 2204.00097v1…