PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging the Domain Gap for Ground-to-Aerial Image Matching

Krishna Regmi, Mubarak Shah

arXiv:1904.11045v2cs.CV

TL;DR

Ground-to-aerial matching is difficult because viewpoint changes create a domain gap between query panoramas and aerial references. The paper synthesizes aerial views with conditional GANs, jointly learns and fuses ground and synthesized features, and reports improvements over state-of-the-art methods on CVUSA.

  • Problem

    Ground panoramas and aerial reference images differ substantially in viewpoint, making cross-view image retrieval challenging.

  • Method

    The method synthesizes aerial representations from ground panoramas, jointly learns triad features, fuses synthesized-aerial and ground features, and aggregates features across scales.

  • Results

    The proposed approach significantly improves retrieval performance over state-of-the-art methods on the CVUSA dataset, including top-1 and top-10 accuracies with feature fusion.

  • Takeaways & Limitations

    Synthesized aerial images provide complementary information that can be incorporated into cross-view matching to obtain more robust ground-query descriptors.

  • Takeaways & Limitations

    Failures include cases with visual differences caused by different capture times and distant buildings that are difficult to infer from street view.

Abstract

from arXiv · show

The visual entities in cross-view images exhibit drastic domain changes due to the difference in viewpoints each set of images is captured from. Existing state-of-the-art methods address the problem by learning view-invariant descriptors for the images. We propose a novel method for solving this task by exploiting the generative powers of conditional GANs to synthesize an aerial representation of a ground level panorama and use it to minimize the domain gap between the two views. The synthesized image being from the same view as the target image helps the network to preserve important cues in aerial images following our Joint Feature Learning approach. Our Feature Fusion method combines the complementary features from a synthesized aerial image with the corresponding ground features to obtain a robust query representation. In addition, multi-scale feature aggregation preserves image representations at different feature scales useful for solving this complex task. Experimental results show that our proposed approach performs significantly better than the state-of-the-art methods on the challenging CVUSA dataset in terms of top-1 and top-1% retrieval accuracies. Furthermore, to evaluate the generalization of our method on urban landscapes, we collected a new cross-view localization dataset with geo-reference information.

1. Introduction

The paper addresses ground-to-aerial image matching by synthesizing an aerial representation of a ground panorama, then jointly learning and fusing complementary multi-scale features to bridge the viewpoint domain gap.

  • Joint Feature Learning uses ground, synthesized aerial, and real aerial images, while Feature Fusion combines synthesized aerial and ground features into a robust query representation.
  • Cross-view matching compares ground queries with aerial references, which capture the same scene differently because of viewpoint variation.
  • Conditional GANs synthesize aerial images from ground panoramas so the transformed representation resembles the target aerial view.
  • The proposed task is to retrieve matching aerial images given a ground-level panorama.
  • Multi-scale feature aggregation preserves representations across feature scales for the complex image-matching task.
  • The approach outperforms state-of-the-art methods on CVUSA, improving top-1 and top-10 retrieval accuracies with feature fusion.

2. Related Works

Prior geolocalization work includes same-view matching, cross-view descriptors, and GAN-based domain translation; this paper instead exploits synthesized aerial images as complementary information for cross-view matching.

  • GAN-based research has learned mappings between domains and enabled image translation between aerial and ground-level views.
  • Cross-view matching addresses the harder setting of aerial and ground views, where objects have limited semantic and viewpoint overlap.
  • Multi-scale features have been used across vision tasks, including detection, segmentation, key-point detection, pose estimation, and localization.
  • Early geolocalization methods matched query street-view images against reference images from the same view, often using hand-crafted features.
  • Unlike approaches that primarily learn view-invariant descriptors, this work uses synthesized aerial images as a complementary information source for street-view understanding.

3. Method

The method synthesizes aerial images from panorama and edgemap inputs, jointly learns ground, synthesized-aerial, and real-aerial representations, and fuses complementary query features for matching.

  • 3.1. Cross-View Image Synthesis: The GAN uses an X-Fork multi-task generator to produce an aerial RGB image and segmentation map from a panorama augmented with a Canny edgemap.
  • 3.2. Joint Feature Learning: Joint Feature Learning trains on ground panoramas, synthesized aerial images, and real aerial images so synthesized features help align ground and aerial representations.
  • 3.2. Joint Feature Learning: Aerial and street-view encoders use separate weights, while synthesized and real aerial encoders share weights because both represent the aerial domain.
  • 3.2. Joint Feature Learning: The synthesized branch is used during training to influence aerial representations but is not required during testing.
  • 3.2.1 Weighted Soft-margin Triplet Loss: The weighted soft-margin triplet loss brings matching aerial features closer to ground features while separating non-matching aerial features.
  • 3.2.1 Weighted Soft-margin Triplet Loss: The joint objective combines losses for ground–real-aerial and synthesized-aerial–real-aerial pairs, balanced by λ1 and λ2.
  • 3.3. Feature Fusion: Feature Fusion concatenates ground and synthesized-aerial features, refines them through a fully connected layer, and learns a robust ground-query representation.

4. Experimental Setup

Experiments use CVUSA and a newly collected Orlando-Pittsburgh dataset, with cross-view synthesis and matching networks trained under specified architectural and optimization settings.

  • Datasets: CVUSA contains 35,532 training and 8,884 testing pairs of satellite and ground-panorama images for cross-view matching.Aerial images are 750×750, while ground panoramas are 224×1232.
  • Datasets: The Orlando-Pittsburgh dataset contains geo-referenced aerial and ground-panorama pairs from urban areas of Orlando and Pittsburgh.It includes 1,910 training and 722 testing pairs and differs from CVUSA’s rural coverage.
  • Implementation Details: The cross-view synthesis generator uses an encoder and two decoders to produce a cross-view image and its segmentation map from a four-channel RGB-plus-edgemap input.The architecture follows X-Fork, with shared decoder blocks and end-to-end GAN training.
  • Implementation Details: Joint feature learning concatenates multi-scale features from each input stream and reduces them to 1,000-dimensional feature vectors.Each stream has seven convolutional layers with ReLU activations, and dropout is applied after the final three ReLU layers.
  • Implementation Details: Feature Fusion uses separate fully connected branches to map fused aerial-ground features and aerial features to 1,000-dimensional representations.The upper branch processes a 2,000-dimensional fused feature.
  • Implementation Details: All matching networks use weighted soft-margin triplet loss with Adam optimization, dropout 0.5, and exhaustive minibatch triplet construction.Batch sizes are B = 30 for two-stream networks and B = 24 for joint feature learning networks.

5. Results

Experiments on CVUSA show that joint feature learning and feature fusion improve cross-view retrieval over baselines and prior state-of-the-art methods. Ablations and OP-dataset evaluation further examine multi-scale features, pooling, weight sharing, and urban geo-localization performance.

  • Results of Our Approach: Synthesized aerial queries achieve 72.58% top-1% recall, below the 82.94% obtained with ground queries, motivating their use alongside real ground images.The synthesized images capture substantial panorama information but are not fully dependable as standalone queries.
  • Results of Our Approach: Joint feature learning improves top-1% retrieval accuracy by about 9% over the two-stream ground–aerial baseline.Training uses image triads, while testing uses only ground–aerial pairs.
  • Results of Our Approach: Feature Fusion adds 3.89% top-1% accuracy over joint feature learning and about 19% top-1 accuracy over it.The method fuses synthesized-aerial and ground-image features into the query representation.
  • Comparison to Existing Methods: Feature Fusion achieves a 4.58% top-1% recall improvement over SOTA CVM-Net and increases top-1 and top-10 accuracy by more than 26% and 18% over CVM-Net-I.The proposed variants outperform existing state-of-the-art approaches for all plotted K values from 1 to 80.
  • Ablation Study: Multi-scale features outperform single-scale features, while global average pooling severely harms retrieval by removing spatial information.Features from conv 6 and conv 7 contribute alongside final-layer conv 8 features.
  • Ablation Study: Removing weight sharing improves top-1 accuracy by about 18%, supporting separate encoders for aerial and ground viewpoints.The ablation attributes this choice to the drastically different representations produced by the two views.

6. Conclusion

The paper presents a practical cross-view retrieval approach that transforms ground queries into the target aerial view and fuses synthesized-aerial and ground features. This approach bridges the aerial–street-view domain gap and improves performance on CVUSA.

  • The method transforms a ground query into the aerial target view to obtain better scene understanding.
  • Joint feature training incorporates synthesized aerial images into the cross-view matching pipeline.
  • Fusing ground features with corresponding synthesized aerial features produces a robust ground-image descriptor.
  • The approach obtains significant improvements over state-of-the-art methods on the challenging CVUSA dataset.

Supplementary Materials

The supplementary materials analyze feature representations, failure cases, and the reverse aerial-to-ground matching problem. They use t-SNE visualizations and qualitative retrieval examples to examine the proposed method.

  • The supplementary material provides an in-depth analysis of the proposed cross-view matching method on CVUSA.
  • It visualizes two-dimensional embeddings from the two-stream baseline and feature fusion representations using t-SNE.
  • It includes qualitative failure examples for the proposed method.
  • It also examines aerial-to-ground image matching using the proposed joint approach.

1. Visualization and Interpretation of Features

Feature fusion changes the embedding structure: corresponding ground and aerial features overlap more, while different image samples become more separated. This supports the proposed method’s role in bridging the cross-view domain gap.

  • The visualization uses 100 CVUSA test images and projects 1000-dimensional image features into two-dimensional embeddings.
  • In the two-stream baseline, corresponding aerial and ground features show less overlap, while features from different pairs can lie close together.
  • Feature fusion yields higher overlap between corresponding ground–aerial pairs and greater separation between different image samples.
  • The synthesized aerial images bring aerial and ground representations closer and improve matching accuracy.

2. Failure Examples

The failure cases identify mismatches caused by missing visual cues, temporal differences, occlusion, and distant large buildings. In one case, the ground truth is retrieved at position 1700.

  • Missing water information in the ground image makes the aerial match challenging, placing the ground truth at position 13.
  • Different capture times create visual differences that can make a visually similar aerial image outrank the ground truth.
  • Tree occlusion hides houses from the street view, and the synthesized aerial image also omits those houses.
  • A distant building can occupy a large region in the aerial image while remaining difficult to interpret from the street view.
  • The method retrieves the ground-truth image at position 1700 in the large-building failure case.

3. Aerial-to-Ground Image Matching

The paper extends its approach to aerial-to-ground matching by synthesizing ground panoramas from aerial queries and applying joint feature learning and feature fusion. Results and qualitative visualizations support matching in both directions.

  • Aerial-to-ground matching uses an aerial image as the query and synthesizes a ground-level panorama before applying joint feature learning and feature fusion.
  • Top-1, top-10, and top-1% accuracies are reported for two-stream baselines, joint feature learning, and feature fusion methods.
  • Top-K recall from K = 1 to 80 compares the proposed method with baselines and supports applying the method in both matching directions.
  • Qualitative results visualize the aerial query, synthesized ground panorama, three closest matches, and ground-truth panorama.
  • The synthesized ground panoramas successfully transform semantic information from the aerial domain to the ground domain.

4. OP Dataset

The OP dataset addresses missing geo-information and limited urban coverage in existing public cross-view matching datasets. It contains cross-view image pairs from urban areas of Orlando and Pittsburgh, with qualitative retrieval examples showing the proposed method's matching behavior.

  • Existing public cross-view matching datasets lack geo-information, while CVUSA images primarily represent rural areas with land and vegetation.
  • The authors collect a new cross-view image-pair dataset covering urban areas of Orlando and Pittsburgh.
  • Despite visually similar aerial images, feature fusion retrieves the ground-truth aerial image as the closest matching image.
  • OP qualitative examples show aerial images, ground queries, ground-truth aerial images, and distances in meters from each aerial image to the query.
Loading 1904.11045v2…