Source-linked AI summary
Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization
Aysim Toker, Qunjie Zhou, Maxim Maximov, Laura Leal-Taixé
TL;DR
Cross-view geo-localization must match street imagery with satellite imagery despite major viewpoint and appearance differences. The paper jointly learns satellite-to-street synthesis and image retrieval end to end, achieving state-of-the-art performance on CVUSA and CVACT while producing realistic street views.
Problem
Cross-view geo-localization seeks to match street views with geo-tagged satellite images, but the domains have drastic viewpoint and appearance differences and feature learning alone remains challenging.
Method
A single end-to-end architecture jointly trains satellite-to-street image synthesis and cross-view retrieval using shared learned features.
Results
The method achieves state-of-the-art cross-view geo-localization on CVUSA and CVACT and generates realistic street views without additional semantic information.
Takeaways & Limitations
Joint synthesis and retrieval promotes coherent cross-domain features while encouraging generated images to preserve scene content for localization.
Takeaways & Limitations
Feature learning alone remains an open challenge, and CVACT evaluation treats satellite matches within 5 meters as correct.
Abstract
from arXiv · showhide
The goal of cross-view image based geo-localization is to determine the location of a given street view image by matching it against a collection of geo-tagged satellite images. This task is notoriously challenging due to the drastic viewpoint and appearance differences between the two domains. We show that we can address this discrepancy explicitly by learning to synthesize realistic street views from satellite inputs. Following this observation, we propose a novel multi-task architecture in which image synthesis and retrieval are considered jointly. The rationale behind this is that we can bias our network to learn latent feature representations that are useful for retrieval if we utilize them to generate images across the two input domains. To the best of our knowledge, ours is the first approach that creates realistic street views from satellite images and localizes the corresponding query street-view simultaneously in an end-to-end manner. In our experiments, we obtain state-of-the-art performance on the CVUSA and CVACT benchmarks. Finally, we show compelling qualitative results for satellite-to-street view synthesis.
1. Introduction
Cross-view geo-localization matches street views to geo-tagged satellite imagery, offering broader coverage but facing severe viewpoint and appearance gaps. The paper addresses this by jointly synthesizing street views and retrieving matches in one end-to-end architecture.
- Street-image retrieval is limited by uneven geographic coverage, especially sparse rural imagery.
- Satellite imagery provides broad geographic coverage, but matching it with street views is difficult because the domains differ substantially in viewpoint and appearance.
- Polar transformations reduce viewpoint differences, yet transformed images still differ from real street views and omit useful facade information.
- The proposed method jointly trains satellite-to-street synthesis and cross-view retrieval within a single end-to-end network.
- The method achieves state-of-the-art geo-localization and produces realistic street views that outperform semantic-label-supervised synthesis approaches.
2. Related work
Prior work addresses the satellite–street domain gap through feature learning, viewpoint transformations, or image synthesis, but each leaves important challenges. This paper instead jointly trains synthesis and retrieval without requiring semantic maps, reporting state-of-the-art results for both tasks.
- Cross-view geo-localization remains challenging because satellite and street images differ sharply in appearance and viewpoint.
- Domain-invariant features: Feature-learning methods improve cross-view representations, but overcoming the domain gap purely through invariant features remains an open problem.
- Viewpoint transformation: Viewpoint-transformation methods explicitly modify inputs, including polar coordinates that approximately preserve satellite content but do not produce realistic street views.
- Viewpoint transformation: GAN-based synthesis creates more realistic images, but weak content preservation can hurt retrieval and often motivates using semantic maps.
- The paper jointly trains synthesis and retrieval end to end, avoiding semantic maps while achieving state-of-the-art geo-localization and cross-view generation.
3. Method
The method jointly synthesizes street views from polar-transformed satellite images and retrieves satellite matches using shared latent features in one end-to-end architecture.
- Multi-task framework: The framework jointly addresses satellite-to-street synthesis and cross-view retrieval by projecting paired images into a shared latent feature space.The two tasks are trained within one architecture so their learned representations interact during learning.
- Polar transformation: A polar coordinate transformation first rearranges satellite content so circular structures become horizontal and radial structures become vertical in the transformed image.This preprocessing produces image pairs with roughly corresponding scene-object arrangements.
- Generator and discriminator networks: The generator uses a conditional GAN to translate a polar-transformed satellite image into a photo-realistic street panorama while preserving reference-scene content.The discriminator compares generated and ground-truth street views, encouraging outputs indistinguishable from real street images.
- Retrieval network: The retrieval branch reuses generator-encoder features for satellite inputs, extracts analogous local features from street views, and aggregates them into global descriptors with spatial-aware attention.This shared encoding lets synthesis and retrieval features reinforce one another.
- Learning: Training jointly updates the generator, discriminator, and retrieval components using a composite learning objective whose components include conditional GAN and reconstruction terms.The learning procedure updates the three model components adversarially.
4. Experiments
Experiments evaluate the method on CVUSA and CVACT for geo-localization and satellite-to-street synthesis, using quantitative comparisons, qualitative examples, and ablations. The results indicate that joint training improves both tasks, while the method generates full panoramic street views without semantic-label supervision.
- Experimental setup: The experiments use the CVUSA and CVACT cross-view benchmarks, which contain 44,416 and 137,218 satellite–panoramic street-view pairs, respectively.CVUSA includes semantic labels that the method does not use; CVACT provides a large test set for examining generalization.
- Cross-view geo-localization: Geo-localization is evaluated with recall-k retrieval accuracy, including R@1, which measures the fraction of correct one-to-one matches.The experiments follow the standard evaluation protocol used in prior work.
- Cross-view geo-localization: The method outperforms prior work on CVUSA R@1 and R@5 and exceeds the best baselines on all considered CVACT retrieval metrics.The authors emphasize CVACT because it contains a large city-scale image collection.
- Ground view image synthesis: Street-view synthesis is evaluated with PSNR, SSIM, Sharpness difference, and task-specific classifier-based similarity metrics.These metrics assess pixel-level properties and similarity between generated and real street views.
- Ground view image synthesis: The method outperforms synthesis baselines that require semantic segmentation supervision, despite not using semantic-label supervision itself.On CVUSA, X-Fork and X-Seq synthesize only the first quarter of the street view, whereas the proposed method generates coherent full panoramas.
- Ground view image synthesis: On CVACT, the method produces highly plausible street views, while the considered baselines cannot be applied because semantic maps are unavailable.Qualitative experiments compare the input satellite–street pair, polar-transformed satellite image, and synthesized street view.
- Ablation Study: Ablations indicate that joint training with the retrieval branch benefits image synthesis and that multi-task learning improves both tasks through superior learned features.Removing the retrieval head reduces synthesis quality, while using only an L1 loss produces blurry images lacking high-frequency details.
5. Conclusion
The framework jointly performs satellite-to-street image synthesis and cross-view retrieval in one end-to-end architecture. It promotes a coherent latent feature space across domains, achieving strong synthesis and state-of-the-art localization on CVUSA and CVACT.
- The model integrates image synthesis and retrieval in one end-to-end trainable architecture.
- Satellite-to-street synthesis promotes a latent feature space coherent across domains, benefiting localization.
- The retrieval branch encourages generated images to faithfully depict scene content.
- The method outperforms existing cross-view synthesis approaches without additional semantic information.
- It achieves state-of-the-art cross-view geo-localization on both CVUSA and CVACT.
A. Architecture details
The architecture combines a residual U-Net generator, a modified PatchGAN discriminator, and a retrieval branch that reuses generator features. These components jointly support satellite-to-street synthesis and satellite-image matching.
- Generator: The generator is a residual U-Net that downsamples satellite inputs by a factor of 64 before residual bottleneck refinement.
- Discriminator: The discriminator is a modified PatchGAN classifier distinguishing real satellite-street pairs from pairs containing generated street images.
- Retrieval: The retrieval branch matches satellite and street panoramas using global feature encodings and reuses local satellite features from the generator encoder.
- The technical specifications provide the exact structures of the generator and discriminator modules.
- Generator: Residual downsampling, residual, and residual upsampling blocks form the generator’s principal building blocks.
B. Implementation details
The model is trained in PyTorch with Adam, fixed image resolution, horizontal-flip augmentation, and weighted retrieval, reconstruction, and adversarial losses.
- Training uses PyTorch and Adam with β1 = 0.5, β2 = 0.999, and a learning rate of 1e-4 for all three networks.
- Ground and polar-transformed satellite images use 112x616 resolution and random horizontal flips during augmentation.
- The soft margin loss uses α = 10, and network updates alternate once per cycle under standard GAN optimization.
- Generator gradients come from both discriminator and retrieval networks, with λret = 1000, λL1 = 100, and λGAN = 1.
C. Additional qualitative results
Additional experiments visualize retrieval and synthesis on CVUSA and CVACT, while ablations show that the full loss and latent representation improve generated-image realism.
- Geo-localization: Recall-k measures whether the ground-truth satellite pair appears among the first k ranked matches for a street image.
- Cross-view synthesis: CVUSA and CVACT examples provide qualitative satellite-to-street synthesis results, with CVACT including validation and large-scale test examples.
- Ablation Study: Removing parts of the full architecture can preserve retrieval relatively well while producing blurrier or less photo-realistic synthesized street views.
- Geo-localization: CVUSA visualizations show closest satellite matches for query street views, with green boxes marking ground-truth matches.
- Geo-localization: CVACT visualizations show the five closest predicted matches and ground truth, allowing multiple matches within 5 meters to count as correct.
- Ablation Study: The qualitative ablation comparisons include synthesized images from two ablations, the full method, and ground-truth street views.