Source-linked AI summary
Lending Orientation to Neural Networks for Cross-view Geo-localization
Liu Liu, Hongdong Li
TL;DR
Ground-to-aerial image-based localization must recover a ground query’s location despite major viewpoint and appearance differences. The paper adds per-pixel orientation maps to a Siamese CNN, reports improved localization performance and a more compact network, and introduces the larger CVACT benchmark. Its orientation parameterization assumes calibrated panoramas and a centered observer relative to the satellite view.
Problem
Ground-to-aerial localization must recover a ground image’s geospatial location from geo-referenced aerial imagery despite drastic viewpoint and appearance differences.
Method
A Siamese CNN explicitly encodes per-pixel orientation for ground panoramas and satellite images, adding orientation maps alongside image appearance.
Results
The method establishes state-of-the-art ground-to-aerial geolocalization performance, including more than 25% higher top 1% recall than without orientations.
Takeaways & Limitations
Orientation maps provide a compact, transparent addition that can also be plugged into other deep-learning frameworks or applications.
Takeaways & Limitations
The orientation parameterization assumes an intrinsically calibrated panorama and an observer standing at the satellite-view center.
Abstract
from arXiv · showhide
This paper studies image-based geo-localization (IBL) problem using ground-to-aerial cross-view matching. The goal is to predict the spatial location of a ground-level query image by matching it to a large geotagged aerial image database (e.g., satellite imagery). This is a challenging task due to the drastic differences in their viewpoints and visual appearances. Existing deep learning methods for this problem have been focused on maximizing feature similarity between spatially close-by image pairs, while minimizing other images pairs which are far apart. They do so by deep feature embedding based on visual appearance in those ground-and-aerial images. However, in everyday life, humans commonly use {\em orientation} information as an important cue for the task of spatial localization. Inspired by this insight, this paper proposes a novel method which endows deep neural networks with the `commonsense' of orientation. Given a ground-level spherical panoramic image as query input (and a large georeferenced satellite image database), we design a Siamese network which explicitly encodes the orientation (i.e., spherical directions) of each pixel of the images. Our method significantly boosts the discriminative power of the learned deep features, leading to a much higher recall and precision outperforming all previous methods. Our network is also more compact using only 1/5th number of parameters than a previously best-performing network. To evaluate the generalization of our method, we also created a large-scale cross-view localization benchmark containing 100K geotagged ground-aerial pairs covering a city. Our codes and datasets are available at \url{https://github.com/Liumouliu/OriCNN}.
1 Australian National University, Canberra, Australia 2 Australian Centre for Robotic Vision
The section provides contact information for the listed affiliations.
- The contact string lists Liu.Liu and hongdong.li.
- Both listed contacts use the anu.edu.au email domain.
- The contact information is formatted as two addresses separated by a semicolon.
1. Introduction
The paper addresses ground-to-aerial image-based localization, where severe viewpoint and appearance differences make matching difficult. It proposes encoding orientation in a CNN to improve localization and introduces the larger CVACT benchmark.
- Ground-to-aerial localization estimates a ground image’s absolute geospatial location by matching it against geo-referenced aerial imagery.
- The task is challenging because ground and aerial images have an ultra-wide baseline and drastically different visual appearances.
- Human localization with maps motivates using orientation, particularly identifying geographic True North in both ground surroundings and aerial imagery.
- The proposed network adds per-pixel orientation maps as CNN signal channels to learn features that combine appearance, location discrimination, and orientation selectivity.
- The work contributes a Siamese CNN, state-of-the-art ground-to-aerial performance, and the larger geotagged CVACT benchmark dataset.
2. Related Works
Prior cross-view localization work increasingly relies on deep CNN feature embeddings, while directional information and efficient cross-view orientation handling remain less developed.
- Modern cross-view localization methods generally use deep CNNs to learn discriminative features from imagery.
- Earlier work explored classification, hybrid, Siamese, and triplet CNN architectures, including soft-margin triplet loss for cross-view matching.
- Vo et al. estimated ground-view orientation and tested multiple aerial orientations, introducing substantial training and testing overhead.
- Other work studied transferring information between ground and aerial views using deep CNNs and conditional generative adversarial networks.
- Directional information has received limited neural-network treatment, including complex-valued neurons that encode one-dimensional direction through phase.
3. Method Overview
The method uses a two-branch Siamese CNN for ground-to-aerial matching and augments appearance with explicit per-pixel orientation geometry. Ground panoramas use spherical angles, while satellite images use polar coordinates encoded as extra channels.
- Method Overview: Deep cross-view localization embeds matching pairs closely while separating non-matchable pairs in feature space.
- Method Overview: The seven-layer Siamese CNN uses separate branch weights because ground and aerial imagery come from different domains and modalities.
- 3.1. Siamese Network Architecture: The baseline network receives a ground-level panorama and a satellite image, learning features by minimizing triplet loss.
- 3.2. Use of Orientation Information: Orientation is added because prior methods emphasize visual appearance while overlooking directional information that can simplify localization.
- 3.2. Use of Orientation Information: The method teaches networks geographic orientation through an efficient per-pixel orientation representation.
- 3.3. Representing Orientation Information: Ground panoramas represent each pixel with azimuth and altitude, while satellite pixels use azimuth and radial range under stated camera and viewpoint assumptions.
- 3.3. Representing Orientation Information: Hue and saturation encode the two orientation parameters as U-V channels appended to the RGB image.
4. Joint Image and Orientation Embedding
The method jointly embeds cross-view images with orientation information in a Siamese network, using multi-scale feature aggregation, generalized-mean pooling, and metric learning.
- Where to inject orientation information?: The Siamese network incorporates orientation maps alongside cross-view images to learn joint feature embeddings.Scheme-I concatenates image and orientation channels at the input, while Scheme-II additionally injects orientation into intermediate convolutional blocks.
- Where to inject orientation information?: Scheme-I injects orientation only at the input, whereas Scheme-II injects it into all seven CNN layers.The two designs are illustrated as alternative orientation-injection schemes.
- Deep feature embedding: Feature maps from the last three layers are resized and concatenated along the feature dimension to form multi-scale tensors X_i.For i ∈ {g, s}, each tensor has dimensions W_i × H_i × D.
- Deep feature embedding: Generalized-mean pooling converts each tensor X_i into a compact embedding vector f_i despite differing cross-view image sizes.The embedding dimension is set to D = 1536, and the resulting features are normalized to unit L2-norm.
- Triplet loss for cross-view metric learning: Triplet-style cross-view metric learning pulls matchable ground-satellite embeddings together and separates non-matchable examples by an empirical margin α.The embeddings are compared using L2 metric distances in a shared space.
5. Experiments
Experiments show that adding orientation information substantially improves cross-view localization, while preserving robustness and compactness across standard and city-scale benchmarks.
- Effect of Orientation Map: The RGB baseline achieves 9.8%, 23.6%, 32.6%, and 68.6% recall at top-1, top-5, top-10, and top-1%, respectively.All ground panoramas are aligned to north for this baseline.
- Effect of Orientation Map: Scheme-1 raises CVUSA recall to 31.7% at top-1, 56.6% at top-5, 67.5% at top-10, and 93.1% at top-1%.The scheme augments RGB input with UV orientation channels.
- Comparisons with Other Methods: Compared with CVM-Net, the method improves recall@top 1% by 1.65% and recall@top-1 by 12.91%.The network uses 30 million parameters and 368MB, versus CVM-Net’s 160 million parameters and 2.0GB.
- Detailed analyses of the proposed network: The learned embeddings place spatially nearby cross-view image pairs close together in the t-SNE visualization.This visualization is reported on CVUSA features learned by the proposed network.
- ACT city-scale cross-view dataset: On CVACT, the method outperforms CVM-Net by 15.84% at top-1, while only 19.90% of queries localize within 5m.The evaluation uses 92,802 pairs and counts a query as localized when one of the top N retrieved images lies within 5 meters.
6. Conclusion
The paper frames image-based geo-localization as a geometry problem but shows that orientation-aware learning improves localization and contributes a new benchmark dataset.
- Image-based geo-localization seeks to recover 6-DoF camera pose, including both location and orientation.
- Previous methods often treated localization as content-based retrieval focused on visual appearance and semantic image contents.
- Adding a simple orientation map taught a Siamese localization network the geometric notion of orientation.
- Over 25% top 1% recall improvement was achieved compared with not using orientations.
- The work introduced the fully annotated, geo-referenced CVACT cross-view image localization dataset as a benchmark contribution.