Source-linked AI summary
Lending Orientation to Neural Networks for Cross-view Geo-localization
Liu Liu, Hongdong Li
TL;DR
Ground-to-aerial localization seeks to recover a query image’s location by matching it against georeferenced aerial imagery, while orientation has been overlooked in recent IBL literature. This paper adds per-pixel orientation maps to a Siamese network and reports a boost of over 25% in top 1% recall rate compared with without using orientations.
Problem
Ground-to-aerial image matching aims to recover a query image’s location from georeferenced aerial imagery, while orientation has been overlooked in recent IBL literature.
Source: §1 · pp. 1–2Method
The method adds per-pixel orientation maps as CNN signal channels and jointly learns appearance and orientation geometry in a Siamese network.
Source: §1 · p. 2Results
Over 25% boost in top 1% recall rate compared with without using orientations.
Source: §6 · p. 8Takeaways & Limitations
The authors argue that geometry-related vision should consider available geometric clues rather than rely on CNNs as blind end-to-end black boxes.
Source: §6 · p. 8The approach assumes intrinsic calibration of input panoramas to estimate relative pixel angles, though a coarse estimate is stated to be adequate for IBL.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
This paper studies image-based geo-localization (IBL) problem using ground-to-aerial cross-view matching. The goal is to predict the spatial location of a ground-level query image by matching it to a large geotagged aerial image database (e.g., satellite imagery). This is a challenging task due to the drastic differences in their viewpoints and visual appearances. Existing deep learning methods for this problem have been focused on maximizing feature similarity between spatially close-by image pairs, while minimizing other images pairs which are far apart. They do so by deep feature embedding based on visual appearance in those ground-and-aerial images. However, in everyday life, humans commonly use {\em orientation} information as an important cue for the task of spatial localization. Inspired by this insight, this paper proposes a novel method which endows deep neural networks with the `commonsense' of orientation. Given a ground-level spherical panoramic image as query input (and a large georeferenced satellite image database), we design a Siamese network which explicitly encodes the orientation (i.e., spherical directions) of each pixel of the images. Our method significantly boosts the discriminative power of the learned deep features, leading to a much higher recall and precision outperforming all previous methods. Our network is also more compact using only 1/5th number of parameters than a previously best-performing network. To evaluate the generalization of our method, we also created a large-scale cross-view localization benchmark containing 100K geotagged ground-aerial pairs covering a city. Our codes and datasets are available at \url{https://github.com/Liumouliu/OriCNN}.
1. Introduction
The paper targets ground-to-aerial image-based localization and proposes a CNN that encodes per-pixel orientation to improve cross-view matching. It also introduces the CVACT benchmark of geotagged street-view panoramas and satellite images.
Introduction
- The task is to estimate a ground-level query image’s absolute geospatial location by matching it against a large database of georeferenced aerial images, without localization sensors.The query may be a spherical omnidirectional image or its panoramic representation.Source: §1 · p. 1
- Ground-to-aerial matching offers broad geographic coverage and accessibility, while ground-level street-view coverage is limited to urban areas.The paper also motivates this matching setup by analogy to a person locating themselves with a paper map and what they see around them.Source: §1 · pp. 1–2
- A per-pixel orientation map adds directional information to CNN inputs, aiming to make learned cross-view features both appearance-driven and orientation-selective.The authors frame orientation as important to human map-based localization because knowing True North in the surroundings and on the map can simplify matching.Source: §1 · p. 2
- The proposed Siamese CNN jointly learns feature embeddings from image appearance and orientation geometry for ground-to-aerial localization.The orientation encoding is described as a simple, efficient, compact way to add per-pixel directional information to CNNs.Source: §1 · p. 2
- The authors claim state-of-the-art ground-to-aerial geolocalization performance and introduce CVACT, a densely covered city-scale benchmark of geotagged panoramas and satellite images.They describe CVACT as much larger than the CVUSA panorama benchmark [36].Source: §1 · p. 2
2. Related Works
Related work covers deep-learning approaches to cross-view localization, methods for transferring between ground and aerial images, and prior efforts to encode directional information in neural networks.
Deep cross-view localization
- Deep CNN features replaced hand-crafted features in most state-of-the-art cross-view localization methods [6] [23] [34] [31].Workman et al. found Places-pretrained features outperformed hand-crafted features and later improved performance through cross-view training [34].Source: §2 · p. 2
- Vo et al. explored several CNN architectures and a soft-margin triplet loss, while their orientation-regression approach required substantial training and testing overhead [31].Hu et al. later obtained state-of-the-art performance with NetVLAD aggregation on ImageNet-pretrained VGG [5] [28]; ground-to-45-degree-view matching is easier because of greater visual overlap.Source: §2 · p. 2
Transfer between ground and aerial images
- Prior work transferred between ground and aerial images with deep CNNs, including conditional GANs, or predicted ground-view semantics from satellite imagery to synthesize panoramas [36] [25].Ground-to-satellite relationships lack a simple geometric or photometric mapping beyond coarse homography or color transfer.Source: §2 · p. 2
Lending orientation to neural networks
- Earlier work encoded one-dimensional direction with complex-valued neurons, representing angular direction as the phase of a complex number; later work developed this idea further [9] [30].The approach extends real-valued neural networks to complex-valued networks with adapted learning rules.Source: §2 · p. 2
3. Method Overview
The method adds per-pixel orientation information to a Siamese CNN for ground-to-aerial matching, complementing visual features with a geometric cue. It represents that information as orientation maps alongside the RGB images.
3.1. Siamese Network Architecture
- The baseline is a 7-layer Siamese CNN with separate ground and aerial branches, trained to match features across the two views.The branches do not share weights because ground and aerial imagery differ in domain and modality; Figure 2 specifies triplet loss [12].Source: §3.1 · p. 3, Figure 2 · p. 3
3.2. Use of Orientation Information
- Orientation can place a ground query in the satellite images’ True North reference frame, reducing the search space for faster and more accurate matching.The paper identifies efficiently teaching neural networks geographic orientation as a key contribution.Source: §3.2 · p. 3
3.3. Representing Orientation Information
- The method rejects complex-valued networks because pixel directions require two spherical angles and complex networks have high time and memory costs.It instead injects per-pixel orientation directly through orientation maps.Source: §3.3 · p. 3
- Panorama pixels use spherical azimuth and altitude, while satellite pixels use polar azimuth and radial distance from the view’s center.The panorama is assumed to cover a 360°×180° field of view and to have intrinsic calibration; the method assumes the observer is at the satellite view’s center.Source: §3.3 · p. 4
- The method encodes each pixel’s orientation in two added U-V channels: azimuth and altitude for panoramas, and azimuth and range for satellite images.The channels are represented using hue and saturation in a color-coding scheme borrowed from optical-flow visualization.Source: §3.3 · p. 4
4. Joint Image and Orientation Embedding
The section defines joint image-orientation embedding by injecting orientation maps into a Siamese network, building compact multi-scale descriptors, and training them with a cross-view ranking loss.
4.1. Where to inject orientation information?
- A seven-layer Siamese network cropped from a view-synthesis generator [25] [14] receives orientation at input only in Scheme-I or at input and every convolutional block in Scheme-II.Each layer uses convolution, leaky-ReLU, and batch normalization.Source: §4.1 · p. 4, Figure 5 · p. 4
4.2. Deep feature embedding
- Resized outputs from the last three layers are concatenated into a tensor and generalized-mean pooled into a 1,536-dimensional unit-L2 feature vector.Generalized-mean pooling is used because cross-view images can differ in size.Source: §4.2 · pp. 4–5
4.3. Triplet loss for cross-view metric learning
- The Siamese network is trained with weighted soft-margin ranking loss, pulling L2 distances for matchable cross-view pairs together and pushing non-matchable pairs apart.The overall architecture passes the learned cross-view feature vectors to a triplet loss.Source: §4.3 · p. 5, Figure 6 · p. 5
5. Experiments
Experiments test orientation-augmented cross-view matching on CVUSA and assess geographic generalization on the newly collected city-scale CVACT benchmark.
5. Experiments
- Experiments use CVUSA’s 35,532 training and 8,884 test pairs, recall at top-1 through top-1%, and random circular panorama shifts to improve robustness to orientation-estimation errors.Training uses a seven-layer Siamese network with exhaustive mini-batches and 2B(B−1) triplets per batch.Source: §5 · pp. 5–6
5.1. Effect of Orientation Map
- Scheme-1 adds UV orientation maps to RGB inputs as a five-channel signal, while Scheme-2 injects UV maps into all seven CNN layers; Scheme-1 is used thereafter and can plug into other architectures.The paper notes VGG, ResNet, U-net, and DenseNet as examples of compatible architectures.Source: §5.1 · p. 6
5.2. Comparisons with Other Methods
- Against CVM-net, our method improves recall@top 1% by +1.65% and recall@top-1 by +12.91%; it uses 30 million parameters and 368 MB versus 160 million and 2.0 GB.The paper also reports about 3 days of training on one GTX1080Ti and 30 ms average query time, about one-third of CVM-net’s.Source: §5.2 · p. 6, Table 3 · p. 6
5.3. Detailed analyses of the proposed network
- On CVUSA, t-SNE shows spatially nearby cross-view pairs embedded closely; recall@top 1% and recall@K both decline gracefully as north-orientation error rises from 0° to 20°.The error test rotates the ground panorama by a random angle to simulate north-estimation errors.Source: §5.3 · p. 7, Figure 10 · p. 7
5.4. ACT city-scale cross-view dataset
- CVACT is a GPS-tagged, city-scale benchmark densely covering Canberra, with 92,802 test pairs—10 times CVUSA’s test set.Its street-view panoramas cover 300 square miles, and satellite imagery is collected at zoom level 20.Source: §5 · p. 5, §5.4 · p. 7
- On CVACT, our method improves top-1 localization over CVM-net by 15.84%, but only 19.90% of queries localize within 5 m.The evaluation uses 92,802 non-overlapping test pairs and counts a query as localized if a top-N retrieved satellite image lies within 5 m of ground truth.Source: §5.4 · p. 7, Figure 12 · p. 8
6. Conclusion
Adding an orientation map to a Siamese localization network improves localization, and the paper contributes the fully annotated, georeferenced CVACT cross-view dataset. The conclusion argues that image-based localization is geometric and that available geometry cues should be exploited.
6. Conclusion
- Image-based geo-localization is a geometry problem whose goal is to recover 6-DoF camera pose, including location and orientation.The conclusion says orientation is a natural and desirable geometric cue for localization.Source: §6 · p. 8
- The authors say most prior methods overlook geometric cues or lack effective ways to incorporate them, instead treating localization as content-based retrieval.Those methods focus on visual similarity in image appearance and semantic content.Source: §6 · p. 8
- Top 1% recall increased by over 25% compared with the network without orientation maps.The improvement followed from adding a simple orientation map to teach the Siamese network the geometric notion of orientation.Source: §6 · p. 8
- The authors introduce CVACT, a large-scale, fully annotated, georeferenced cross-view image localization dataset.They present it as an addition to the localization benchmark and literature.Source: §6 · p. 8
- The simple, transparent orientation-map approach may extend to other deep networks or applications, and the authors urge exploiting available geometry cues rather than using CNNs as blind black boxes.They hope the idea will inspire researchers working on related problems.Source: §6 · p. 8