Source-linked AI summary
Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching
Yujiao Shi, Xin Yu, Dylan Campbell, Hongdong Li
TL;DR
Cross-view geo-localization must estimate both a ground camera’s location and azimuth, but existing methods neglect orientation alignment, increasing matching ambiguity. This paper combines polar-transformed aerial images with Dynamic Similarity Matching to estimate orientation during localization and reports significantly improved location recall across practical scenarios.
Problem
Cross-view geo-localization seeks a query camera’s geographical location and azimuth, while existing approaches treat it as location-only and neglect orientation alignment.
Method
The method applies a polar transform to aerial images and uses Dynamic Similarity Matching to correlate ground and aerial features across azimuth angles.
Results
The method significantly improves state-of-the-art performance across geolocalization scenarios, including unknown orientation and limited Field of View.
Takeaways & Limitations
Jointly recovering location and orientation enables cross-view matching when neither is known and supports disoriented ground images with limited Field of View.
Takeaways & Limitations
Orientation is treated as a 1-DoF azimuth angle, and panoramic images are used as an example for correspondence relationships extending to limited Field of View images.
Abstract
from arXiv · showhide
Cross-view geo-localization is the problem of estimating the position and orientation (latitude, longitude and azimuth angle) of a camera at ground level given a large-scale database of geo-tagged aerial (e.g., satellite) images. Existing approaches treat the task as a pure location estimation problem by learning discriminative feature descriptors, but neglect orientation alignment. It is well-recognized that knowing the orientation between ground and aerial images can significantly reduce matching ambiguity between these two views, especially when the ground-level images have a limited Field of View (FoV) instead of a full field-of-view panorama. Therefore, we design a Dynamic Similarity Matching network to estimate cross-view orientation alignment during localization. In particular, we address the cross-view domain gap by applying a polar transform to the aerial images to approximately align the images up to an unknown azimuth angle. Then, a two-stream convolutional network is used to learn deep features from the ground and polar-transformed aerial images. Finally, we obtain the orientation by computing the correlation between cross-view features, which also provides a more accurate measure of feature similarity, improving location recall. Experiments on standard datasets demonstrate that our method significantly improves state-of-the-art performance. Remarkably, we improve the top-1 location recall rate on the CVUSA dataset by a factor of 1.5x for panoramas with known orientation, by a factor of 3.3x for panoramas with unknown orientation, and by a factor of 6x for 180-degree FoV images with unknown orientation.
1. Introduction
Cross-view geo-localization must recover both a ground camera’s location and azimuth while handling viewpoint, orientation, domain, and limited-FoV differences. The proposed approach aligns aerial imagery geometrically and uses feature correlation to estimate orientation and improve location retrieval.
- Cross-view geo-localization estimates a ground camera’s latitude, longitude, and azimuth by matching it against geo-tagged aerial imagery.
- Unknown relative orientation increases localization ambiguity and search space, while limited FoV exposes only local information that may match multiple aerial images.
- Polar transformation approximately aligns aerial imagery with ground-view geometry by mapping corresponding horizontal and vertical scene structures.
- The two-stream CNN extracts spatially aware ground and aerial feature volumes, whose correspondences support cross-view matching.
- Dynamic Similarity Matching correlates ground and aerial features across azimuth angles, selects the maximum as orientation, and crops the corresponding aerial region for retrieval.The resulting feature distance provides a similarity measure for location retrieval, including limited-FoV images.
- The method jointly estimates position and orientation and reports significant improvements over state-of-the-art methods across geolocalization scenarios.
2. Related Work
Prior cross-view localization work uses deep features to address ground–aerial matching, but generally does not jointly estimate orientation. This paper instead treats location and orientation as a joint matching problem.
- Existing cross-view methods use CNN-based representations to replace hand-crafted feature matching for ground–aerial localization.
- The proposed framework combines a polar transform, a two-stream CNN, and Dynamic Similarity Matching to align domains and account for orientation.
- Prior approaches include fine-tuned, classification, hybrid, Siamese, triplet, transformation, generative, and feature-transport networks.
- Many existing methods focus on location estimation, and some require orientation as an input rather than estimating it.
- This paper jointly estimates ground-image location and orientation to facilitate cross-view matching for panoramas and limited-FoV images.
3. Location and Orientation Estimation by Cross-view Image Matching
The method jointly estimates cross-view location and orientation by reducing projection differences, preserving spatial layouts, and matching ground features against polar-transformed aerial features across azimuth offsets.
- 3.1. A Polar Transform to Bridge the Domain Gap: The polar origin corresponds to the aerial image’s geo-tag location, while 0° denotes northward direction and aligned-image upward direction.
- 3.1. A Polar Transform to Bridge the Domain Gap: Polar transformation approximately aligns aerial and ground spatial layouts, allowing CNNs to focus on feature correspondences rather than learning the geometric relationship between domains.The transform maps aerial-image geometry into a representation compatible with ground-view structure.
- 3.2. A Spatially-Aware Feature Representation: Spatially-aware feature volumes preserve scene layout information, but orientation changes make them difficult to compare, especially for limited-FoV ground images.A limited-FoV image may contain only a small sector of the aerial scene.
- 3.3. Dynamic Similarity Matching (DSM): Dynamic Similarity Matching computes cross-view feature correlation across azimuth offsets, using the maximum similarity position to estimate ground-image orientation.The ground feature acts as a sliding window over aerial features.
- 3.3. Dynamic Similarity Matching (DSM): For limited-FoV images, the method crops aerial features at the maximum-similarity position, re-normalizes them, and uses their L2 distance for matching.For panoramas, the maximum correlation is directly converted into an L2 distance from normalized features.
- 3.4. Training DSM: During training, DSM handles matching and non-matching pairs: it learns discriminative azimuth features for matches and minimizes maximum similarity for non-matches.Non-matching pairs are challenging when aligned because their similarity is larger.
4. Experiments
Experiments evaluate localization on CVUSA and CVACT, including orientation-aligned panoramas and randomly rotated, limited-FoV images. The method improves retrieval on standard and fine-grained benchmarks and estimates orientation from correctly localized queries.
- Datasets: Experiments use CVUSA and CVACT, including CVACT_test for fine-grained city-scale geo-localization.CVUSA and CVACT contain 35,532 training pairs and 8,884 testing pairs; CVACT_test contains 92,802 pairs.
- Datasets: The constructed realistic test setting randomly rotates ground images and crops them to predetermined FoVs, removing known orientation.The experiments use CVUSA and CVACT_val image pairs with varying fields of view.
- Evaluation: Top-K recall measures whether the corresponding aerial image appears among the K retrieved candidates.Retrieval ranks aerial images by L2 distance between global descriptors.
- Orientation-Aligned Panoramas: The method achieves superior results to state-of-the-art methods on the challenging CVACT_test fine-grained localization benchmark.Success is defined by retrieving a top-K aerial image within 5 meters of the query location.
- Unknown Orientation and Limited FoV: For unknown orientation and limited FoV, DSM reduces orientation-induced ambiguity and improves recall over existing methods across 180°, 90°, and 70° FoVs.Lower FoV makes ground images less discriminative, especially when orientation is unknown.
- Unknown Orientation and Limited FoV: Orientation is predicted correctly for almost all correctly localized ground images with 360° and 180° FoV.The orientation experiments evaluate ground images that were correctly localized at top-1.
5. Conclusion
The paper presents joint cross-view matching for recovering both location and orientation when neither is known. Polar transformation and Dynamic Similarity Matching support higher location recall in disoriented, limited-FoV scenarios.
- Conclusion: The proposed algorithm jointly recovers location and orientation when neither is known.It performs joint cross-view image matching.
- Conclusion: The framework combines polar transformation with a Dynamic Similarity Matching module that regresses relative orientation.These components help extract aerial features when the ground image is disoriented and has limited FoV.
- Conclusion: The method obtains higher location recall and significantly improves state-of-the-art performance across practical cross-view matching scenarios.The conclusion specifically highlights disoriented and limited-FoV settings.
A.1. Location Estimation
The method consistently outperforms state-of-the-art algorithms for unknown-orientation localization across varying FoVs. Larger test FoVs improve performance, and matching training and testing FoVs generally performs best.
- The method achieves consistently better localization performance than state-of-the-art algorithms across unknown-orientation scenarios and varying FoVs.
- Models trained with a FoV similar to the test image generally produce better localization results.The authors advise using a pretrained model with a similar FoV.
A.2. Orientation Estimation
DSM estimates ground-image orientation across 360°, 180°, 90°, and 70° FoVs, but small FoVs remain ambiguous in symmetric scenes. Multiple similarity peaks can yield incorrect orientation even when location is correct.
- DSM estimates ground-image orientations across 360°, 180°, 90°, and 70° FoVs.
- Small FoVs create high orientation ambiguity because the ground image may cover only a limited sector of the aerial scene.
- Symmetric roads or scene contents can produce two similarity peaks, making a 180° orientation error possible.
- When similar scene contents occur in multiple directions, orientation may be incorrect while the estimated location remains correct.
B. Time Efficiency
Similarity scores are visualized across azimuth angles, with the maxima indicating the estimated orientation of each ground–aerial image pair.
- Similarity curves across azimuth angles use their maxima to indicate the estimated ground-image orientation.
C. Trainable Parameters
The network outperforms recent methods while using a more compact model, supporting easier deployment.
- The network outperforms state-of-the-art methods and is more compact, facilitating deployment.