Source-linked AI summary
CIPER: A Unified Framework for Cross-view Image-retrieval and Pose-estimation
Yurim Jeon, Dongseong Seo, Seung-Woo Seo
TL;DR
Cross-view geo-localization traditionally separates broad retrieval from precise pose estimation, leaving a need for unified city-scale search and 3-DoF localization. CIPER combines both tasks through a shared transformer encoder, task-specific tokens, bidirectional cross-attention, and set prediction, achieving competitive performance on multiple datasets, particularly under limited fields of view and arbitrary orientations.
Problem
Existing cross-view geo-localization methods separately target wide-area retrieval or precise pose estimation, while practical localization requires both capabilities.
Method
CIPER is a unified transformer architecture with a shared encoder, task-specific retrieval and pose tokens, a two-way pose decoder, and set prediction for joint retrieval and 3-DoF estimation.
Results
CIPER achieves reliable and competitive performance on VIGOR, KITTI, and Ford Multi-AV, especially under limited fields of view and arbitrary orientations.
Takeaways & Limitations
CIPER provides a robust unified baseline for cross-view geo-localization in challenging real-world conditions.
Abstract
from arXiv · showhide
Cross-view geo-localization estimates the geographic location of a ground image by matching it against an aerial image database. Existing methods tackle this through either large-scale retrieval or precise pose estimation, but not both: retrieval-based methods enable wide-area search at the cost of localization accuracy, while pose estimation methods achieve high precision within only a narrow search space. Naively cascading these pipelines introduces error propagation and inconsistent feature representations. We formulate cross-view geo-localization as a unified problem requiring simultaneous city-scale retrieval and precise 3-DoF pose estimation. We propose CIPER (Cross-view Image-retrieval and Pose-estimation transformER), a single architecture that jointly performs both tasks through mutually beneficial feature learning. CIPER uses a shared transformer encoder with task-specific tokens to disentangle global retrieval features from spatial localization cues. To bridge the large domain gap between ground and aerial views, we introduce a two-way transformer pose decoder that uses ground features as spatial queries for bidirectional cross-attention. A set prediction strategy further enables stable 3-DoF regression under a unified multi-task objective. Experiments on VIGOR, KITTI, and Ford Multi-AV demonstrate competitive performance, especially under limited field-of-view and arbitrary orientation conditions. Code is available at https://github.com/yurimjeon1892/CIPER.
1 Introduction
Cross-view geo-localization must combine city-scale retrieval with precise 3-DoF pose estimation, rather than treating them as separate stages. CIPER addresses this need with shared task-specific features and reports strong performance under challenging viewpoint conditions.
- Cross-view geo-localization estimates a ground image’s location from a reference database of aerial images, supporting GPS-denied navigation and autonomous driving.
- Retrieval offers broad search coverage but depends on aerial-image sampling frequency, whereas pose estimation provides precision within a narrower search space.
- A two-way transformer pose decoder uses bidirectional cross-attention to align ground and aerial features, while set prediction supports stable direct 3-DoF regression.
- Experiments on VIGOR, KITTI, and Ford Multi-AV show reliable competitive performance, especially with limited fields of view and arbitrary orientations.
- CIPER reframes the task as simultaneous city-scale image retrieval and precise 3-DoF pose estimation within one unified framework.
- A shared transformer encoder with class and pose tokens learns global retrieval descriptors alongside localization-sensitive spatial features.
2 Related Work
Related work develops separate retrieval and pose-estimation paradigms for cross-view geo-localization. Practical deployment motivates integrating broad search with accurate 3-DoF localization without redundant processing.
- Image Retrieval: Cross-view image retrieval matches a ground query with aerial database images using descriptors and similarity computation.
- Image Retrieval: Methods reduce ground–aerial domain differences through spatial-aware attention, image transformation, or cross-view image synthesis.
- Image Retrieval: VIGOR and TransGeo address retrieval when viewpoints and image centers are misaligned, including arbitrary query poses on reference aerial images.
- Pose Estimation: Cross-view pose estimation predicts the 3-DoF offset and orientation between the ground viewpoint and aerial-image center.
- Pose Estimation: Existing pose methods include dense probability estimation, geometric projection with iterative optimization, and directional feature aggregation for limited-field-of-view images.
- Unified Localization: The related paradigms motivate an end-to-end approach that combines wide search with high localization accuracy instead of cascading disjoint models.
3 Methods
CIPER formulates cross-view geo-localization as joint retrieval and precise 3-DoF pose estimation. Its shared transformer separates global retrieval descriptors from localization features and uses bidirectional cross-attention with set prediction for pose regression.
- Network overview: CIPER jointly performs aerial-image retrieval and precise 3-DoF pose estimation through one end-to-end network.
- Encoder: Full self-attention supports global interactions among image patches, which helps model non-local and geometrically distorted ground–aerial correspondences.
- Encoder: The ViT encoder uses class and pose tokens alongside patch embeddings to produce task-specialized global and spatial representations.
- Pose decoder: The two-way pose decoder alternates token-to-image and image-to-token cross-attention to align ground queries with aerial spatial embeddings.
- Pose decoder: Set prediction processes 64 learnable queries, each producing a candidate 3-DoF pose and confidence score, with the highest-confidence candidate selected.
- Loss function: The unified objective combines triplet loss for retrieval with BCE and MSE losses for pose estimation, using λcls = 0.2.
4 Experiments
Experiments evaluate CIPER across VIGOR, KITTI, and Ford multi-AV using datasets with different geographic splits and fields of view. The VIGOR visualization illustrates retrieval of aerial candidates and subsequent pose-based localization.
- Datasets: VIGOR contains 105,214 ground panoramas and 90,618 aerial images from four U.S. cities, with same-area and cross-area train/validation splits.
- Datasets: KITTI experiments use 19,655 training, 3,773 test1, and 7,542 test2 samples, with ±47° ground-image fields of view.
- Datasets: Ford multi-AV experiments use Log1 and Log2, totaling separate training and testing sets with ±40° ground-image fields of view.
- Visualization: The VIGOR pipeline retrieves the top five aerial candidates, maps their locations, estimates pose from the highest-similarity candidate, and combines both outputs.
- Implementation: Training uses 256 × 1024 ground images, 512 × 512 aerial images, batch size 12, AdamW, and a 0.0001 learning rate and weight decay.
4.2 Computational Efficiency Analysis
The efficiency analysis compares conventional separate retrieval and pose-estimation networks with CIPER’s simultaneous unified architecture. Sharing the backbone and intermediate representations avoids redundant computation while supporting both tasks.
- The conventional two-stage pipeline independently processes the same visual inputs, causing redundant feature extraction and increased computational cost.
- CIPER performs retrieval and pose estimation simultaneously in one transformer architecture, rather than summing the computational costs of two networks.
4.3 Visualization of Unified Cross-view Geo-localization
The VIGOR visualization shows how retrieval candidates and pose estimation are combined into a final geo-localization result. Under aligned ground and aerial orientations, the reported pose contains latitude and longitude.
- Visualization: The visualization begins with a ground query and top-five aerial retrieval candidates, then displays candidate locations, estimated pose, and the combined final result.
- Quantitative comparisons: Tables 2–4 compare retrieval and pose-estimation performance on VIGOR, KITTI, and Ford multi-AV, with red and blue marking best and second-best results where stated.
- Candidate analysis: The top-five candidates are closely located on the map, indicating that pose estimation can be applied beyond only the top-1 retrieval candidate.
- Evaluation setting: VIGOR experiments assume aligned ground and aerial orientations, so localization results are expressed only as latitude and longitude.
4.4 Experiments on Cross-view Image Retrieval
CIPER is evaluated for cross-view image retrieval on VIGOR using top-k recall, where success requires including a ground-truth aerial image among the highest-scoring predictions. It outperforms competing methods at larger-k metrics while remaining comparable at r@1.
- Results: CIPER outperformed state-of-the-art methods on r@5, r@10, and r@1% retrieval metrics.These results indicate effective candidate search in a city-scale aerial image database.
- Results: CIPER achieved performance comparable to other methods at r@1.
4.5 Experiments on Cross-view Pose Estimation
CIPER is evaluated for 3-DoF pose estimation on KITTI and Ford Multi-AV under varied rotation priors, translations, sensor configurations, and environmental conditions. It is especially robust when orientation is unconstrained and produces poses close to ground truth even under aerial-view occlusion.
- Experimental setup: Pose estimation is evaluated on KITTI and Ford Multi-AV with random aerial rotations within each prior and translations of ±20m.
- Metrics: Lower mean and median location and orientation errors indicate better pose estimation, alongside recall thresholds for positional and angular accuracy.
- Quantitative results: CIPER achieves the best or second-best performance across most settings and remains robust across sensor configurations, offsets, rotations, and environments.
- Orientation robustness: With an orientation prior of ±180°, CIPER significantly outperforms all baselines, whereas its gap is minor under a ±10° prior.
- Qualitative results: CIPER estimates poses closest to ground truth in both location and orientation, including a bridge scene where aerial visibility is occluded.
4.6 Ablation Study
The ablation study examines dual-token disentanglement and the bidirectional pose decoder. Token swapping reveals task specialization, while two-way cross-attention remains more stable than one-way attention under large orientation uncertainty.
- Study design: The ablation study evaluates transformer token configuration and pose-decoder directionality on VIGOR and KITTI.
- Token disentanglement: Swapping the class and pose tokens causes a small but consistent performance gap, indicating specialized retrieval and pose representations.Retrieval uses translation-invariant semantics, whereas pose estimation requires translation-variant spatial structure.
- Token disentanglement: Separating token roles alleviates the multi-task bottleneck associated with relying on a single representation.
- Pose decoder: Under a 180° orientation prior, the one-way decoder degrades substantially, while the two-way decoder remains stable and consistently outperforms it.The advantage holds in both same-area and cross-area evaluations.
5 Conclusion
CIPER unifies cross-view image retrieval and precise pose estimation in an end-to-end transformer network. Its shared dual-token encoder and reciprocal cross-attention decoder reduce redundant extraction and support robust 3-DoF localization in challenging orientations and fields of view.
- Conclusion: CIPER is an end-to-end transformer network that jointly performs cross-view image retrieval and pose estimation.
- Conclusion: Its dual-token shared encoder and reciprocal cross-attention decoder mitigate redundant feature extraction in cascaded pipelines.
- Conclusion: CIPER achieves robust accuracy under arbitrary orientations and limited fields of view, supporting potential use in GPS-denied autonomous navigation.