PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Cross-view Image Retrieval: Highly Accurate Vehicle Localization Using Satellite Image

Yujiao Shi, Hongdong Li

arXiv:2204.04752v2cs.CV

TL;DR

Cross-view camera localization seeks a ground camera pose by matching its image to an overhead satellite map, but retrieval estimates are limited by image sampling density. The paper instead aligns projected satellite features with observed ground features through pose optimization; it reports effective lateral refinement despite longitudinal ambiguity.

  • Problem

    Retrieval-based cross-view localization assigns the query the pose of a matching satellite patch, making accuracy dependent on the database images’ sampling density.

    Source: §1 · p. 2

  • Method

    The method projects satellite features into the ground-view domain and uses differentiable LM optimization to refine the camera pose by matching them to ground features.

    Source: §1 · p. 2, §7 · p. 8

  • Results

    Despite longitudinal ambiguity, the method still refines lateral poses effectively.

    Source: §6.1 · p. 6

  • Takeaways & Limitations

    City-scale retrieval can provide an initial camera estimate that this method then refines to higher accuracy.

    Source: §7 · p. 8

    The S2GP neural-network ablation was infeasible because per-update parameter Jacobians exceed the capacity of existing 12/24G GPUs.

    Source: §6.2.1 · p. 7

Abstract

from arXiv · show

This paper addresses the problem of vehicle-mounted camera localization by matching a ground-level image with an overhead-view satellite map. Existing methods often treat this problem as cross-view image retrieval, and use learned deep features to match the ground-level query image to a partition (eg, a small patch) of the satellite map. By these methods, the localization accuracy is limited by the partitioning density of the satellite map (often in the order of tens meters). Departing from the conventional wisdom of image retrieval, this paper presents a novel solution that can achieve highly-accurate localization. The key idea is to formulate the task as pose estimation and solve it by neural-net based optimization. Specifically, we design a two-branch {CNN} to extract robust features from the ground and satellite images, respectively. To bridge the vast cross-view domain gap, we resort to a Geometry Projection module that projects features from the satellite map to the ground-view, based on a relative camera pose. Aiming to minimize the differences between the projected features and the observed features, we employ a differentiable Levenberg-Marquardt ({LM}) module to search for the optimal camera pose iteratively. The entire pipeline is differentiable and runs end-to-end. Extensive experiments on standard autonomous vehicle localization datasets have confirmed the superiority of the proposed method. Notably, e.g., starting from a coarse estimate of camera location within a wide region of 40m x 40m, with an 80% likelihood our method quickly reduces the lateral location error to be within 5m on a new KITTI cross-view dataset.

1. Introduction

Cross-view localization matches a vehicle-mounted camera image to a satellite map to estimate the camera pose. The paper replaces patch retrieval with feature-based pose optimization, addressing retrieval’s dependence on satellite-image sampling density.

1. Introduction

  • Existing retrieval methods assign the query the GPS tag of the most similar satellite patch, so their accuracy is limited by database-image sampling density.A prior displacement-regression approach also struggles with accurate regression across the cross-view domain gap.Source: §1 · p. 2
  • The proposed method uses two CNNs for ground and satellite features, projects satellite features to a ground viewpoint, and refines pose with differentiable Levenberg–Marquardt optimization.The optimizer seeks a pose that makes predicted satellite-derived features match observed ground-view features.Source: §1 · p. 2
  • The evaluation uses KITTI and Ford multi-AV, supplemented with high-definition satellite maps from Google Map.The datasets provide vehicle-mounted ground images and ground-truth poses, but not satellite maps.Source: §1 · p. 2

2. Prior Arts

Prior work localizes images through retrieval, 3D scene models, or SLAM/visual odometry. The paper positions satellite-based pose refinement as an alternative when 3D scene models are unavailable and as a possible complement to SLAM/VO.

Image based localization

  • Satellite-image retrieval remains effective at scale, but assigning the pose of the top-ranked reference image yields coarse estimates.Ground-to-ground matching also cannot localize queries without available reference counterparts.Source: §2 · p. 2

3D structure based localization

  • 3D-structure methods use scene models; the cited LM approaches require 3D keypoint coordinates and address ground-to-ground localization, unlike this paper’s satellite-only reference setting.The paper targets ground-to-satellite localization when 3D scene models are unavailable.Source: §2 · p. 2

SLAM/VO

  • SLAM/VO integrates relative poses across frames and can accumulate drift, whereas the proposed method relies on a single frame and may complement it as satellite-image loop closure.The comparison describes a possible complementary role, not a reported integrated system.Source: §2 · p. 2

3. Method Overview

Given a coarse ground-camera pose, the method aligns ground and satellite features and optimizes pose rather than retrieving a satellite patch. Its three components are a two-branch feature network, Geometry Projection, and a differentiable LM optimizer.

3. Method Overview

  • The framework combines a two-branch feature network, Geometry Projection, and a differentiable Levenberg–Marquardt optimizer.It starts from a coarse ground-camera pose and refines that pose against a companion satellite map.Source: §3 · p. 2

3.1. Deep robust feature learning

  • A two-branch U-net extracts multi-scale deep features separately from ground and satellite images.The learned features are intended to be robust to viewpoint changes and suitable for visual localization.Source: §3.1 · p. 2

3.2. Cross-view feature alignment

  • Geometry Projection uses cross-view geometry and relative camera pose to project satellite features into the ground-view domain.This aligns the two views’ features in the ground-view domain.Source: §3.2 · p. 2

3.3. Iterative pose refinement

  • The LM optimizer iteratively adjusts camera pose to match projected satellite features with observed ground features.A coarse-to-fine multi-scale update mitigates local minima when the initial pose is far from ground truth: coarser features support global search, while finer features support precise refinement.Source: §3.3 · p. 3, Figure 2 · p. 3

4. Highly-accurate Pose Optimization

The method estimates camera pose by comparing multi-scale ground and satellite features, using pose-conditioned geometry projection and differentiable Levenberg–Marquardt optimization. Its ground-plane projection is approximate, and pose errors train the feature network end-to-end.

4. Highly-accurate Pose Optimization

  • Separate ground and satellite CNN branches produce L2-normalized features at three scales, with matching architectures but unshared weights.Source: §4 · p. 3

4.1. Satellite-to-ground geometry projection

  • Satellite features are projected into ground-view features using the estimated pose and a ground-plane homography, an approximation whose feature-level comparison is less sensitive to object height.Source: §4.1 · pp. 3–4

4.2. Multi-level LM optimization

  • At each feature level, LM minimizes squared feature residuals in a coarse-to-fine sequence that continues until convergence or a maximum of five iterations.Source: §4.2 · p. 4
  • A confidence map intended to weight salient features was omitted because it did not improve results consistently across test sets.Source: §4.2 · p. 4

4.3. Training objective

  • Ground-truth pose supervision sums rotation and translation L1 errors across iterations and levels; optimization errors backpropagate to train features end-to-end.Source: §4.3 · p. 4

5. Satellite-augmented KITTI and Ford Multi-AV dataset

The evaluation augments KITTI and Ford multi-AV with satellite maps and defines dataset splits and directional pose metrics. It also highlights that longitudinal location is ambiguous in some scenes, including highways.

Satellite map augmentation

  • Satellite maps were added to calibrated KITTI and Ford multi-AV data, using zoom 18 imagery with 0.20 m per pixel for KITTI and 0.22 m for Ford.Source: §5 · p. 4

KITTI

  • KITTI is divided into Training, Test1 from the same region, and Test2 from a different area to evaluate generalization.Source: §5 · p. 4

Evaluation Metrics

  • Longitudinal localization is ambiguous when facades are absent from overhead imagery or highway scenes are repetitive, while lateral position can be determined more reliably.Source: Figure 3 · p. 5, §5 · p. 5

Benchmark comparisons

  • The reported benchmark tables compare methods on KITTI and compare image-retrieval approaches on Ford multi-AV.Source: Table 1 · p. 5, Table 2 · p. 5

Ford multi-AV dataset

  • Ford evaluation uses V2’s front-left camera across six trajectories and two drives, with training and testing split by drive.Source: §5 · p. 5

Evaluation Metrics

  • The study evaluates 3-DoF pose using separate longitudinal and lateral location errors and thresholds of 1, 3, and 5 m, plus rotation thresholds of 1°, 3°, and 5°.Source: §5 · p. 5

6. Experiments

The experiments compare pose optimization with fine-grained retrieval and examine the geometry-projection and optimization components. They report stronger localization from pose optimization and advantages for satellite-to-ground projection and LM optimization in the tested settings.

6. Experiments

  • The experiments refine a coarse location within a 40 × 40 m region and compare 15 pose-search solutions with a 4 × 4 fine-grained retrieval grid.The grid size was selected for a fair comparison with the pose search.Source: §6 · p. 6, §6.1 · p. 6

6.1. Comparison with fine-grained image retrieval

  • Fine-grained retrieval baselines perform poorly for high-accuracy localization; VIGOR improves on them, while pose optimization significantly outperforms fine-grained retrieval.Source: §6.1 · p. 6
  • With known orientation, performance consistently improves; despite longitudinal ambiguity, the method refines lateral poses, while Fig. 5 visualizes intermediate updates across regions.Source: §6.1 · pp. 6–7, Figure 4 · p. 6, Figure 5 · p. 7

6.2.1 Effectiveness of GP geometry projection

  • S2GP outperforms G2SP, and homography outperforms polar transformation; the authors suggest polar transformation is less suited to pinhole-camera images.Source: §6.2.1 · p. 7
  • The NN mapping ablation was tested only in G2SP because S2GP loses information; NN performs better on rotation but has limited translation optimization.Regenerating NN maps at each LM step would require Jacobians beyond 12/24G GPU capacity; the authors favor explicit geometry when available.Source: §6.2.1 · pp. 7–8

6.2.2 Superiority of LM optimization

  • LM significantly outperforms SGD and ADAM; the network-based optimizer performs better on rotation but worse on translation optimization than LM.The authors attribute LM’s advantage to adaptive second-order optimization and describe SGD’s zigzagging as slow to converge.Source: §6.2.2 · p. 8

7. Conclusion

The paper frames accurate cross-view localization as iterative pose optimization, refining an initial estimate by aligning ground and satellite features. It also describes possible roles alongside retrieval and SLAM/VO, while leaving depth-based improvement for future work.

7. Conclusion

  • Geometry Projection aligns satellite and ground-view features, while differentiable Levenberg–Marquardt optimization progressively refines the relative camera pose end to end.This approach departs from retrieval-based localization to address the cross-view domain gap.Source: §7 · p. 8
  • City-scale place retrieval can provide an initial camera estimate, which the proposed method then refines to higher accuracy.Source: §7 · p. 8
  • The method may complement SLAM/VO as a loop-closure mechanism; combining it with VO may resolve longitudinal ambiguity, while depth-based improvement remains future work.Source: §7 · p. 8

A. Training and Testing Splits of the KITTI and the Ford Multi-AV dataset

The supplementary evaluation filters noisy GPS-tagged examples before training and testing. The authors report improved performance from filtering and provide dataset splits and results for additional Ford logs.

A. Training and Testing Splits of the KITTI and the Ford Multi-AV dataset

B. Increasing the Grid Sample Density for Image Retrieval-based Methods

The supplementary experiments test whether denser sampling improves retrieval-based localization. They report no consistent benefit and point to the difficulty of distinguishing already similar database images.

B. Increasing the Grid Sample Density for Image Retrieval-based Methods

C. Different Initial Values

The supplementary material examines search-range and ablation results, specifies the local pose coordinates, and notes a broader-impact privacy risk.

E. Coordinates Illustration and Pose Parameterization

  • The local pose uses lateral translation Δx, longitudinal translation Δz, and azimuth θ, with the world coordinate system set at the initial camera estimate.The coordinate illustration is only illustrative; the code uses slightly different coordinates.Source: E. Coordinates Illustration and Pose… · p. 14, Figure 9 · p. 14

C. Different Initial Values

D. Additional Comparisons

  • Removing the longitudinal loss hurts feature discrimination, while the C2F update strategy is said to explore more solutions than C2F Global.Longitudinal ambiguity is high, but its constraints still contribute to learning discriminative features.Source: D. Additional Comparisons · p. 14, Table 11 · p. 15

F. Broader Impact

  • The authors warn that misuse of vehicle or camera localization could violate privacy because those positions are sensitive information.They advocate careful data protection and model management.Source: F. Broader Impact · p. 15
Loading 2204.04752v2…