Source-linked AI summary
Ground-to-Satellite Localization in Unconstrained Image Collections for 3D Scene Reconstruction
Angel Daruna, Ben Southall, Niluthpol Chowdhury Mithun, Kshitij Minhas, Nicholas Meegan, Qiao Wang, Bogdan Matei, Supun Samarasekera, Rakesh Kumar
TL;DR
Unconstrained ground-image collections lack the panoramic views, reliable priors, and temporal structure assumed by many cross-view localization methods, hindering geo-localized 3D reconstruction. The paper combines coarse-to-fine satellite matching with SfM geometric consensus and KDE-based outlier filtering. Experiments report robust localization and reconstruction benefits including metric-scale recovery, doppelgänger rejection, and merging of disjoint models.
Problem
Ground-to-satellite localization remains challenging for unconstrained collections because existing methods rely on panoramic imagery, accurate priors, or temporal continuity unavailable in many in-the-wild datasets.
Method
The framework generates coarse-to-fine pose hypotheses, then aggregates them across SfM submodels with KDE to recover geographic consensus and filter outliers.
Results
The method achieves robust geo-localization and enables metric-scale recovery, doppelgänger detection, and merging of disjoint SfM reconstructions.
Takeaways & Limitations
Satellite-referenced alignment produces more complete, geo-localized, and scale-accurate scene models than SfM alone can reliably provide.
Abstract
from arXiv · showhide
Ground image localization with respect to satellite imagery is a key enabler for metrically-accurate, geo-localized 3D scene reconstruction from unconstrained image collections. Existing cross-view localization methods have strict requirements such as panoramic imagery or known initial locations, limiting their applicability for in-the-wild reconstruction settings. We propose a robust hierarchical cross-view localization framework that leverages geometric constraints from Structure-from-Motion (SfM) models derived from unconstrained ground image collections. Our method generates coarse-to-fine pose hypotheses through a cross-view matching approach and aggregates noisy predictions across SfM model(s) using Kernel Density Estimation to recover consensus alignments while filtering outliers. Experiments demonstrate reliable localization performance from challenging image collections. Empirically we found satellite-referenced alignment enables accurate metric scale estimation, doppelgänger detection, and merging of disjoint SfM reconstructions, resulting in more complete, geo-localized site models than are possible with SfM alone.
I. INTRODUCTION
Unconstrained ground imagery makes ground-to-satellite localization difficult because limited views, heterogeneous capture, and visual doppelgängers create ambiguous matches. The proposed framework combines hierarchical matching with SfM-based consensus to support robust localization and improved reconstruction.
- Challenge: Unconstrained collections contain heterogeneous cameras, uneven coverage, limited FOV, no sequential motion, and unreliable initial poses.These conditions reduce available context for cross-view matching and increase ambiguity from geographically distinct but visually similar structures.
- Challenge: Existing methods often depend on panoramic queries or accurate location priors, limiting their reliability for unconstrained global localization.Coarse retrieval can lose accuracy with limited-FOV inputs, while fine-grained refinement requires reliable priors.
- Approach: The framework generates coarse-to-fine pose hypotheses and aggregates them across SfM clusters with KDE to filter outliers and recover consensus alignment.SfM submodels provide geometric constraints that disambiguate noisy single-image satellite matches.
- Results: Experiments show robust geo-localization on unconstrained image collections and enable metric-scale recovery, doppelgänger detection, and merging of disjoint SfM submodels.The resulting reconstructions are described as more consistent, scale-accurate, and complete than those produced with SfM alone.
II. RELATED WORKS
Prior cross-view localization methods commonly use learned ground–satellite retrieval or fine-grained pose estimation, but their assumptions can fail on limited-FOV, single-image inputs. This work instead treats predictions as proposals refined through geometric consensus.
- Method categories: Satellite imagery supplies a widely available global reference, and existing methods span coarse-grained retrieval and fine-grained pose estimation.The two categories address approximate location and orientation versus precise localization from a coarse prior.
- Coarse-grained retrieval: Coarse retrieval methods match ground queries to satellite tiles, often using Vision Transformers to learn a shared embedding space across viewpoints.Panorama-BEV Co-Retrieval, TransGeo, and Sample4Geo are cited examples.
- Fine-grained pose estimation: Fine-grained methods use dense feature correlations to align ground BEV projections with satellite orthographic images given a coarse location prior.Examples include the Correlation-Aware Homography Estimator, Dense-Flow, and FG2.
- Limitations: Most existing approaches rely on single images and often panoramic views, producing noisy and ambiguous estimates with limited-FOV cameras.The proposed alternative treats cross-view predictions as proposals rather than final outputs and refines them through geometric consensus.
B. Sequence and Multi-Image Localization.
Temporal and multi-image localization methods use continuous capture or inertial measurements to enforce relative pose constraints. The proposed approach replaces those requirements with SfM-derived geometry for sparse, unordered, heterogeneous collections.
- Existing multi-image methods: Video-based methods aggregate temporal information to smooth trajectories and filter noise, while robotics and AR methods use visual-inertial odometry for relative pose constraints.These methods exploit continuous image streams matched to satellite data.
- Unconstrained settings: Temporal continuity and inertial measurements are unavailable in unconstrained image collections, motivating a different source of multi-image geometric constraints.The paper identifies this availability gap as a limitation of these approaches.
- Proposed approach: The proposed pipeline fuses SfM relative geometry with noisy absolute ground-to-satellite cues to geo-register sparse, unordered, heterogeneous datasets.Robust integration with KDE enables accurate geo-registration without temporal continuity.
A. Hierarchical Cross-View Matching
The hierarchical matching stage estimates absolute camera poses independently, first retrieving candidate satellite locations and then refining each candidate to a precise 3-DoF pose.
- Coarse-to-fine matching: Each image is localized independently through a coarse-to-fine strategy that produces a distribution of pose hypotheses from fragmented ground observations.This stage does not require other images in the collection for the initial absolute-pose estimate.
- Coarse retrieval: The coarse retrieval model identifies candidate satellite locations using street-view images and transformed BEV projections as complementary query inputs.Its dual-branch design addresses the geometric domain gap between ground and satellite imagery.
- Fine matching: For each retrieved tile, a differentiable spherical transform and correlation-aware homography estimator recover the precise 3-DoF pose (x, y, θ).The estimator aligns transformed ground imagery with satellite maps while reducing the impact of unobservable areas.
- Pose recovery: The metric location is obtained by mapping the transformed ground-image center to satellite coordinates, while orientation uses a reference point above the image center.This converts the predicted homography into the ground camera pose used by the pipeline.
B. Structure-from-Motion Submodel Generation
The pipeline builds local SfM submodels from image correspondences, using graph-based image pairing and doppelgänger pruning to estimate precise relative poses. Fragmented connectivity can produce multiple disjoint submodels.
- B. Structure-from-Motion Submodel Generation: NetVLAD descriptors construct a dense similarity graph by connecting each image to its N most similar neighbors.The example uses N = 40 neighbors.
- B. Structure-from-Motion Submodel Generation: Doppelgänger detection prunes graph edges between visually similar but geographically distinct images to mitigate perceptual aliasing.Repeated architectural elements are given as an example of perceptual aliasing.
- B. Structure-from-Motion Submodel Generation: RDD local-feature correspondences between remaining image pairs drive COLMAP geometry estimation.The resulting SfM process estimates relative poses from matched keypoints.
- B. Structure-from-Motion Submodel Generation: The process yields M ≥ 1 disjoint submodels, although the desired outcome is a single coherent model with M = 1.Graph fragmentation can leave multiple independent image clusters.
- B. Structure-from-Motion Submodel Generation: Within each submodel, every image has a relative pose in a local coordinate system known with high precision.Each submodel also contains image locations, camera intrinsics, and sparse 3D points.
C. Robust Alignment via Kernel Density Estimation
The method converts noisy single-image satellite-ground pose estimates into a robust submodel alignment by exploiting rigid SfM geometry. KDE identifies the consensus origin and supports outlier rejection and confidence-based reliability checks.
- C. Robust Alignment via Kernel Density Estimation: SfM submodels provide rigid geometric constraints that help resolve high-variance pose proposals caused by limited view context and visual doppelgängers.The approach addresses the gap between noisy single-image matches and a consistent 3D scene model.
- C. Robust Alignment via Kernel Density Estimation: For each image, the method combines its relative SfM pose with its noisy absolute predicted pose to hypothesize the global origin of the submodel.The relative pose is expressed in the submodel’s local coordinate system.
- C. Robust Alignment via Kernel Density Estimation: KDE treats the origin hypotheses as samples, with true matches expected to cluster and alignment errors appearing as scattered outliers.The estimated probability density function identifies the likely geo-location of the submodel.
- C. Robust Alignment via Kernel Density Estimation: An image is retained as an inlier only when its hypothesis lies within distance threshold δ of the robust origin estimate.Hypotheses outside the threshold are rejected.
- C. Robust Alignment via Kernel Density Estimation: The KDE peak density at the robust origin acts as a localization confidence score, and low-confidence submodels are flagged as unreliable.A high peak indicates stronger consensus among hypotheses.
D. Final Global Reconstruction
Filtered geo-registration results are transformed into a local ENU frame and aligned to SfM submodels with a RANSAC-estimated Similarity transform. Structural inconsistencies can trigger recursive splitting before reassembly into one geo-referenced model.
- D. Final Global Reconstruction: Satellite-based location estimates are transformed into an ENU coordinate system centered on the mean geo-location.The ENU coordinates provide the frame for aligning SfM submodels.
- D. Final Global Reconstruction: RANSAC estimates a Similarity transform that accounts for rotation, translation, and scale when aligning SfM submodels to the transformed coordinates.The transform uses the robustly filtered geo-registration results.
- D. Final Global Reconstruction: Images with alignment errors greater than 15 meters are labeled as outliers by RANSAC.The labels are assigned per image location.
- D. Final Global Reconstruction: Three or more outliers in a submodel indicate possible structural inconsistency and cause the outlier images to be split into a separate submodel for recursive geo-localization.Iteration continues until fewer than three outliers remain.
IV. EXPERIMENTS
Experiments on WRIVA datasets evaluate hierarchical cross-view matching and its robust-matching and failure-detection variants using pose-error metrics. Robust matching improves performance across datasets, while failure detection further reduces error in the full pipeline.
- IV. EXPERIMENTS: Experiments use WRIVA public datasets organized by theme, location, challenge severity, and revision identifiers.The naming convention is t{TT} v{VV} s{SS} r{RR} {Name}.
- IV. EXPERIMENTS: The coarse retrieval model uses ConvNeXt-B, while fine matching uses ImageNet-pretrained EfficientNet-B0.Both networks are trained with AdamW, using 50 epochs for coarse retrieval and 100 epochs for fine matching.
- IV. EXPERIMENTS: The evaluation uses an 80% training split and a 20% testing split shared across ablations of robust matching and failure detection.The compared variants include +Robust, +Failure Detection, and a baseline excluding both components.
- IV. EXPERIMENTS: Across all datasets, robust matching consistently improves cross-view matching, and failure detection further reduces error by excluding unreliable predictions.The full pipeline therefore achieves the most accurate geo-localization among the evaluated variants.
- IV. EXPERIMENTS: Figure 2 shows the distributions of GPS localization and orientation/heading errors over test images in the evaluated WRIVA datasets.
B. 3D Reconstruction Performance
The evaluation compares satellite/ground alignment with aligned SfM using localization errors, scale estimates, and qualitative reconstructions. Alignment produces near-metric scale, while SfM can recover more complete models and join visually disjoint subsets.
- Evaluation setup: The evaluation compares satellite/ground estimates with aligned SfM using median 2D localization errors and metric-scale estimates on WRIVA sets.The valid columns cover images aligned in both modalities, while the table also reports input and jointly localized image counts.
- Localization and completeness: When satellite/ground localization fails, SfM can reconstruct more complete models because local feature matching overcomes image artifacts such as blur, compression, and saturation.This pattern is particularly visible in the progressively corrupted t05 v01 s{00-04} test vector.
- Metric scale: With satellite/ground alignment, model scale is close to metric, with mean 1.0 and standard deviation 0.04; without it, SfM scales are arbitrary.The scale factor is estimated by fitting a similarity transform between estimated cameras and ground-truth locations.
- Qualitative reconstruction: On the JHU APL office-building set, alignment resolves most doppelgänger matches and includes many more cameras in the final model.Without alignment, mismatches across the building’s North and South faces fracture the model at corners.
- Qualitative reconstruction: For a two-zone set with no visual overlap, satellite/ground alignment joins the two SfM models that would otherwise remain separate.Figure 3 reports this as a result unavailable without the proposed alignment.
V. CONCLUSIONS
The paper presents a scale-accurate framework combining SfM with coarse-to-fine satellite matching for unconstrained image collections. By enforcing consistency across SfM clusters, it aggregates noisy proposals into geographic consensus and enables metric scale recovery, doppelgänger rejection, and disjoint-model merging.
- The framework integrates SfM with coarse-to-fine satellite matching and enforces geometric consistency across SfM clusters to aggregate noisy proposals.