Source-linked AI summary
Cross-View Image Matching for Geo-localization in Urban Environments
Yicong Tian, Chen Chen, Mubarak Shah
TL;DR
The paper addresses cross-view geo-localization when a query image must be matched with geo-tagged imagery from another viewpoint. It detects and matches buildings using deep networks and dominant-set optimization, and reports better accuracy than other approaches while generalizing to unseen locations.
Problem
Cross-view geo-localization must estimate a query image’s GPS location from geo-tagged images in another view despite substantial viewpoint and appearance differences.
Method
The framework detects buildings with Faster R-CNN, retrieves candidate building matches with a Siamese network, and selects consistent multiple matches using dominant sets.
Results
The method outperforms other cross-view geo-localization approaches and generalizes to images from unseen locations.
Takeaways & Limitations
Cross-view geo-localization can use semantic building matching and global consistency to localize queries against geo-tagged imagery from the other view.
Abstract
from arXiv · showhide
In this paper, we address the problem of cross-view image geo-localization. Specifically, we aim to estimate the GPS location of a query street view image by finding the matching images in a reference database of geo-tagged bird's eye view images, or vice versa. To this end, we present a new framework for cross-view image geo-localization by taking advantage of the tremendous success of deep convolutional neural networks (CNNs) in image classification and object detection. First, we employ the Faster R-CNN to detect buildings in the query and reference images. Next, for each building in the query image, we retrieve the $k$ nearest neighbors from the reference buildings using a Siamese network trained on both positive matching image pairs and negative pairs. To find the correct NN for each query building, we develop an efficient multiple nearest neighbors matching method based on dominant sets. We evaluate the proposed framework on a new dataset that consists of pairs of street view and bird's eye view images. Experimental results show that the proposed method achieves better geo-localization accuracy than other approaches and is able to generalize to images at unseen locations.
1. Introduction
The paper frames geo-localization as matching images across street-level and bird’s-eye views when same-view geo-tagged references may be unavailable. It proposes a building-centered framework and a scalable multiple-neighbor matching method, evaluated with a new three-city dataset.
- Same-view geo-localization can depend on geo-tagged references that may be unavailable for some locations.
- Cross-view matching is difficult because viewpoints, lighting, seasons, and viewpoint mappings differ, while SIFT and HOG features may fail.
- The framework detects and matches buildings as semantic information rather than relying on local features such as SIFT and HOG.
- Cross-view matching estimates a query street view image’s GPS location from matching geo-tagged bird’s-eye images, or the reverse.
- The paper introduces a scalable multiple nearest neighbors matching method based on dominant sets and evaluates it in both matching directions.
- The new dataset contains annotated street view and bird’s-eye view image pairs collected from three United States cities.
2. Related Work
Ground-level geo-localization commonly retrieves matching geo-tagged images from the same view, but limited city coverage motivates cross-view methods using more widely available overhead data. This framework instead matches buildings across views, aggregates multiple nearest neighbors for global consistency, and requires no depth or other metadata.
- Ground-level Geo-localization: Ground-level methods estimate a query’s location by retrieving matching geo-tagged images from the same view and using their locations.
- Ground-level Geo-localization: Limited coverage of ground-level imagery in cities worldwide has hindered scaling image-to-image geo-localization globally.
- Cross-view Geo-localization: Overhead satellite, aerial, and elevation data offer more complete coverage, motivating cross-view geo-localization research.
- Cross-view Geo-localization: The proposed framework differs by using buildings as reference objects because semantic information is more robust to viewpoint changes than local appearance features.
- Cross-view Geo-localization: Multiple nearest-neighbor matching extends localization from one query image to multiple query buildings while incorporating global consistency.
- Cross-view Geo-localization: The approach does not require depth maps or other metadata.
3. Proposed Cross-view Geo-localization Method
The proposed method matches buildings across street-view and bird’s-eye-view images, then combines multiple candidate matches using local similarity and GPS-based global consistency to estimate location.
- Building detection: Faster R-CNN detects buildings in query and reference images because semantic building information is more robust to viewpoint changes than appearance features.
- Multiple nearest-neighbor matching: For each query building, the method retrieves k nearest reference buildings and forms a cluster of candidate matches.The candidates are selected using building-matching scores before graph-based optimization.
- Building matching: A Siamese network learns cross-view building representations by bringing matched pairs close and separating unmatched pairs using contrastive loss.The network receives building-image pairs from street-view and bird’s-eye-view training sets.
- Multiple nearest-neighbor matching: The graph connects candidates from different clusters with weights combining reference-building similarity and GPS distance, and the objective selects at most one candidate per cluster while maximizing total weight.GPS distance supplies a global measure, while building-matching scores supply local measures.
- Geo-localization using dominant sets: Dominant sets select a coherent set of reference buildings, after which the estimated location is the mean GPS position of the selected buildings.The selected set is coherent according to both global and local measures.
- Multi-view queries: Using four query images with headings 0°, 90°, 180°, and 270° provides more detected buildings for matching than a single query image.The paper illustrates this setup with street-view images captured at the same GPS location.
4. Experiments
Experiments evaluate the dataset, detection and matching components, parameter choices, baseline comparisons, and generalization to an unseen city. The proposed building-matching framework performs strongly across these evaluations.
- Experimental Setup: The dataset contains street-view and bird’s-eye-view image pairs from Pittsburgh, Orlando, and Manhattan, with Manhattan held out for testing unseen-city generalization.The sampled locations number 1,586 in Pittsburgh, 1,324 in Orlando, and 5,941 in Manhattan.
- Experimental Setup: The dataset uses unaligned urban images that usually contain multiple buildings, unlike prior datasets designed around single-building or building-free imagery.This setup targets cross-view building matching in realistic urban scenes.
- Building Detection: Faster R-CNN detects buildings accurately in both views, including crowded scenes where buildings occlude one another.Separate detectors are trained for street-view and bird’s-eye-view images.
- Building Matching: 0.32 average precision is achieved by the fine-tuned Siamese model, compared with 0.11 for the pre-trained model on cross-view building matching.Figure 9 additionally shows ranked visual matching examples with the correct match marked by a red box.
- Parameter Analysis: Using k=100 for street-view queries and k=10 for bird’s-eye-view queries provides the selected nearest-neighbor settings because street-view images generally contain fewer buildings.Figure 10 evaluates localization results under different k values at a fixed 300m error threshold.
- Comparisons and Generalization: Building matching outperforms SIFT matching and random selection, while four directional query images substantially improve accuracy over one query image.Full-image matching performs worse than building matching with four-view queries; dominant sets are also more efficient than GMCP as nearest-neighbor or cluster counts increase.
- Comparisons and Generalization: On Manhattan, localization curves are lower because the test area is larger, yet the method remains much better than SIFT, demonstrating generalization to the unseen city.The Manhattan experiment operates at city scale and includes challenging tall, crowded buildings.
5. Conclusion
The paper presents a cross-view geo-localization framework that matches buildings between street-view and bird’s-eye-view images using deep learning and dominant sets. Experiments on a new paired dataset show improved performance over other approaches, while future work extends matching beyond buildings.
- Conclusion: The framework localizes a query image by matching it to geo-tagged images from the other view, using building detection, cross-view building matching, and dominant-set matching of multiple query buildings.It supports street-view queries against bird’s-eye references and the reverse direction.
- Conclusion: The introduced dataset contains large-scale pairs of street-view and bird’s-eye-view images for cross-view geo-localization evaluation.The dataset supports the paper’s experiments on urban environments and unseen locations.
- Conclusion: Experiments show that the proposed method outperforms other approaches for cross-view geo-localization on the introduced dataset.The conclusion reports the overall experimental outcome without specifying a single aggregate metric.
- Future Work: Future work will extend matching to areas without buildings by using other objects and semantic information such as road structure and water reservoirs.The paper suggests generalizing building matching to multiple-attribute matching.