PaperlayerView exact revision on arXiv

Source-linked AI summary

Localizing and Orienting Street Views Using Overhead Imagery

Nam Vo, James Hays

arXiv:1608.00161v2cs.CV

TL;DR

The paper addresses fine-scale matching of street-view images to overhead imagery for geolocalization despite viewpoint, lighting, seasonal, and orientation differences. It develops deep-learning matching methods with improved loss functions and rotation handling, finding that DBL and supervised alignment improve representation and ranking performance.

  • Problem

    Fine-scale street-to-overhead matching is challenging because of large viewpoint variation and changing lighting or seasons, while overhead imagery offers broad geographic coverage for geolocalization.

  • Method

    The paper studies deep CNN matching and retrieval approaches, introducing a distance based logistic layer and rotation-invariance and orientation-regression training mechanisms.

  • Results

    Supervised orientation regression improves the 360°RI network's performance by 30% relatively, while the proposed DBL layer significantly improves representation-learning networks.

  • Takeaways & Limitations

    Cross-view ranking benefits from supervised alignment information, and the proposed methods improve rotationally invariant representations for street-to-overhead matching.

  • Takeaways & Limitations

    Using hundreds of rotation samples is computationally impractical and provides no significant improvement.

Abstract

from arXiv · show

In this paper we aim to determine the location and orientation of a ground-level query image by matching to a reference database of overhead (e.g. satellite) images. For this task we collect a new dataset with one million pairs of street view and overhead images sampled from eleven U.S. cities. We explore several deep CNN architectures for cross-domain matching -- Classification, Hybrid, Siamese, and Triplet networks. Classification and Hybrid architectures are accurate but slow since they allow only partial feature precomputation. We propose a new loss function which significantly improves the accuracy of Siamese and Triplet embedding networks while maintaining their applicability to large-scale retrieval tasks like image geolocalization. This image matching task is challenging not just because of the dramatic viewpoint difference between ground-level and overhead imagery but because the orientation (i.e. azimuth) of the street views is unknown making correspondence even more difficult. We examine several mechanisms to match in spite of this -- training for rotation invariance, sampling possible rotations at query time, and explicitly predicting relative rotation of ground and overhead images with our deep networks. It turns out that explicit orientation supervision also improves location prediction accuracy. Our best performing architectures are roughly 2.5 times as accurate as the commonly used Siamese network baseline.

1 Introduction

The paper frames ground-to-overhead matching as a challenging but attractive route to fine-scale image geolocalization, exploiting dense overhead coverage and deep cross-domain representations. It studies multiple deep learning approaches and introduces methods for improving retrieval performance.

  • Overhead imagery offers dense Earth coverage, unlike unevenly sampled geotagged ground photographs, making ground-to-overhead matching attractive for geolocalization.
  • Large viewpoint, lighting, and seasonal differences make matching street-view images to overhead imagery challenging, even for humans.
  • The paper studies deep CNN architectures for matching, verification, ranking, and retrieval across ground-level and overhead images.
  • The proposed distance based logistic layer and rotation-related training methods target stronger representations while retaining retrieval applicability.
  • A new large-scale dataset pairs randomly cropped Google street-view panoramas with finest-scale overhead images across eleven U.S. cities.

3 Cross-view matching and ranking with CNN

The paper compares CNN architectures for cross-view matching and ranking, separating classification-based models from representation-learning networks. It introduces distance-based logistic loss for Siamese and triplet networks while preserving efficient test-time matching and ranking.

  • The study evaluates classification, hybrid, Siamese, and triplet CNN architectures for cross-view matching and retrieval.Classification recognizes matches directly, whereas representation-learning networks embed cross-view images into a shared feature space.
  • Classification-based networks are used as a loose accuracy upper bound because they are impractical for large-scale retrieval.They permit only partial feature precomputation, unlike embedding-based approaches.
  • Siamese networks independently encode two images, compare their feature distance, and classify the pair as a match when the distance is sufficiently small.In this implementation, the subnetworks do not share weights because street-view and overhead images belong to different domains.
  • Contrastive loss pulls matched features together and separates non-matched features until their distance exceeds a margin.D is the squared feature distance, and m is the margin parameter controlling when non-match penalization stops.
  • Triplet networks learn ranking relevance from an anchor, a matched image, and a non-matched image using a hinge loss over their distance gap.The loss encourages the matched pair to be closer than the non-matched pair, with a margin determining when penalization ends.
  • Distance-based logistic loss replaces thresholded distance penalties with log-loss optimization for Siamese and triplet networks, while learned features remain usable for efficient matching and ranking at test time.For triplets, the loss models the probability that the matched image is more relevant to the anchor than the non-matched image.

4 Learning to perform rotation invariant matching

The paper addresses unknown orientation in fine-grained street-to-overhead matching by learning rotation-tolerant representations and adding explicit orientation supervision. It explores partial rotation invariance, test-time rotation sampling, feature averaging, and orientation regression.

  • 4 Learning to perform rotation invariant matching: Because street-view orientation is unknown at test time, spatial matching requires handling relative orientation rather than assuming north-aligned overhead images.The paper uses orientation-aligned training data as stronger supervision despite unaligned test conditions.
  • 4.1 Partial rotation invariance by data augmentation: Partial rotation invariance is encouraged by randomly rotating aligned overhead training images within a controlled range.The range controls the trade-off between invariance and discriminative ability; 360°RI denotes full rotation invariance.
  • 4.1 Partial rotation invariance by data augmentation: Partially rotation-invariant representations require multiple rotated query samples at test time to find a good match.For example, 180°RI requires at least two samples 180° apart, while 360°RI requires one sample.
  • 4.1 Partial rotation invariance by data augmentation: Multi-orientation feature averaging combines features from multiple rotated samples during indexing to obtain more stable representations.With full rotation invariance, averaging all samples preserves one-sample query-time cost.
  • 4.2 Learning better representations with orientation regression: Orientation regression adds an auxiliary Euclidean-loss branch that predicts the training rotation applied to overhead images.The branch concatenates fc7 features and applies two fully connected layers; its purpose is to make representations more orientation-aware.

5 Experiments

Experiments evaluate CNN architectures, retrieval, and rotation handling on a large cross-view dataset. Classification and hybrid models are strongest for matching, while DBL-enhanced embedding networks improve scalable retrieval; rotation supervision and averaging further help.

  • Dataset and evaluation: More than 1 million matched pairs from 11 U.S. cities provide training and test data, with 900k examples from 8 cities used for training.The remaining 3 cities form test sets of around 70k examples each.
  • 5.1 Comparison of CNN architectures: Classification networks outperform Siamese and triplet networks because they jointly extract and exchange information from both input images.The hybrid network performs better than the pure classification network in these experiments.
  • 5.1 Comparison of CNN architectures: Triplet networks outperform Siamese networks on both matching and ranking tasks, while contrastive loss crowds matched distances near zero.That distance behavior may compromise fine-grained retrieval, where small distances must be compared.
  • 5.1 Comparison of CNN architectures: DBL-enhanced Siamese and triplet networks significantly outperform their original counterparts, with triplet DBL-Net slightly better at ranking.The triplet DBL-Net is used for subsequent experiments.
  • 5.2 Rotation invariance: Increasing training-time rotation invariance improves performance, while limited invariance requires more test-time rotated samples.Using 16 rotations, the 90°RI network achieves the highest performance; hundreds of samples are computationally impractical.
  • 5.2 Rotation invariance: 30% relative improvement results from adding orientation regression to the 360°RI network, although it does not improve the 90°RI network.The orientation prediction error is 17° for ground-truth matching overhead images.
  • 5.2 Rotation invariance: Averaging features from 16 rotations produces performance comparable to exhaustively testing 16 samples for the 360°RI plus OR network.The strategy also slightly improves partial-RI networks.
  • 5.3 Efficient training: The eDBL-Net with 360°RI, OR, and average-16 converges faster, matching 150k-iteration performance after about 30k iterations and improving ranking after 80k.The result uses the same batch size as the earlier 150k-iteration experiments.

6 Conclusion

The paper introduces a large-scale cross-view dataset and evaluates CNN architectures for matching street-view and overhead imagery. DBL improves representation learning, while supervised alignment information further improves ranking through rotationally invariant representations.

  • The authors introduce a large-scale dataset of street scenes paired across ground-level and overhead views.
  • The Siamese network with contrastive loss is the least competitive approach for cross-view matching.
  • The proposed DBL layer significantly improves representation learning networks.
  • Supervised alignment information further improves ranking by learning a rotationally invariant representation.

1 GTCrossView dataset

The dataset was collected from selected U.S. cities, with city choice influenced by image-quality consistency. Most images depict rural-like scenes because urban areas occupy relatively small portions of the cities.

  • The dataset’s cities were selected while considering inconsistent image quality across locations.
  • Most dataset images are rural-like because urban areas are relatively small even in big cities.
  • Figure 1 identifies the cities used for training and testing, marking them in black and red respectively.

2 Network visualization

Visualization of early and later learned features shows that filters often specialize in one view, with more filters emphasizing street-view inputs. Higher-level features respond to scene patterns and cross-view structural similarities.

  • The classification network’s first-layer filters use six channels split between street-view and overhead images.
  • Most first-layer filters focus on one image, and 111 of 192 filters emphasize street-view channels more than overhead channels.
  • Representation-learning filters show view-specific differences while remaining similar to corresponding classification-network filters.
  • Classification features respond to similar scenes or scene pairs, while other cross-view features detect buildings with similar structural patterns.
  • Figure 4 displays images producing extremely large or small values for an output feature.

3 Ranking performance

Ranking experiments compare several CNN-based matching networks on cross-view datasets and geolocalization examples. The Siamese baseline performs relatively poorly, while DBL, rotation-related training, orientation regression, and mini-batch exhausting improve triplet performance.

  • The Siamese baseline performs relatively poorly for ranking, suggesting limited suitability for retrieval applications.
  • DBL, rotation-related methods, orientation regression, and mini-batch exhausting each improve the triplet network significantly.
  • Geolocalization examples infer likely street-view scene positions from overhead-image locations.
  • The prior-dataset experiment evaluates ranking using about 80,000 matched pairs across seven cities, training on 31,000 pairs.
  • Triplet eDBL-Net outperforms the prior method and traditional triplet network on most test sets, with direct comparability limited by smaller training data.

4 Orientation regression performance

The auxiliary OR loss enables the network to predict the orientation difference between street-view and overhead images. On the Denver test set, predictions have an average absolute error of around 17°, with most errors near 0° and a small symmetry-related peak near ±180°.

  • Orientation regression performance: The auxiliary OR loss enables prediction of the orientation difference between street-view and overhead images.Orientation prediction is reported as a by-product and is not used for image geolocalization.
  • Orientation regression performance: 17° average absolute error is achieved on the Denver test set.The error is computed as the absolute difference between true and predicted orientations.
  • Orientation regression performance: Most orientation errors lie near 0°, while a small peak near −180°/180° corresponds to scenes appearing symmetrical from above.The error distribution is shown as a histogram in Figure 7.

5 Residual network

This section compares AlexNet with a deeper ResNet-101 model and presents orientation-prediction examples alongside aligned overhead imagery.

  • Residual network: A ResNet-101 model is trained to examine whether a deeper network can provide benefits over AlexNet.The comparison results are reported in Table 2.
  • Residual network: Table 2 compares the AlexNet and ResNet-101 models.The supplied table caption identifies the two architectures but does not state the numerical results.
  • Residual network: Figure 8 shows street-view images, ground-truth aligned overhead images, and alignments produced using predicted orientation.The three rows correspond respectively to the street views, ground-truth alignments, and predicted-orientation alignments.
Loading 1608.00161v2…