PaperlayerView exact revision on arXiv

Source-linked AI summary

PlaNet - Photo Geolocation with Convolutional Neural Networks

Tobias Weyand, Ilya Kostrikov, James Philbin

arXiv:1602.05314v1cs.CV

TL;DR

Photo geolocation is difficult because images may contain few or ambiguous location cues. PlaNet formulates the task as classification over geographic cells using a CNN trained on millions of geotagged images, and extends it with an LSTM for albums. It outperforms previous approaches, reaches superhuman performance in some cases, and achieves a 50% improvement over the single-image model on photo albums.

  • Problem

    Photo geolocation is challenging because many images provide few or ambiguous cues, while prior approaches often focus on restricted settings.

  • Method

    PlaNet classifies pixels into geographic cells with a CNN trained on millions of geotagged images, then uses an LSTM to exploit temporal coherence across photo albums.

  • Results

    PlaNet outperforms previous geolocation methods and reaches superhuman performance in some cases, while album geolocation improves 50% over the single-image model.

  • Takeaways & Limitations

    Probability distributions let PlaNet represent uncertainty and assign plausible locations for diverse photos, while album context helps geolocate uncertain images.

  • Takeaways & Limitations

    PlaNet’s comparison with Im2GPS is confounded by training-data scale because PlaNet used 14x more data.

Abstract

from arXiv · show

Is it possible to build a system to determine the location where a photo was taken using just its pixels? In general, the problem seems exceptionally difficult: it is trivial to construct situations where no location can be inferred. Yet images often contain informative cues such as landmarks, weather patterns, vegetation, road markings, and architectural details, which in combination may allow one to determine an approximate location and occasionally an exact location. Websites such as GeoGuessr and View from your Window suggest that humans are relatively good at integrating these cues to geolocate images, especially en-masse. In computer vision, the photo geolocation problem is usually approached using image retrieval methods. In contrast, we pose the problem as one of classification by subdividing the surface of the earth into thousands of multi-scale geographic cells, and train a deep network using millions of geotagged images. While previous approaches only recognize landmarks or perform approximate matching using global image descriptors, our model is able to use and integrate multiple visible cues. We show that the resulting model, called PlaNet, outperforms previous approaches and even attains superhuman levels of accuracy in some cases. Moreover, we extend our model to photo albums by combining it with a long short-term memory (LSTM) architecture. By learning to exploit temporal coherence to geolocate uncertain photos, we demonstrate that this model achieves a 50% performance improvement over the single-image model.

1. Introduction

Photo geolocation is difficult because visual cues are sparse and ambiguous, while prior approaches often target restricted settings. PlaNet instead classifies photos into geographic cells, producing uncertainty-aware predictions and extending them to albums with temporal context.

  • 1. Introduction: Ambiguous or scarce visual cues make photo geolocation exceptionally challenging, even when landmarks may have replicas elsewhere.Typical beaches and landmarks can correspond to multiple locations.
  • 1. Introduction: Humans can combine world knowledge with cues such as sign language and driving direction, whereas traditional algorithms depend on training-provided features.
  • 1. Introduction: Prior work commonly covered restricted subsets, including landmarks, street-view cities, or locations with dense internet-photo coverage.PlaNet aims to localize any photo taken at any location.
  • 1. Introduction: PlaNet treats geolocation as classification over geographic cells and trains a CNN on millions of geotagged images to output a probability distribution over Earth.The distribution assigns each cell a likelihood for the input photo.
  • 1. Introduction: PlaNet localizes diverse landmarks, street scenes, and nature images, often representing ambiguous cases with multiple plausible geographic modes.It outperforms Im2GPS and reaches superhuman performance in a small street-view experiment.
  • 1. Introduction: 50% improvement over the single-image model is achieved for photo albums by combining PlaNet with an LSTM that exploits temporal coherence.The sequence model helps geolocate uncertain photos using nearby album images.

2. Related Work

Related work largely uses retrieval, landmark recognition, geometric matching, or auxiliary geographic information, often within urban or densely covered areas. PlaNet is positioned as a CNN-based classification approach designed for broader photo geolocation, with an LSTM extension for image sequences.

  • 2. Related Work: Im2GPS retrieves similar geotagged Flickr images using global image descriptors and assigns the closest match’s location to the query.Its effectiveness improves with sufficient data despite the simplicity of the approach.
  • 2. Related Work: Satellite-aerial methods address sparse rural coverage by matching ground images to aerial imagery or transforming ground features into aerial-image space.
  • 2. Related Work: Local-feature retrieval improves building matching but requires more space and lacks invariance for natural scenes or articulated objects.Consequently, many such methods focus on city-level localization.
  • 2. Related Work: Pose-estimation systems recover exact 6-dof camera poses by matching query interest points against structure-from-motion 3D models and solving PnP.
  • 2. Related Work: Landmark recognition retrieves or classifies images associated with landmark clusters, including systems using BoVW-based SVMs and exemplar SVMs.
  • 2. Related Work: CNN superiority in SUN scene recognition over global and local descriptors motivates using CNNs for image geolocation.
  • 2. Related Work: Prior sequence-geolocation methods exploit temporal coherence with landmark-sequence models, HMMs, or structured SVMs.These approaches differ in their class definitions and use of temporal information.
  • 2. Related Work: PlaNet addresses a gap left by approaches restricted mainly to urban areas and is presented as the first direct CNN classification approach to geolocation.

3. Image Geolocation with CNNs

PlaNet formulates photo geolocation as CNN classification over adaptively partitioned geographic cells, producing probability distributions over possible locations. It localizes diverse images across geographic scales, outperforms Im2GPS, and exposes both strengths and evaluation caveats.

  • Method: PlaNet classifies image pixels into geographic cells and outputs a probability distribution over the world, allowing uncertainty to be represented.The classification formulation uses a one-hot cell target during training and cell confidences at inference.
  • Method: The earth is partitioned into hierarchical S2 cells whose adaptive subdivision limits the number of photos per cell and yields 26,263 cells.S2 cells are preferred over latitude/longitude regions because they remain closer to quadratic and more uniform in size.
  • Results: 48.0% of photos are localized at continent level, 28.4% at country level, 10.1% at city level, and 3.6% at street level.Using the best of five predictions roughly doubles correct localization at street, city, region, and country levels.
  • Qualitative Results: PlaNet handles landmarks, street scenes, landscapes, characteristic architecture, locally typical objects, plants, and animals, while ambiguous scenes can produce multiple plausible locations.Its learned representation for a cell can combine diverse landmarks, landscapes, and animals typical of a region.
  • Comparison: PlaNet localizes 236% more images accurately than Im2GPS at street level and 51% more at country level.The comparison is qualified because PlaNet was trained on 14 times more data, although its model requires 377 MB versus an estimated 8.5 GB for Im2GPS descriptors.
  • Human Comparison: PlaNet won 28 of 50 human-comparison rounds, with median localization error of 1131.7 km versus 2320.75 km for humans.Neither humans nor PlaNet localized the panoramas below street or city level.

4. Sequence Geolocation with LSTMs

The paper extends PlaNet to geolocate photo albums with LSTMs that exploit temporal coherence, improving localization over single-image and averaging approaches while exposing sequence-length and ordering trade-offs.

  • Motivation: Photo albums provide geographic correlation that can help localize ambiguous images lacking sufficient visual cues.The LSTM accumulates state from previously seen photos and uses it with the current image.
  • Model architecture: The album model feeds PlaNet image embeddings through an LSTM and classifies chronological photos into geographical cells.The LSTM output is passed to a SoftMax layer, while the single-image PlaNet parameters are reused for the image encoder.
  • Label offset: Label offsets do not improve localization accuracy because they complicate input-to-label mapping and do not universally correct early uncertain predictions.The approach postpones inference for several time steps but can still fail when the first confident image appears later than the offset.
  • Repeated sequences: 7.8% relative improvement at street level is achieved by repeated-sequence LSTMs over single-pass LSTMs, but inference time doubles.The first pass encodes the sequence state and the second pass produces predictions conditioned on all images.
  • Bi-directional LSTM: 16.6% relative improvement on street level is achieved by BLSTMs over repeated LSTMs on 25-image sequences, although full-sequence unrolling limits tractability.The 25-image limit lowers total accuracy because longer albums typically yield higher accuracy, so the repeated model may remain preferable in practice.

5. Conclusion

PlaNet frames image geolocation as CNN-based classification over geographical cells and extends it to albums with LSTMs. It outperforms other methods, reaches superhuman performance in some settings, and improves album localization by 50% over the single-image model.

  • Contribution: PlaNet uses a CNN to classify image pixels into geographical cells and outputs a probability distribution over the globe.This distribution represents uncertainty and can assign probability mass to potential locations.
  • Album geolocation: 50% higher performance than the single-image model is achieved by combining PlaNet with LSTMs for photo album geolocation.The improvement comes from using contextual information for image-based localization.
Loading 1602.05314v1…