PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization

Vicente Vivanco Cepeda, Gaurav Kumar Nayak, Mubarak Shah

arXiv:2309.16020v2cs.CVcs.LG

TL;DR

Worldwide geo-localization is difficult because global image galleries are infeasible and discrete geographic classes limit precision and coverage. GeoCLIP instead aligns CLIP-based image features with continuous GPS embeddings using hierarchical random-Fourier location encoding and contrastive learning. The method is competitive with limited training data, supports text-query geo-localization qualitatively, and performs competitively across benchmark datasets.

  • Problem

    Worldwide geo-localization requires precise global GPS prediction, but image galleries are infeasible and discrete geographic classes can mislocalize images away from class centers.

  • Method

    GeoCLIP aligns CLIP-based image features with GPS embeddings produced by a hierarchical location encoder using random Fourier features and contrastive learning.

  • Results

    GeoCLIP achieves competitive worldwide geo-localization, including with substantially reduced training data, and qualitatively supports text-query localization.

  • Takeaways & Limitations

    Continuous GPS retrieval provides a global alternative to image-gallery retrieval and predefined geographic-cell classification, while the learned location encoder is suitable for use beyond geo-localization.

  • Takeaways & Limitations

    Uniformly sampling 1 million GPS coordinates covers only 1,000,000 km², making inclusion of a true coordinate within a 1km reference radius unlikely over Earth's 148,326,000 km² land area.

Abstract

from arXiv · show

Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere on Earth. This task has considerable challenges due to immense variation in geographic landscapes. The image-to-image retrieval-based approaches fail to solve this problem on a global scale as it is not feasible to construct a large gallery of images covering the entire world. Instead, existing approaches divide the globe into discrete geographic cells, transforming the problem into a classification task. However, their performance is limited by the predefined classes and often results in inaccurate localizations when an image's location significantly deviates from its class center. To overcome these limitations, we propose GeoCLIP, a novel CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations. GeoCLIP's location encoder models the Earth as a continuous function by employing positional encoding through random Fourier features and constructing a hierarchical representation that captures information at varying resolutions to yield a semantically rich high-dimensional feature suitable to use even beyond geo-localization. To the best of our knowledge, this is the first work employing GPS encoding for geo-localization. We demonstrate the efficacy of our method via extensive experiments and ablations on benchmark datasets. We achieve competitive performance with just 20% of training data, highlighting its effectiveness even in limited-data settings. Furthermore, we qualitatively demonstrate geo-localization using a text query by leveraging CLIP backbone of our image encoder. The project webpage is available at: https://vicentevivan.github.io/GeoCLIP

1 Introduction

Worldwide geo-localization seeks GPS coordinates globally, but image galleries are infeasible and geographic-cell classification is limited by discrete centers and sparse coverage. GeoCLIP addresses these constraints by aligning image and continuous GPS embeddings through a CLIP-inspired retrieval framework.

  • Worldwide geo-localization determines image latitude and longitude across the globe, including locations without distinctive landmarks or tourist hotspots.
  • Image-to-image retrieval is impractical globally because constructing a gallery covering all possible images worldwide is infeasible.
  • Cell-based classification can produce localization error when the true location deviates from its predicted class center and limits predictions to approximately 21k locations.
  • GeoCLIP retrieves unseen-image GPS coordinates by matching image and location encoders in a shared embedding space.
  • The location encoder uses random Fourier features, multi-resolution hierarchical representations, and contrastive learning to model GPS coordinates continuously across geographic scales.
  • GeoCLIP uses a CLIP image backbone and a dynamic queue of GPS coordinates, enabling additional negatives and qualitative localization from text queries.

3 Proposed Approach

GeoCLIP formulates worldwide geo-localization as image-to-GPS retrieval using continuous GPS representations rather than predetermined geographic classes. Its architecture combines a frozen CLIP-based image encoder with a hierarchical location encoder and contrastive training that uses augmented views and additional GPS negatives.

  • Problem setup: GeoCLIP trains a worldwide model on image–GPS pairs to predict the coordinates of unseen query images from any location on Earth.The task is formulated as retrieval over GPS locations rather than image galleries or fixed geographic classes.
  • Model architecture: The architecture uses a frozen CLIP ViT-L/14 image backbone followed by two trainable linear layers, alongside a dedicated GPS location encoder.The two encoders project images and locations into representations suitable for matching.
  • Location encoder: Random Fourier Features encode projected GPS coordinates with fixed Gaussian-sampled frequencies to capture high-frequency spatial detail that standard MLPs may miss.The location encoder uses positional encoding because direct MLP encoding can suffer from spectral bias.
  • Location encoder: Equal Earth Projection transforms GPS coordinates before positional encoding, reducing distortions in standard latitude–longitude representations near the poles.The projected longitude is scaled to −1 to 1, with latitude scaled proportionally.
  • Hierarchical representation: An exponential range of RFF frequencies creates hierarchical location features from coarse to fine resolutions, whose independently processed vectors are added into a joint representation.This multiscale design lets the encoder specialize in spatial information at different resolutions.
  • Training: Contrastive training aligns each image with its GPS feature while separating other locations, using augmented image–GPS views, coordinate noise, and a dynamic queue of additional GPS negatives.The queued GPS embeddings notably improve performance, particularly at smaller scales.

4 Experiments

GeoCLIP is evaluated on multiple worldwide geo-localization benchmarks, ablations, limited-data settings, and a text-query application. The experiments show consistent benchmark gains, data efficiency, and benefits from hierarchical GPS encoding.

  • Comparison with State-of-the-art methods: GeoCLIP yields consistent gains across Im2GPS3k and GWS15k benchmarks at different distance thresholds.
  • Comparison with State-of-the-art methods: On GWS15k, GeoCLIP surpasses the previous SOTA model by +1.6%, +8.2%, +18.8%, and +23.6% at the 25km, 200km, 750km, and 2500km thresholds.
  • Performance in Limited Data settings: The classification method’s performance degrades much faster than GeoCLIP’s as training data is reduced, indicating greater data efficiency.
  • Performance in Limited Data settings: GeoCLIP remains competitive when training data is reduced exponentially, including settings with only 5% of the data.
  • Ablations: Equal Earth Projection, Random Fourier Features, and a dynamic queue improve GPS encoding over an MLP baseline, especially at fine localization scales.
  • Ablations: Combining encoders with different sigma values outperforms any individual sigma across all five localization scales, supporting hierarchical representation.
  • Qualitative Results: Geo-localization using Text-based query: GeoCLIP maps text embeddings to GPS embeddings and produces similarity maps showing the worldwide spatial distribution of queried concepts.
  • Qualitative Results: Geo-localization using Text-based query: The learned location representation achieves state-of-the-art performance in an additional GPS-based image-classification application.

5 Conclusion

GeoCLIP formulates worldwide geo-localization as image-to-GPS retrieval and remains competitive with substantially less training data. Its hierarchical location encoder improves multiscale localization and supports text queries and classification beyond geo-localization.

  • GeoCLIP formulates worldwide geo-localization as an image-to-GPS retrieval problem.
  • GeoCLIP remains competitive in limited-data settings, even when training data is reduced significantly.
  • Hierarchical learning combines location features across scales and further boosts performance over coarse- or fine-grained RFF choices.
  • GeoCLIP supports qualitative geo-localization with text queries by leveraging its CLIP image-encoder backbone.
  • The location encoder also aids classification problems beyond geo-localization.

Supplementary for: “GeoCLIP: Clip-Inspired

The supplementary document covers additional datasets, limited-data experiments, ablations, implementation analyses, qualitative demonstrations, and ethical issues.

  • Additional sections report results on another dataset and limited-data settings for YFCC26k and GWS15k.
  • The supplementary experiments include ablations of gallery size and the number of hierarchies, plus alternative GPS gallery construction choices.
  • Further analyses examine runtime and memory footprint and the motivation for using pretrained CLIP as the image-encoder backbone.
  • Qualitative sections demonstrate GeoCLIP with image and text queries and show the distribution of correct predictions across datasets.
  • The supplementary document also includes a section discussing ethical issues and possible mitigation.

1 Results on additional dataset

GeoCLIP is evaluated on the YFCC26k dataset, where it achieves state-of-the-art performance across most distance-threshold metrics.

  • The supplementary evaluation extends GeoCLIP experiments to the YFCC26k dataset.
  • GeoCLIP achieves state-of-the-art performance on YFCC26k for the majority of distance-threshold metrics.
  • +1.51%, +2.59%, +7.87%, and +7.02% accuracy improvements are reported at 1km, 200km, 750km, and 2500km, respectively.

2 Results for limited data settings on YFCC26k and GWS15k datasets

GeoCLIP maintains competitive performance on YFCC26k and GWS15k when training data is substantially reduced, including reductions to 5%.

  • The supplementary experiments investigate limited-data performance on YFCC26k and GWS15k.
  • Performance is not affected considerably by limited training data, with the trend holding across both datasets.
  • Even with training data reduced exponentially to 5%, GeoCLIP still achieves competitive performance.

3 Additional Ablations

These ablations examine gallery size, negative-sample queues, GPS noise, Fourier-feature scales, and hierarchy depth. Results show that choices balancing fine- and coarse-scale localization are important.

  • Gallery size: 14.11% accuracy at 1km results when the Im2GPS3k gallery grows from 21K to 100K, up from 11.88%.On YFCC26k, 1km accuracy similarly rises from 8.44% to 11.61%.
  • Gallery size: 0.6% accuracy at 1km is achieved on GWS15k with a 500K gallery, up from 0.18%.Larger galleries provide no further gains and can reduce fine-scale performance.
  • Queue length: 9.84%, 33.10%, 49.32%, 69.07%, and 83.30% accuracy are obtained by a queue of 4096 GPS negatives at 1km, 25km, 200km, 750km, and 2500km.The 4096-length queue generally outperforms the other tested queue lengths.
  • GPS noise: 150 meters yields the best performance for Gaussian noise added to GPS coordinates.The same noise value is used across datasets in the main experiments.
  • σ for Random Fourier Features: Higher σ values perform better at 1km and 25km, whereas lower σ values perform better at 200km, 750km, and 2500km.A single σ is not optimal across all distance metrics.
  • Hierarchical learning: Hierarchical learning outperforms a single intermediate σ across multiple distance metrics, with performance improving through three hierarchies before slightly declining.The dynamic queue also improves performance over a static queue, and 1000-meter noise adds gains across thresholds.

4 Different selection choices for GPS Gallery Construction

GeoCLIP can construct GPS galleries from training coordinates, evenly spaced global coordinates, land-only coordinates, or test-set locations. Gallery construction affects fine-scale coverage and generalizability.

  • Training-derived galleries: Training-derived GPS galleries use reference coordinates sampled from the training dataset rather than predefined geographic classes.This follows the assumption that training data reflects likely geographical regions of interest.
  • Evenly spaced coordinates: 1 million globally uniform coordinates cover land and ocean using a Fibonacci lattice without assuming prior geographical knowledge.A land-only variant restricts the same-sized gallery to terrestrial regions when images are known to be taken on land.
  • Evenly spaced coordinates: Uniform global sampling is weak at 1km because the gallery covers only a small fraction of Earth’s land area, but it becomes more competitive at larger thresholds.Fine-scale accuracy requires a reference coordinate near the true location.
  • Test-set coordinates: Test-set GPS galleries produce strong localization across thresholds but can reduce the task to retrieval among known labeled locations.This construction therefore may lack generalizability.

5 Analysis of Runtime and Memory Footprint

GeoCLIP trades more runtime and memory than ISNs for substantially lower costs than Translocator. It outperforms both baselines across all reported distance thresholds.

  • Efficiency comparison: GeoCLIP outperforms ISNs and Translocator across all distance-threshold metrics.The comparison concerns runtime, memory footprint, and localization effectiveness.
  • Efficiency comparison: GeoCLIP uses more runtime and memory than ISNs but substantially less than Translocator.ISNs avoid gallery search, while Translocator uses auxiliary scene context and a larger dual-branch architecture.

6 Motivations for using Pretrained CLIP as Image encoder Backbone

GeoCLIP uses a frozen pretrained CLIP image encoder to leverage generalization and reduce computation. CLIP’s aligned text and image spaces also enable qualitative text-query geo-localization without separate text training.

  • Computational motivation: Precomputing frozen CLIP features reduces one MP-16 training epoch from approximately 27 hours to 15 minutes.The reduction substantially lowers training overhead.
  • Frozen backbone: The frozen CLIP backbone performs better than fine-tuning in the reported empirical evaluations.The paper suggests catastrophic forgetting may explain why fine-tuning does not improve performance.
  • Text-query localization: Aligning CLIP’s pretrained image encoder with the location encoder implicitly aligns its text encoder with geographic locations.This allows GeoCLIP to process text queries without separate text training.
  • Qualitative demonstrations: Qualitative demonstrations cover image queries, text queries, hierarchy visualizations, and prediction distributions across benchmark data.The text-query examples include cities and specific landmarks such as the Pyramids of Giza, Burj Khalifa, and Great Wall of China.

8 Discussion on Ethical Issues and Possible Mitigation

GeoCLIP's global image geo-localization capability raises privacy and security concerns, particularly when geographic locations are revealed without consent. The authors propose adversarial noise as a defense that can disrupt image embeddings while remaining imperceptible to humans.

  • Global geo-localization can reveal individuals’ geographic locations without consent, creating privacy and security concerns.
  • Adversarial noise can significantly alter CLIP image embeddings and disrupt GeoCLIP’s localization while remaining imperceptible to humans.
Loading 2309.16020v2…