Source-linked AI summary
Geography-Aware Self-Supervised Learning
Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, Stefano Ermon
TL;DR
Geo-located datasets offer abundant unlabeled imagery but limited labels, while standard contrastive learning can still lag supervised learning. The paper exploits temporal alignment and geo-location in self-supervised training, closing the gap across remote-sensing tasks and improving geo-tagged ImageNet representations.
Problem
Geo-located remote-sensing data has plentiful unlabeled imagery but scarce labels, creating a need for effective self-supervised representations.
Method
The method combines temporal positive pairs from spatially aligned images with geo-location-based pre-text tasks in geography-aware contrastive learning.
Results
Across fMoW classification, detection, and segmentation, the method improves MoCo-v2 by approximately 8% accuracy, 2% AP, and 1% mIoU, respectively.
Takeaways & Limitations
The framework closes the self-supervised–supervised gap on remote sensing and extends to other geo-tagged image datasets.
Takeaways & Limitations
Temporal positives are unavailable for ImageNet because it lacks images from the same area over time, so GeoImageNet uses geo-location classification instead.
Abstract
from arXiv · showhide
Contrastive learning methods have significantly narrowed the gap between supervised and unsupervised learning on computer vision tasks. In this paper, we explore their application to geo-located datasets, e.g. remote sensing, where unlabeled data is often abundant but labeled data is scarce. We first show that due to their different characteristics, a non-trivial gap persists between contrastive and supervised learning on standard benchmarks. To close the gap, we propose novel training methods that exploit the spatio-temporal structure of remote sensing data. We leverage spatially aligned images over time to construct temporal positive pairs in contrastive learning and geo-location to design pre-text tasks. Our experiments show that our proposed method closes the gap between contrastive and supervised learning on image classification, object detection and semantic segmentation for remote sensing. Moreover, we demonstrate that the proposed method can also be applied to geo-tagged ImageNet images, improving downstream performance on various tasks. Project Webpage can be found at this link geography-aware-ssl.github.io.
1. Introduction
The paper examines self-supervised learning for geo-located imagery, where remote-sensing annotations are costly and existing contrastive methods trail supervised learning. It proposes geography-aware contrastive learning using temporal image pairs and geo-location information, improving performance across several downstream tasks.
- Remote-sensing annotation is costly because labels often require domain expertise.
- MoCo-v2 shows an 8% top 1 accuracy gap behind supervised learning on the fMoW image classification benchmark.
- Geography-aware contrastive learning uses spatially aligned images over time as positive pairs and geo-location information in unsupervised pre-text tasks.
- Temporal positive pairs encourage representations to become invariant to changes such as seasonality, supporting tasks focused on spatial variation.
- The method improves MoCo-v2 by approximately 8% classification accuracy, 2% AP for object detection, and 1% mIoU for semantic segmentation on fMoW.
- On geo-tagged ImageNet, geography-aware learning improves MoCo-v2 by approximately 2% on image classification.
2. Related Work
Related work covers self-supervised pre-text tasks and contrastive learning, while emphasizing that large-scale remote-sensing applications remain underexplored. This paper uses geo-location classification and geo-tagged data to improve unsupervised representation learning.
- Self-supervised learning uses unlabeled data to learn representations transferable to downstream tasks through pre-text tasks or contrastive learning.
- Pre-text methods learn representations by solving proxy problems such as rotation prediction, image colorization, or grid permutation.
- Geo-location classification trains a network to predict a coarse location of where an image was captured.
- Contrastive methods pull representations of positive pairs closer while pushing representations of negative pairs from different instances farther apart.
- Contrastive learning had not been explored on large-scale remote-sensing datasets, motivating an approach based on MoCo-v2 for geo-located data.
- Prior remote-sensing unsupervised studies commonly use small regions, few classes, or specialized modalities such as hyperspectral imagery.
- Existing geo-location research uses coordinates as recognition priors or predicts locations, whereas this work uses geo-tags to improve unsupervised learning.
3. Problem Definition
The paper defines geo-tagged visual sequences and datasets for learning representations without labels, using spatial and temporal structure in remote sensing data. It also introduces fMoW and GeoImageNet as evaluation settings with geographic metadata.
- Problem Definition: Geo-tagged visual data associate images with latitude and longitude, and remote sensing sequences may contain multiple spatially aligned views of one location over time.Temporal sequences can span months to years without viewpoint changes between images.
- Problem Definition: The representation-learning goal is to produce transferable visual representations without human supervision beyond geographic coordinates.Representation quality is evaluated through downstream-task performance.
- Functional Map of the World: fMoW contains approximately 363,571 training images and 53,041 test images across 62 classes, with temporal views and geo-location metadata.Most locations have multiple views, with temporal counts ranging from 1 to 21.
- GeoImageNet: GeoImageNet contains 543,435 geo-tagged ImageNet images across 5,150 classes and is more imbalanced and class-diverse than ImageNet-1k.Coordinates were obtained for an ImageNet subset through the FLICKR API.
- GeoImageNet: Geo-location can provide visual cues transferable to recognition tasks, such as distinguishing geographically associated elephant categories.The paper motivates predicting image location as a pre-text task for learning such cues.
4. Method
The method adapts contrastive learning to geo-located data by using temporally aligned images as positives and geographic labels as an auxiliary pretext task. These components are combined to learn representations that capture both spatio-temporal similarity and location.
- 4.1. Contrastive Learning Framework: MoCo-v2 learns representations by bringing positive pairs closer and pushing negative pairs farther apart using query and key encoders.The framework uses augmented views of images to form positive pairs, with negatives drawn from a queued dictionary.
- 4.1. Contrastive Learning Framework: Temporal positive pairs use two spatially aligned images from the same area, potentially captured at different times, after applying MoCo-v2 perturbations.When the two timestamps are identical, the pair reduces to the positive-pair construction used by MoCo-v2.
- 4.1. Contrastive Learning Framework: TemporalInfoNCE encodes perturbed temporal positives with query and key encoders while contrasting them against encoded negative samples using a temperature hyperparameter.The objective is designed to make representations invariant to changes over time.
- 4.1. Contrastive Learning Framework: Temporal invariance may be undesirable for change detection, although the authors state it should not degrade image classification or object detection.The method therefore introduces an inductive bias whose suitability depends on the downstream task.
- 4.2. Geo-location Classification as a Pre-text Task: Geolocation pretraining clusters image coordinates into K categorical geo-labels and trains a predictor with cross-entropy loss to learn location-aware representations.The learned location-aware representations are intended to potentially transfer to different downstream tasks.
- 4.3. Combining Geo-location and Contrastive Learning Losses: The combined objective integrates geo-location prediction into contrastive learning and jointly optimizes TemporalInfoNCE with geo-classification loss.Coefficients α and β control the relative importance of contrastive learning and geo-location learning losses.
5. Experiments
Experiments evaluate unsupervised representations across classification, detection, segmentation, and land-cover tasks on remote-sensing and geo-tagged image datasets. Geography-aware objectives and temporal positives improve performance over MoCo-v2 and supervised or random-initialization comparisons.
- Experimental Setup: The study evaluates learned representations on image recognition, object detection, semantic segmentation, and land-cover classification benchmarks.Experiments use fMoW, GeoImageNet, xView, SpaceNet, and NAIP-related datasets.
- Experimental Setup: fMoW clustering uses K = 100 geo-clusters derived from latitude and longitude, with differences between fMoW and GeoImageNet cluster distributions.GeoImageNet has fewer clusters per class and more unique classes per cluster than fMoW in the reported analysis.
- fMoW Classification: 60.69% accuracy for frozen-feature MoCo-v2 on single-image fMoW classification is 8% below supervised learning, while geo-location and temporal positives raise accuracy to 68.32%.Temporal positives reduce the gap to supervised learning to less than 1%.
- fMoW Classification: Area-specific temporal inference improves classification accuracy by 4-8% over image-specific inference, with the proposed methods outperforming MoCo-v2 by 4-6% and supervised learning by 1-2%.The evaluation covers 11,231 unique areas represented by multiple images over time.
- Transfer Learning: The final model outperforms random initialization by 7% AP and supervised learning on fMoW by 3.3% AP for xView object detection.The best results combine temporal positive pairs and the geo-location classification pre-text task.
- Transfer Learning: The final model exceeds random initialization and supervised learning by 3.58% and 2.94% IoU scores, respectively, on SpaceNet segmentation.The strongest IoU scores use both temporal positives and geo-location classification.
- Transfer Learning: On NAIP land-cover classification, the method outperforms random initialization by 6.34% and supervised learning by 3.77%.The task covers 66 land-cover classes with 100,000 training and 50,000 test images.
- GeoImageNet: On GeoImageNet, MoCo-v2 reaches 38.51 top-1 accuracy, about 3.47% above supervised learning, while geo-location classification adds 1.45%.GeoImageNet contains 5,150 highly imbalanced classes, approximately five times more than ImageNet-1k.
6. Conclusion
The paper presents a self-supervised framework for remote sensing, where unlabeled data is plentiful and labeled data is scarce. It uses temporal positive pairs and geo-location pre-text tasks to close the supervised-learning gap across several downstream tasks and geo-tagged image datasets.
- Conclusion: The framework leverages spatially aligned images over time and geo-location pre-text tasks for self-supervised learning.It targets remote sensing and other geo-tagged image datasets.
- Conclusion: The method closes the gap between self-supervised and supervised learning on image classification, object detection, and semantic segmentation.The conclusion covers remote sensing and other geo-tagged image datasets.