Source-linked AI summary
Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data
Oscar Mañas, Alexandre Lacoste, Xavier Giro-i-Nieto, David Vazquez, Pau Rodriguez
TL;DR
Remote sensing provides vast unlabeled imagery, but annotation scarcity and domain gaps limit conventional supervised and ImageNet-based transfer learning. SeCo collects uncurated Sentinel-2 data and applies seasonal contrastive self-supervision with representations that can encode temporal invariance and variation. Across BigEarthNet, EuroSAT, and OSCD, SeCo outperforms the considered ImageNet and MoCo baselines.
Problem
Remote sensing has abundant imagery but scarce costly annotations, while ImageNet pre-training may not generalize reliably across the remote-sensing domain.
Method
SeCo combines unsupervised collection of uncurated Sentinel-2 imagery with multi-augmentation contrastive learning that uses seasonal views and separate embedding sub-spaces.
Results
SeCo outperforms ImageNet and MoCo pre-training on BigEarthNet, EuroSAT, and OSCD tasks.
Takeaways & Limitations
Domain-specific unsupervised pre-training is more effective for the evaluated remote sensing applications than standard ImageNet pre-training or MoCo.
Takeaways & Limitations
The sampling strategy assumes most relevant variability can be observed in areas around cities.
Abstract
from arXiv · showhide
Remote sensing and automatic earth monitoring are key to solve global-scale challenges such as disaster prevention, land use monitoring, or tackling climate change. Although there exist vast amounts of remote sensing data, most of it remains unlabeled and thus inaccessible for supervised learning algorithms. Transfer learning approaches can reduce the data requirements of deep learning algorithms. However, most of these methods are pre-trained on ImageNet and their generalization to remote sensing imagery is not guaranteed due to the domain gap. In this work, we propose Seasonal Contrast (SeCo), an effective pipeline to leverage unlabeled data for in-domain pre-training of remote sensing representations. The SeCo pipeline is composed of two parts. First, a principled procedure to gather large-scale, unlabeled and uncurated remote sensing datasets containing images from multiple Earth locations at different timestamps. Second, a self-supervised algorithm that takes advantage of time and position invariance to learn transferable representations for remote sensing applications. We empirically show that models trained with SeCo achieve better performance than their ImageNet pre-trained counterparts and state-of-the-art self-supervised learning methods on multiple downstream tasks. The datasets and models in SeCo will be made public to facilitate transfer learning and enable rapid progress in remote sensing applications.
1. Introduction
Remote sensing offers abundant imagery for global monitoring, but scarce annotations and domain mismatch limit supervised and transferred learning. SeCo addresses this with unsupervised Sentinel-2 data collection and seasonal, multi-space self-supervised representations evaluated across remote sensing tasks.
- Motivation: Over 700 satellites generate terabytes of imagery daily, but downstream tasks remain constrained by costly, scarce annotations and underexplored remote-sensing applications of label-efficient methods.Annotations may require expert knowledge or expensive ground sensors.
- Motivation: Existing remote sensing datasets are carefully curated, so discarding labels does not remove the human effort involved in selecting examples.SeCo instead constructs Sentinel-2 data without human curation or annotation.
- Method: Seasonal image pairs provide natural, semantically meaningful positives that complement or replace artificial transformations in contrastive learning.They are obtained from the same location at different points in time.
- Method: Seasonal Contrast (SeCo) combines an unsupervised data-acquisition procedure with a self-supervised model for transferable remote sensing representations.The method leverages satellite revisit information and seasonal variation.
- Method: Separate embedding sub-spaces let SeCo encode representations that are variant or invariant to seasonal changes rather than enforcing one universal temporal invariance.A shared backbone produces a common representation for multiple downstream uses.
- Results: SeCo pre-training is more effective than ImageNet and MoCo pre-training on BigEarthNet, EuroSAT, and OSCD tasks.The evaluations cover land-cover classification and change detection.
2. Background
Self-supervised learning derives training signals from data transformations, while contrastive methods learn by comparing similar and dissimilar examples. InfoNCE formalizes this comparison by bringing positive representations closer than negative ones.
- Self-supervised learning: Self-supervised learning uses the data itself to define a pretext task, such as predicting occluded or perturbed content.Image examples include patch-position prediction, jigsaw solving, rotation prediction, and colorization.
- Contrastive learning: Contrastive learning pulls representations of similar examples together and pushes representations of dissimilar examples apart.Without labels, each example is treated as defining its own class for pair construction.
- Contrastive learning: A contrastive dictionary-look-up maps augmented query and key views into an embedding space and compares the designated positive key with negative keys.The encoder produces q, k+, and k− representations for the comparison.
- Contrastive learning: InfoNCE is a common contrastive objective over positive and negative pairs, with temperature τ scaling the distribution of distances.The temperature controls the loss’s distance-distribution scaling.
3. Method
SeCo combines unsupervised Sentinel-2 dataset construction with multi-augmentation contrastive learning to pre-train transferable remote sensing representations. Its multi-head embedding architecture separates invariances to seasonal and artificial transformations while preserving a general representation for transfer learning.
- Unsupervised Dataset Construction: SeCo constructs an unsupervised pre-training dataset from Sentinel-2 patches sampled worldwide across multiple locations and seasonal timestamps.The dataset uses approximately 200K locations, five images per location, and dates separated by roughly three months.
- Unsupervised Dataset Construction: The sampling strategy targets geographic diversity by sampling around the 10k most populated cities rather than uniformly across the Earth.Coordinates are drawn from a Gaussian distribution with a 50 km standard deviation around each selected city, with cloud-filtered Sentinel-2 observations sought near quarterly dates.
- Views Generation: SeCo uses temporal views from the same location as positive pairs, complementing artificial image augmentations with natural seasonal variation.A query and keys are formed from three images at different times, while one key receives only artificial transformations such as cropping, color jitter, or flipping.
- Multiple Embedding Sub-spaces: A shared encoder projects image representations into three normalized sub-spaces that encode distinct invariance and variance patterns.Z0 is invariant to all augmentations; Z1 is season-invariant but artificial-augmentation-variant; Z2 is artificial-augmentation-invariant but season-variant.
- Multiple Embedding Sub-spaces: The contrastive objective sums losses across sub-spaces, defining positive and negative pairs according to each sub-space’s intended invariances.The general embedding space V is used for transfer learning because downstream-task invariances are not assumed in advance; a specific Zi can be used when they are known.
4. Experiments
The experiments evaluate SeCo representations across land-cover classification and change detection, using multiple baselines, backbones, label budgets, and dataset sampling strategies. SeCo generally outperforms ImageNet and self-supervised baselines, while its temporal design supports both seasonal invariance and change sensitivity.
- Tasks and evaluation: SeCo is evaluated on BigEarthNet and EuroSAT land-cover classification and OSCD change detection.The evaluation uses linear probing and fine-tuning, with frozen or updated encoders respectively.
- BigEarthNet: SeCo consistently outperforms MoCo-v2+TP on BigEarthNet linear probing and improves over ImageNet features, including a 1.1% higher accuracy with 1M images and ResNet-50.The comparison spans pre-training methods, backbones, unlabeled-set sizes, and labeled-data percentages.
- BigEarthNet: With 1% of BigEarthNet labels, SeCo surpasses ImageNet pre-training using 100% of the labels and matches MoCo-v2 using 20%.Across label percentages, SeCo maintains an approximately 4% improvement over MoCo-v2+TP; with fine-tuning, it matches ImageNet using 50% of the labels.
- EuroSAT: 6.7% higher accuracy than ImageNet and 3.6% higher accuracy than MoCo-v2+TP are reported for SeCo on EuroSAT.The results are based on linear probing and indicate generalization beyond BigEarthNet.
- OSCD change detection: 6.8% higher F1 score than MoCo-v2+TP is achieved by SeCo on OSCD change detection, while SeCo also exceeds all listed baselines in recall and F1.Compared with MoCo-v2, SeCo obtains a 6.2% higher F1 score, which the authors associate with multiple embedding sub-spaces disentangling temporal variation from augmentations.
- OSCD change detection: SeCo qualitative masks cover more changed pixels without excessive false negatives than the compared baselines on OSCD validation samples.Figure 4 presents input images, ground-truth masks, and generated change masks for each sample.
5. Related Work
Prior work explores unsupervised representation learning through uncurated data, augmentation invariance, and remote-sensing-specific signals, but unsupervised learning in remote sensing remains underexplored.
- Unsupervised feature learning has often used small or curated datasets, while uncurated raw data can reduce transfer-task feature quality.
- Multi-augmentation contrastive methods learn transferable representations by enforcing invariance to image transformations, but their assumptions can conflict with downstream tasks.
- Unsupervised learning remains underexplored in remote sensing despite abundant data and domain-specific opportunities for self-supervision.
- Remote-sensing approaches use geography, georeferenced text, or multispectrality to construct task-specific self-supervised learning signals.
6. Conclusions
The paper presents SeCo, a transfer-learning pipeline that combines unsupervised remote-sensing data collection with seasonal contrastive learning. SeCo outperforms ImageNet and MoCo pre-training baselines across three downstream tasks.
- SeCo combines a multi-timestamp data collection strategy with a self-supervised algorithm that accounts for seasonal changes.
- SeCo outperforms ImageNet pretraining and MoCo pre-training on BigEarthNet, EuroSAT, and OSCD tasks.
- The comparisons span different backbones and dataset sizes, supporting domain-specific unsupervised pre-training over standard ImageNet or MoCo alternatives.