PaperlayerView exact revision on arXiv

Source-linked AI summary

SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation

Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M Albrecht, Xiao Xiang Zhu

arXiv:2211.07044v2cs.CVcs.AI

TL;DR

Earth-observation pre-training draws on ImageNet or medium-sized labeled remote-sensing datasets, while existing datasets have limitations in scale, coverage, or modality. SSL4EO-S12 provides an unlabeled, global, multimodal, multi-season Sentinel-1/2 dataset, and experiments report improved downstream performance over existing datasets and results close to or above supervised learning.

  • Problem

    EO pre-training commonly uses ImageNet or medium-sized labeled RS datasets, while available RS datasets have limitations in geographic and seasonal coverage, modality, and patch overlap.

    Source: §I · p. 1, A. Motivation · p. 15

  • Method

    SSL4EO-S12 assembles Sentinel-1 and Sentinel-2 imagery from globally distributed locations across four seasonal snapshots for self-supervised pre-training.

    Source: §I · p. 1, C. Collection process · p. 16

  • Results

    Across scene classification, semantic segmentation, and change detection, SSL4EO-S12 pre-training improves downstream performance over existing datasets, and resulting models perform close to or above supervised learning.

    Source: §I · p. 1

  • Takeaways & Limitations

    The dataset is intended for diverse remote-sensing applications and can also serve as a baseline for future EO pre-training datasets.

    Source: E. Uses · p. 16

    Coverage is limited in polar regions, cloud filtering introduces geographical bias, overlap is not fully eliminated, and the data represent only a subset of EO imagery.

    Source: §VII · p. 6

Abstract

from arXiv · show

Self-supervised pre-training bears potential to generate expressive representations without human annotation. Most pre-training in Earth observation (EO) are based on ImageNet or medium-size, labeled remote sensing (RS) datasets. We share an unlabeled RS dataset SSL4EO-S12 (Self-Supervised Learning for Earth Observation - Sentinel-1/2) to assemble a large-scale, global, multimodal, and multi-seasonal corpus of satellite imagery from the ESA Sentinel-1 \& -2 satellite missions. For EO applications we demonstrate SSL4EO-S12 to succeed in self-supervised pre-training for a set of methods: MoCo-v2, DINO, MAE, and data2vec. Resulting models yield downstream performance close to, or surpassing accuracy measures of supervised learning. In addition, pre-training on SSL4EO-S12 excels compared to existing datasets. We make openly available the dataset, related source code, and pre-trained models at https://github.com/zhu-xlab/SSL4EO-S12.

I. INTRODUCTION

SSL4EO-S12 addresses limited scale, geographic coverage, seasonal diversity, and modality in EO pre-training data with a global Sentinel-1/2 dataset. Experiments report improved downstream performance over existing datasets across several SSL methods and tasks.

II. RELATED WORK

  • ImageNet is a poor fit for remote sensing because of domain differences, while existing RS datasets have limited scale, class balance, temporal coverage, or geographic coverage.The paper cites ImageNet's domain gap and limitations in labeled RS datasets; it also notes that RS imagery is not object-centric and spans multiple sensors and temporal variation.Source: §I · p. 1, §II · pp. 1–2

I. INTRODUCTION

  • SSL4EO-S12 samples 250K global locations across Sentinel-2 L1C, Sentinel-2 L2A, and Sentinel-1, with four seasonal snapshots and about 3 million patches.The locations are sampled to avoid overlap, improving geospatial coverage.Source: §I · p. 1
  • Pre-training on SSL4EO-S12 improves downstream performance over existing datasets across scene classification, semantic segmentation, and change detection, with ablations supporting RS-specific augmentations.The evaluated methods are MoCo, DINO, MAE, and data2vec; the reported augmentation benefits include multi-sensor, multi-temporal, and atmospheric correction.Source: §I · p. 1

II. RELATED WORK

  • The benchmark covers contrasting negative-sample methods, teacher-student distillation, masked reconstruction, and masking with joint embeddings.MoCo, DINO, MAE, and data2vec represent these categories, respectively.Source: §II · p. 1
  • Prior datasets leave gaps: SEN12MS has limited temporal coverage, SeCo is optical-only, and both contain strongly overlapping patches.SSL4EO-S12 addresses these gaps with more locations, multiple seasons, Sentinel-1 and two Sentinel-2 products, and overlap filtering.Source: §II · p. 2

III. SSL4EO-S12 DATASET

SSL4EO-S12 is assembled by sampling locations around populated cities, collecting four seasonal observations, and filtering overlapping patches. The resulting dataset contains more than three million medium-resolution, multimodal images across global locations.

A. Data curation & assembly

  • Locations are sampled around the top 10,000 populated cities, with four annual-season observations and cloud-filtered Sentinel-1/2 imagery collected for each accepted patch.The sampling uses a Gaussian distribution around city centers with a 50 km standard deviation; accepted patches are 2640m×2640m.Source: A. Data curation & assembly · p. 2
  • Overlap filtering checks patch-center distances and uses a one-degree grid to limit comparisons to nearby patches.A distance below three-quarters of a patch width indicates non-negligible overlap; the grid strategy assumes overlap between distinct grid cells is statistically negligible.Source: A. Data curation & assembly · pp. 2–3

B. Data characteristics & volume

  • The dataset contains 251,079 triplets of Sentinel-1 dual-polarization SAR, Sentinel-2 top-of-atmosphere multispectral, and Sentinel-2 surface-reflectance multispectral images across four seasonal timestamps.It comprises more than 3 million images and about 3.7 TB uncompressed.Source: B. Data characteristics & volume · p. 3
  • Figures 2 and 3 depict the dataset's geographic distribution and the effect of overlap filtering in the Tokyo metropolitan area, respectively.Figure 3 shows patches without filtering on the left and with filtering on the right.Source: B. Data characteristics & volume · p. 3, Figure 2 · p. 3, Figure 3 · p. 3

IV. EXPERIMENTAL SETUP

The experiments pre-train representative self-supervised methods and transfer their models to scene classification, semantic segmentation, and change detection. Across the reported classification comparisons, self-supervised pre-training performs competitively with or better than supervised training.

A. Self-supervised pre-training

  • Pre-training uses MoCo(-v2), DINO, MAE, and data2vec with ResNet or ViT backbones, typically using Sentinel-2 L1C and season-based augmentation.For MoCo and DINO, positive views are randomly picked from two seasons; MAE and data2vec assign one random season per patch.Source: A. Self-supervised pre-training · p. 3
  • The cited result reports that pre-training one ResNet/ViT model for 100 epochs takes 7–25 hours on 4 NVIDIA A100 GPUs.Source: A. Self-supervised pre-training · p. 3

B. Transfer learning

  • Transfer evaluations cover EuroSAT, BigEarthNet, and So2Sat-LCZ42 for scene classification, plus DFC2020 segmentation and OSCD change detection.The study uses both linear probing, which freezes the pre-trained encoder, and fine-tuning.Source: B. Transfer learning · p. 3

1) Comparison of SSL methods:

  • Linear probing comes within approximately ±1% of supervised training on small BigEarthNet label splits or EuroSAT, while the best models beat the supervised baseline by about 1% to 4% on So2Sat-LCZ42.On BigEarthNet with all labels, the gap to fully supervised training stays below 5%; So2Sat-LCZ42 has a challenging geographical split.Source: 1) Comparison of SSL methods · p. 3, Table III · p. 4
  • Fine-tuned self-supervised methods outperform supervised learning by 1% to 6% across EuroSAT, BigEarthNet, and So2Sat-LCZ42.Top models score 99.1% on EuroSAT and over 90% on BigEarthNet.Source: 1) Comparison of SSL methods · pp. 3–4, Table IV · p. 4

2) Comparison of pre-training datasets:

The dataset comparisons show gains over existing pre-training datasets, while transfer evaluations cover label efficiency, land-cover segmentation, and change detection. These results are qualified by task-specific comparisons and data conditions.

2) Comparison of pre-training datasets:

3) Comparison of different amounts of labels:

  • On BigEarthNet, fine-tuning with 10% of labels outperforms supervised training with 50%; with ViT-S/16, 50%-label fine-tuning outperforms 100%-label supervised training.Figure 4 presents transfer-learning performance across varying fractions of labeled samples.Source: 3) Comparison of different amounts of… · p. 4, Figure 4 · p. 5

1) Land cover segmentation:

  • On DFC2020 land-cover segmentation, SSL4EO-S12 shows notable improvements over SeCo but trails SEN12MS in average accuracy and mean intersection over union.The authors attribute the SEN12MS comparison to DFC2020’s similar data distribution and direct reference to SEN12MS.Source: 1) Land cover segmentation · p. 4, Table VI · p. 4
  • On OSCD change detection, SSL4EO-S12 yields superior recall and F1-score to SeCo and SEN12MS, but lower precision.The authors link the precision result to class imbalance: predicting all pixels as unchanged would score well on precision.Source: 2) Change detection · pp. 4–5, Table VII · p. 5

VI. ADDITIONAL STUDIES

The additional studies examine multimodality, seasonal information, product level, and data scale, using MoCo-v2 with ResNet50 as a common setting. They report benefits from both multimodal pre-training and seasonal augmentation.

VI. ADDITIONAL STUDIES

  • The additional studies use MoCo-v2 with ResNet50 to document dataset characteristics including multimodality, multi-temporal information, product level, and data scale.The main experimental section uses optical data for comparisons with prior work; these studies also examine multimodal pre-training.Source: §VI · p. 5, 1) Benefits of multimodality · p. 5

1) Benefits of multimodality:

  • On BigEarthNet, multimodal pre-training improves results by 1%–3% over unimodal pre-training for both 100% and 1% label splits.Single-modality pre-training works for both Sentinel-2 and Sentinel-1, while using both modalities further improves performance.Source: 1) Benefits of multimodality · p. 5, Table VIII · p. 5

2) Ablation of seasonal information:

  • The seasonal-information ablation reports benefits from seasonal augmentation on BigEarthNet-10% and EuroSAT.The study compares seasonal augmentation with random-season and fixed-season alternatives on a 50k subset of SSL4EO-S12.Source: 2) Ablation of seasonal information · p. 5, Table IX · p. 5

3) Atmospheric correction as data augmentation:

The section examines Sentinel-2 product levels as pre-training augmentation, the effect of pre-training scale, and learned representations. It closes by describing SSL4EO-S12’s benefits and limitations.

3) Atmospheric correction as data augmentation:

  • Pre-training on both Sentinel-2 L1C and L2A yields up to 4% improvement over a single product level, while matching the downstream product level gives a slight ∼1% edge.The comparison uses linear probing; Table X reports results for different Sentinel-2 product levels.Source: 3) Atmospheric correction as data… · p. 5, Table X · p. 5

4) Impact of pre-training scale:

  • At 50% (500K) or less pre-training data, BigEarthNet-10% performance drops significantly; the gaps diminish from 75% (750K), with saturation also depending on model size.The authors also cite computational cost when discussing dataset scale.Source: 4) Impact of pre-training scale · pp. 5–6, Table XI · p. 6

B. Representation visualization

  • On EuroSAT, SSL-encoded features form well-clustered t-SNE representations, unlike features from a randomly initialized encoder.A frozen MoCo-v2 ResNet50 encoder produces one 128-dimensional representation per image; colors indicate classes.Source: Figure 5 · p. 6, B. Representation visualization · p. 6

VII. CONCLUSION

  • The authors present SSL4EO-S12 as a large-scale multimodal, multi-temporal unlabeled dataset, and report promising benchmark benefits alongside limited polar coverage, cloud-filtering bias, residual overlap, and restricted sensor scope.The stated sensor scope is medium-resolution radar and multispectral imagery.Source: §VII · p. 6

APPENDIX-1: ADDITIONAL DATASET INFORMATION

This appendix describes the Sentinel-1 and Sentinel-2 products in SSL4EO-S12, its dataset statistics, storage, metadata, and seasonal examples. It also gives the main pre-training setup and MoCo and DINO configurations.

A. Sentinel-1/2

  • SSL4EO-S12 uses Sentinel-1 GRD imagery with VH and VV polarization and Sentinel-2 L1C top-of-atmosphere and L2A atmospherically corrected imagery.Sentinel-1 imagery is 10 m resolution; Sentinel-2 bands range from 10 m to 60 m, with 13 L1C and 12 L2A bands.Source: A. Sentinel-1/2 · p. 8

B. Dataset statistics

C. Data storage

  • Each patch is stored as per-band GeoTiff files; raw products occupy about 500–800 GB each, while JPEG encoding reduces each product to under 50 GB.The appendix states that this lossy encoding has little effect on downstream performance.Source: C. Data storage · p. 8

D. Metadata

  • Each patch has a metadata file describing its image properties, with details listed for Sentinel-1 GRD and Sentinel-2 L1C.The appendix points to Tables XX and XXI for metadata details.Source: D. Metadata · p. 8

E. Example visualization

  • The example tile visualization arranges Sentinel-1 GRD, Sentinel-2 L1C, and Sentinel-2 L2A imagery by rows and spring, summer, fall, and winter by columns.The rows show grayscale and false-color imagery.Source: E. Example visualization · p. 8, Figure 7 · p. 9

A. Pre-training

  • Main pre-training uses Sentinel-2 L1C converted to uint8, ResNet50 or ViT-S/16, 100 epochs, and a total batch size of 256 across four A100 GPUs.Training time ranges from 7 hours for MAE to 25 hours for DINO.Source: A. Pre-training · p. 10

1) MoCo: / 2) DINO:

  • MoCo uses RandomSeasonContrast among its augmentations; DINO adds Multi-Crop and Solarization to the MoCo augmentation set.The appendix gives separate optimizer and learning-rate settings for ResNet50 and ViT-S/16.Source: 1) MoCo · p. 10, 2) DINO · p. 10

3) MAE:

This appendix specifies MAE and data2vec pre-training, downstream evaluation setups, and preprocessing and input-size findings. It reports comparable performance across several preprocessing choices and an advantage from upsampling downstream inputs.

3) MAE: / 4) data2vec:

  • MAE pre-training uses a 0.7 mask ratio, RandomResizedCrop, RandomHorizontalFlip, and RandomSeason; data2vec uses Resize/CenterCrop, RandomHorizontalFlip, and RandomSeason.Both use AdamW and a cosine learning-rate schedule, with different learning rates.Source: 3) MAE · p. 10, 4) data2vec · p. 10

1) EuroSAT: / 2) BigEarthNet:

  • EuroSAT evaluation uses 21,600 training and 5,400 test images; BigEarthNet uses 311,667 training and 103,944 test images.BigEarthNet uses MultiLabelSoftMarginLoss and a 0.8 training crop scale; EuroSAT uses CrossEntropyLoss.Source: 1) EuroSAT · p. 10, 2) BigEarthNet · p. 10

3) So2Sat-LCZ42:

  • So2Sat-LCZ42 evaluation uses 352,366 training and 24,119 test images, with training and test data drawn from different cities.The appendix uses the culture-10 version of So2Sat-LCZ42.Source: 3) So2Sat-LCZ42 · p. 10

4) DFC2020: / 5) OSCD:

  • DFC2020 evaluation uses 5,128 training and 986 test images; OSCD uses 14 of 24 multispectral image pairs for training and the remaining pairs for testing.OSCD images are split into non-overlapping 96 × 96-pixel patches.Source: 4) DFC2020 · p. 10, 5) OSCD · p. 10

A. Effect of data pre-processing

  • Sixteen-bit and uint8 pre-processing yield similar performance, as do L1C and L2A and normalized and unnormalized inputs; the main experiments therefore use unnormalized uint8 L1C.The uint8 conversion divides by 10,000 and multiplies by 255; compressed input saves storage and computing time.Source: A. Effect of data pre-processing · p. 11, Table XIV · p. 11

B. Effect of input image size

C. Effect of MAE masking ratio

The appendix examines MAE masking and pre-training protocols, compares dataset performance, and documents SSL4EO-S12’s motivation and composition. It describes the dataset as a representative but imperfect sample of Sentinel imagery.

C. Effect of MAE masking ratio

D. Effect of different pre-training protocols 1) Different ImageNet pre-training protocols:

  • ImageNet pre-training generalizes to remote sensing with RGB but not all bands; self-supervised ImageNet pre-training further improves remote-sensing performance over supervised ImageNet pre-training when using RGB.The comparison reports fine-tuning on BigEarthNet.Source: 1) Different ImageNet pre-training… · p. 11, Table XVII · p. 12
  • On EuroSAT, self-supervised pre-training on remote-sensing data outperforms supervised pre-training on remote-sensing data.The comparison includes supervised BigEarthNet pre-training and self-supervised BigEarthNet and SSL4EO-S12 pre-training.Source: 2) Supervised pre-training on RS… · p. 11, Table XVIII · p. 12

E. Additional dataset comparison results

  • Under fine-tuning evaluation, SSL4EO-S12 pre-training outperforms the other compared datasets for downstream classification, retaining an approximately 5% margin over ImageNet.The authors note that fine-tuning is more computationally expensive than linear probing.Source: E. Additional dataset comparison results · p. 12, Table XIX · p. 12

APPENDIX-4: METADATA

A. Motivation

  • SSL4EO-S12 was created for unsupervised Earth-observation pre-training, combining global coverage, multiple modalities, and multiple timestamps to address limitations of existing datasets.The stated limitations include ImageNet’s domain gap, BigEarthNet’s regional coverage, SeCo’s single modality, SEN12MS’s single timestamp, and patch overlap in SEN12MS and SeCo.Source: A. Motivation · p. 15

B. Composition

  • The dataset contains unlabeled Sentinel-1/2 imagery with metadata, location, and acquisition time; it samples Sentinel imagery and users may select the full dataset or subsets of modalities or locations.The datasheet reports 251,079 geographical patches and 1M patches each for Sentinel-1 GRD, Sentinel-2 L1C, and Sentinel-2 L2A.Source: B. Composition · p. 15
  • The datasheet identifies potential overlap around grid-cell boundaries and residual cloud noise from inaccurate cloud filtering.It also reports that the dataset is self-contained.Source: B. Composition · p. 15

C. Collection process

SSL4EO-S12 was collected from public Sentinel-1/2 data using Google Earth Engine and Python, with locations and seasonal timestamps sampled to broaden coverage. Online preprocessing filtered cloudy and overlapping patches; the raw data was not retained.

C. Collection process

  • The data came from publicly available Sentinel-1/2 sources and was collected with Google Earth Engine and Python.Collection occurred in February and March 2022; the images were captured in 2020/2021.Source: C. Collection process · p. 16
  • Locations were Gaussian-sampled within 50 km of city centers selected from the 10,000 most populated cities, and four seasonal timestamps were sampled from 2020/2021 imagery.The data was automatically collected and verified by the authors.Source: C. Collection process · p. 16

D. Preprocessing/cleaning/labeling

E. Uses

The dataset is intended for self-supervised pre-training and downstream Earth-observation applications, and is publicly distributed with ongoing author maintenance. Sentinel-1 orbit directions are not unified, a consideration for SAR applications.

E. Uses

  • The paper uses the dataset to benchmark several self-supervised methods on downstream tasks, with related information to be maintained in a repository.Source: E. Uses · p. 16
  • SSL4EO-S12 is intended for self-supervised pre-training and downstream applications, including image retrieval, domain adaptation, and style transfer.It can also serve as a baseline for further Earth-observation pre-training datasets.Source: E. Uses · p. 16
  • Sentinel-1 ascending and descending orbits are not unified, so SAR applications should account for orbit direction; metadata records it for further processing.Source: E. Uses · p. 16

F. Distribution

  • The dataset is publicly available as a tarball on mediaTUM, with access provided through the project repository; distribution began in June 2022 under CC-BY.The passages report no third-party IP-based or other restrictions and no export-control restrictions.Source: F. Distribution · pp. 16–17

G. Maintenance

  • The dataset is hosted by mediaTUM and maintained by the authors, who will post and version updates.Source: G. Maintenance · p. 17
  • Older versions may remain hosted or be reproducible from the newest version through an update log, depending on the updates.Others can contact the authors to contribute to extensions or augmentations.Source: G. Maintenance · p. 17

H. Author statement of responsibility

Related papers

Loading 2211.07044v2…