Source-linked AI summary
SEN12MS -- A Curated Dataset of Georeferenced Multi-Spectral Sentinel-1/2 Imagery for Deep Learning and Data Fusion
Michael Schmitt, Lloyd Haydn Hughes, Chunping Qiu, Xiao Xiang Zhu
TL;DR
Remote-sensing deep learning needs large, curated datasets, but existing resources are limited in coverage, diversity, or sample count. SEN12MS combines Sentinel-1 dual-polarimetric SAR, Sentinel-2 multi-spectral imagery, and MODIS land-cover maps into 180,662 curated triplets. Its global, multi-season coverage supports remote-sensing learning applications, while sample size remains a scope limitation relative to highly versatile datasets.
Problem
Remote-sensing deep learning lacks sufficiently large, diverse, and broadly covered curated datasets for developing well-generalizing models.
Method
SEN12MS constructs georeferenced triplets of Sentinel-1 dual-polarimetric SAR, Sentinel-2 multi-spectral imagery, and MODIS-derived land-cover maps.
Results
180,662 curated patch triplets provide large patch size, global scene distribution, and diverse remote-sensing information.
Takeaways & Limitations
The dataset is intended to support well-generalizing machine-learning models and sophisticated automatic analysis of Sentinel satellite data.
Takeaways & Limitations
Despite its versatility, SEN12MS has fewer samples than datasets such as ImageNet, and northern-hemisphere seasonal labels have limited semantic value without climatic metadata.
Abstract
from arXiv · showhide
The availability of curated large-scale training data is a crucial factor for the development of well-generalizing deep learning methods for the extraction of geoinformation from multi-sensor remote sensing imagery. While quite some datasets have already been published by the community, most of them suffer from rather strong limitations, e.g. regarding spatial coverage, diversity or simply number of available samples. Exploiting the freely available data acquired by the Sentinel satellites of the Copernicus program implemented by the European Space Agency, as well as the cloud computing facilities of Google Earth Engine, we provide a dataset consisting of 180,662 triplets of dual-pol synthetic aperture radar (SAR) image patches, multi-spectral Sentinel-2 image patches, and MODIS land cover maps. With all patches being fully georeferenced at a 10 m ground sampling distance and covering all inhabited continents during all meteorological seasons, we expect the dataset to support the community in developing sophisticated deep learning-based approaches for common tasks such as scene classification or semantic segmentation for land cover mapping.
1. INTRODUCTION
SEN12MS addresses the need for large, curated remote-sensing datasets suitable for deep learning by extending an earlier Sentinel-1/2 dataset with full multi-spectral, geocoded imagery.
- Large curated datasets are crucial for training remote-sensing machine-learning models with strong generalization performance.
- Remote sensing lacks massive labeled imagery databases comparable to computer-vision resources such as ImageNet.
- The earlier SEN1-2 dataset provided about 280,000 corresponding Sentinel-1 SAR and Sentinel-2 optical image pairs.
- SEN12MS is designed for remote-sensing needs by providing full multi-spectral information in geocoded imagery.
2. THE DATA BASIS
The dataset combines freely available Sentinel-1 SAR, Sentinel-2 multi-spectral imagery, and MODIS land-cover information into a multi-sensor geospatial data basis.
- SEN12MS combines Sentinel-1 SAR imagery, Sentinel-2 multi-spectral imagery, and MODIS-derived land-cover information.
- Sentinel-1 provides weather-independent C-band SAR acquisition, dual polarization, and short revisit times for time-series applications.
- The dataset uses Sentinel-1 ground-range-detected products in interferometric wide-swath mode, retaining both VV and VH polarization channels.
- Sentinel-2 contributes full multi-spectral image cubes from precisely georeferenced granules, with cloud avoidance handled through mosaicking.
- MODIS supplies four-band land-cover patches from 2016 data, upsampled from 500 m ground sampling distance to 10 m pixel spacing.
3. GOOGLE EARTH ENGINE FOR DATA PREPARATION
Google Earth Engine supports large-scale multi-sensor patch generation through random regional sampling, cloud-free Sentinel-2 mosaicking, standardized export, and expert curation.
- The preparation pipeline uses Google Earth Engine and random sampling of regions of interest across land masses and global urban areas.
- Cloud-free Sentinel-2 mosaicking: Cloud-free Sentinel-2 generation consists of querying images, assigning cloud and shadow quality scores, and merging the best images into mosaics.
- Cloud-free Sentinel-2 mosaicking: Sentinel-1 and MODIS data require no cloud mosaicking because they are not affected by clouds.
- Data export: Prepared sensor data and land-cover information are exported together as GeoTIFFs at a 10 m scale and transformed into local UTM coordinates.
- Data curation: Expert inspection discarded scenes and patches containing large no-data areas, clouds, or artifacts before finalizing the dataset.
4. THE SEN12MS DATASET
SEN12MS contains 180,662 georeferenced triplets combining Sentinel-1 SAR, Sentinel-2 multispectral imagery, and MODIS land-cover data. The dataset is organized by northern-hemisphere seasons and supports user-defined deterministic splits.
- Dataset contents: 180,662 patch triplets combine Sentinel-1 dual-polarization SAR, Sentinel-2 multispectral imagery, and MODIS land-cover maps.The data are stored as multi-channel GeoTIFF images and require 421.3 GiB of storage.
- Organization: The dataset branches into four season-labeled ROI groups associated with seeds 1158, 1868, 1970, and 2017.These branches correspond to spring, summer, fall, and winter under the northern-hemisphere seasonal convention.
- Organization: File names encode the ROI seed, season, data identifier, and unique patch identifier using the scheme ROIsSSSS SEASON DD pXXX.tif.The identifiers s1, s2, and lc distinguish Sentinel-1, Sentinel-2, and land-cover files.
- Caveat: The accompanying seasons.csv file enables climatic-season sub-structuring because the directory seasons use northern-hemisphere reference seasons.This metadata distinguishes the actual climatic acquisition seasons of scenes.
- Usage: Users can create deterministic disjoint training and test sets from meteorological seasons or individual ROIs.The dataset is distributed under the CC-BY license through a persistent TUM link.
- Dataset contents: Example triplets present false-color Sentinel-1, Sentinel-2 RGB and SWIR, IGBP land cover, and LCCS land cover layers.All patches share a 10 m ground sampling distance, although source resolutions range from 10 m to 500 m.
5. APPLICATION TO LAND COVER MAPPING
The paper demonstrates SEN12MS for land-cover mapping with classification and semantic-segmentation networks. On held-out Munich and Rome areas, predictions increased map detail while preserving overall agreement with MODIS-derived labels.
- Experimental setup: Table 3 reports the baseline network training configurations used for the land-cover mapping experiments.The configurations accompany the ResNet-110 classification setup described for the baseline experiment.
- Classification: ResNet-110 performs patch classification from 64×64 Sentinel-2 patches using ten spectral bands and majority LCCS labels.The experiment uses the summer subset and assigns one scene label to each patch.
- Semantic segmentation: DenseNet performs semantic segmentation on full-sized 256×256 ten-band Sentinel-2 patches for a Rome test area.The network assigns an LCCS class label to every input pixel.
- Results: Both held-out test areas produced higher-resolution land-cover maps with more detail while preserving overall agreement with the low-resolution MODIS-derived maps.The test areas were not included in SEN12MS, so the examples provide an initial indication of generalization beyond the training data.
- Results: Training on only the summer subset still yielded independently evaluated accuracy metrics for the Munich and Rome experiments.The authors expect broader use of both modalities, all ROIs, and all seasons to support large-scale mapping applications.
6. DISCUSSION
SEN12MS is unusually versatile in spatial coverage, patch size, and sensor content, but its global scope is paired with fewer samples than highly broad datasets such as ImageNet. The authors nevertheless report potential for well-generalizing models and suggest transfer learning with other datasets.
- Strengths: SEN12MS covers all regions of Earth across meteorological seasons, making its scene distribution more versatile than datasets restricted to local, national, or continental areas.Its patches also contain full multispectral Sentinel-2 and dual-polarimetric Sentinel-1 data.
- Strengths: SEN12MS ranks among the five largest datasets by image-patch count, while its 256×256 patches and multisensor spectral content make it larger in data volume than competitors.The comparison contrasts SEN12MS with datasets using smaller patches or fewer bands.
- Limitations: Its global versatility also creates a challenge because SEN12MS has relatively few samples compared with ImageNet.The authors suggest combining SEN12MS with datasets such as BigEarthNet through transfer learning.
7. SUMMARY AND CONCLUSION
SEN12MS provides a large, globally distributed remote sensing dataset combining Sentinel-1, Sentinel-2, and MODIS information for machine learning research.
- 180,662 triplets combine Sentinel-1 dual-polarimetric SAR data, Sentinel-2 multispectral images, and MODIS-derived land cover maps.
- The dataset offers large patch sizes, global scene distribution, and diverse remote sensing information.
- The authors position SEN12MS as the largest remote sensing dataset available to date.