Source-linked AI summary
The SEN1-2 Dataset for Deep Learning in SAR-Optical Data Fusion
Michael Schmitt, Lloyd Haydn Hughes, Xiao Xiang Zhu
TL;DR
Remote sensing lacks large, aligned datasets for learning across heterogeneous SAR and optical sensors. This paper releases SEN1-2, a globally distributed, season-spanning dataset of 282,384 corresponding patch-pairs, demonstrates applications including colorization and matching, and reports 93% matching accuracy on a test subset.
Problem
Large, accurately aligned multi-sensor datasets are difficult and costly to gather, limiting training data for SAR-optical data-fusion research.
Method
The paper constructs and releases SEN1-2 from Sentinel-1 and Sentinel-2 imagery, using tiled and manually inspected corresponding patches for data-fusion applications.
Results
SEN1-2 contains 282,384 quality-controlled SAR-optical patch-pairs, and a pseudo-siamese matching model identifies corresponding patches with 93% accuracy on a test subset.
Takeaways & Limitations
SEN1-2 provides a large, globally and seasonally diverse data source for deep learning in SAR-optical data fusion and remote sensing.
Takeaways & Limitations
The dataset uses only RGB Sentinel-2 imagery and Level-1C top-of-atmosphere reflectances rather than full multispectral, atmospherically corrected data.
Abstract
from arXiv · showhide
While deep learning techniques have an increasing impact on many technical fields, gathering sufficient amounts of training data is a challenging problem in remote sensing. In particular, this holds for applications involving data from multiple sensors with heterogeneous characteristics. One example for that is the fusion of synthetic aperture radar (SAR) data and optical imagery. With this paper, we publish the SEN1-2 dataset to foster deep learning research in SAR-optical data fusion. SEN1-2 comprises 282,384 pairs of corresponding image patches, collected from across the globe and throughout all meteorological seasons. Besides a detailed description of the dataset, we show exemplary results for several possible applications, such as SAR image colorization, SAR-optical image matching, and creation of artificial optical images from SAR input data. Since SEN1-2 is the first large open dataset of this kind, we believe it will support further developments in the field of deep learning for remote sensing as well as multi-sensor data fusion.
1. INTRODUCTION
The paper addresses the need for large, aligned multi-sensor datasets to advance deep learning for SAR-optical data fusion. It introduces SEN1-2, a globally sampled dataset of 282,384 Sentinel-1/Sentinel-2 patch pairs collected across all four seasons.
- Large, perfectly aligned SAR-optical datasets are needed to develop deep learning methods for multi-sensor data fusion.Collecting such data is technically difficult and remote sensing imagery has historically been expensive.
- Sentinel-1’s free data availability through the Copernicus program changed access to large remote sensing datasets.
- SEN1-2 provides 282,384 corresponding Sentinel-1/Sentinel-2 SAR-optical patch pairs.The dataset spans locations across the land masses of Earth and all four meteorological seasons.
- The paper describes SEN1-2’s generation, characteristics, features, and pilot applications.
2. SENTINEL-1/2 REMOTE SENSING DATA
The dataset combines Sentinel-1 SAR and Sentinel-2 optical observations, which measure different physical properties of Earth scenes. The authors use selected sensor products and restrict the optical data to RGB channels and the SAR data to VV polarization.
- Sentinel-1 uses C-band SAR and can acquire imagery regardless of weather.
- The Sentinel-1 dataset uses GRD products acquired in IW mode with VV polarization and 5 m azimuth and 20 m range pixel spacing.
- No speckle filtering was applied so end users can adapt preprocessing to their tasks.
- The Sentinel-2 dataset uses red, green, and blue channels—bands 4, 3, and 2—and initially selects granules with cloud coverage no greater than 1%.
3. THE DATASET
The dataset is generated as a multi-sensor collection of well-aligned SAR-optical patch pairs. Google Earth Engine provides the data catalogue and computing environment used to acquire and prepare the imagery.
- Generating the dataset requires substantial remote sensing data with very good spatial alignment between SAR and optical observations.
- Google Earth Engine is used to generate the patch-pair dataset in a mostly automatic manner.
3.1 Data Preparation in Google Earth Engine
The authors use Google Earth Engine to sample global scenes, select seasonally matched Sentinel imagery, mosaic and export aligned data, and manually remove defective scenes and patches. Tiling and quality control produce 282,384 final patch pairs.
- 3.1 Data Preparation in Google Earth Engine: Google Earth Engine supplies a multi-petabyte imagery catalogue and programming interface for selecting, preparing, and downloading Sentinel data.
- 3.1 Data Preparation in Google Earth Engine: Sampling combines globally uniform land-mass points with an artificial urban bias to represent diverse and visually complex environments.
- 3.1 Data Preparation in Google Earth Engine: Four seed values are used for random ROI sampling, and the resulting distributions are illustrated in Figure 2a.
- 3.1 Data Preparation in Google Earth Engine: The imagery is organized into winter, spring, summer, and fall periods to select recent 2017 observations.
- 3.1 Data Preparation in Google Earth Engine: Candidate scenes are filtered by Sentinel-2 cloud coverage and Sentinel-1 IW/VV availability, reducing approximately 600 ROIs to about 429.
- 3.1 Data Preparation in Google Earth Engine: Mosaicking and clipping create one image per ROI, using Sentinel-2 bands 4, 3, and 2 for RGB imagery.
- 3.1.5 First Manual Inspection: 258 scenes/ROIs remain after visual removal of severe no-data, cloud, and color problems.
- 3.1.6 Tiling: 298,790 Sentinel-1/Sentinel-2 patch pairs are produced by tiling 256×256-pixel images with a stride of 128.
3.2 Dataset Availability
SEN1-2 is openly available under the CC-BY license through a persistent Technical University of Munich repository link.
- The dataset is shared under the open access CC-BY license.
- Researchers must cite this paper when using SEN1-2 for research.
4. EXAMPLE APPLICATIONS
The paper demonstrates SEN1-2 across SAR colorization, multimodal matching, and artificial optical-image generation, using deep learning models trained on dataset patch pairs.
- 4.1 Colorizing Sentinel-1 Images: SAR colorization combines SAR-optical fusion training examples with a variational autoencoder and mixture density network to learn conditional color distributions.
- 4.2 SAR-optical Image Matching: Multimodal matching supports applications including image co-registration, 3D stereo reconstruction, and change detection.
- 4.2 SAR-optical Image Matching: 93% accuracy was achieved in identifying corresponding SAR-optical patches with a pseudo-siamese convolutional neural network.
- 4.3 Generating Artificial Optical Images from SAR Inputs: 108,221 SEN1-2 patch pairs were used to train pix2pix GAN examples predicting artificial optical imagery from SAR inputs.
5. STRENGTHS AND LIMITATIONS OF THE DATASET
SEN1-2 offers large-scale, globally distributed, seasonally varied co-registered SAR-optical data, while limiting Sentinel-2 inputs to RGB top-of-atmosphere reflectances.
- Strengths: SEN1-2 provides 282,384 co-registered patch pairs distributed across the globe and all meteorological seasons.
- Strengths: SEN1-2 can be split deterministically by scene or season to create independent training and testing datasets for evaluations on unseen data.
- Strengths: Compared with SARptical’s 10,000 patches from one scene, SEN1-2 supplies a substantially larger and geographically broader dataset.
- Limitations: Sentinel-2 data are restricted to RGB channels, which may be insufficient for exploiting the full radiometric bandwidth of multispectral imagery.
- Limitations: The dataset uses Sentinel-2 Level-1C top-of-atmosphere reflectances rather than atmospherically corrected bottom-of-atmosphere information.
6. SUMMARY AND CONCLUSION
The paper releases SEN1-2, a dataset of 282,384 paired SAR and optical image patches from Sentinel-1 and Sentinel-2 scenes. It is intended to support machine-learning research in satellite remote sensing and SAR-optical data fusion.
- 282,384 pairs of SAR and optical image patches comprise the released SEN1-2 dataset.
- The patch pairs are extracted from versatile Sentinel-1 and Sentinel-2 scenes.
- The authors assume SEN1-2 will foster machine-learning, especially deep-learning, approaches in satellite remote sensing and SAR-optical data fusion.