PaperlayerView exact revision on arXiv

Source-linked AI summary

SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal

Patrick Ebel, Yajin Xu, Michael Schmitt, Xiaoxiang Zhu

arXiv:2201.09613v1cs.CVeess.IV

TL;DR

Clouds and haze limit continuous optical satellite monitoring, motivating cloud removal methods that use both temporal and cross-sensor information. The paper introduces SEN12MS-CR-TS and evaluates sequence-to-point and sequence-to-sequence models, finding benefits from multi-modal and multi-temporal observations for reconstructing cloud-covered imagery.

  • Problem

    Clouds and haze hinder continuous optical satellite monitoring and impair remote-sensing applications such as classification, segmentation, and change detection.

  • Method

    The paper curates a globally distributed dataset of paired Sentinel-1 radar and Sentinel-2 multispectral time series and evaluates sequence-to-point and sequence-to-sequence cloud-removal networks.

  • Results

    The experiments show that both approaches benefit from paired SAR measurements and that the proposed sequence-to-point network outperforms most baselines across evaluated metrics.

  • Takeaways & Limitations

    SEN12MS-CR-TS supports research on global, all-season multi-modal and multi-temporal cloud removal and is released to facilitate future image-reconstruction research.

Abstract

from arXiv · show

About half of all optical observations collected via spaceborne satellites are affected by haze or clouds. Consequently, cloud coverage affects the remote sensing practitioner's capabilities of a continuous and seamless monitoring of our planet. This work addresses the challenge of optical satellite image reconstruction and cloud removal by proposing a novel multi-modal and multi-temporal data set called SEN12MS-CR-TS. We propose two models highlighting the benefits and use cases of SEN12MS-CR-TS: First, a multi-modal multi-temporal 3D-Convolution Neural Network that predicts a cloud-free image from a sequence of cloudy optical and radar images. Second, a sequence-to-sequence translation model that predicts a cloud-free time series from a cloud-covered time series. Both approaches are evaluated experimentally, with their respective models trained and tested on SEN12MS-CR-TS. The conducted experiments highlight the contribution of our data set to the remote sensing community as well as the benefits of multi-modal and multi-temporal information to reconstruct noisy information. Our data set is available at https://patrickTUM.github.io/cloud_removal

I. INTRODUCTION

SEN12MS-CR-TS combines multi-modal radar-optical observations with temporal information to address cloud removal, extending prior approaches that typically use only one modality or one time point.

  • Clouds and haze hinder reliable, continuous optical satellite monitoring and affect land-cover classification, segmentation, and change detection.
  • Earlier methods commonly used SAR or other atmospheric-robust sensors at a single time point, while newer methods reconstructed cloudy observations across time.
  • SEN12MS-CR-TS provides one-year paired, co-registered Sentinel-1 radar and Sentinel-2 multispectral time series from regions worldwide.
  • The paper evaluates a 3D-convolution sequence-to-point model and a sequence-to-sequence model for predicting cloud-free images or time series.
  • Unlike earlier datasets relying on synthetic or limited cloud conditions, the dataset includes naturally occurring clouds and spans the complete cloud-coverage spectrum.

II. DATA

SEN12MS-CR-TS is a globally distributed, all-season dataset of paired Sentinel-1 and Sentinel-2 observations, designed to represent diverse regions, seasons, and cloud conditions.

  • The dataset contains 53 regions of interest, with 40 training regions and 13 held-out regions covering over 80,000 km2.
  • Each region provides 30 co-registered and paired Sentinel-1 and Sentinel-2 full-scene images sampled throughout 2018.
  • The regions exceed 4000×4000 px2 and represent globally distributed, heterogeneous landcover sampled across seasons.
  • The dataset includes observations ranging from clear views to dense clouds, modeling the complete spectrum of cloud coverage rather than excluding heavily clouded samples.
  • Training and testing regions are spatially separated, while the dataset’s region-of-interest distribution is visualized alongside compatible SEN12MS-CR regions.

B. Preprocessing

The preprocessing pipeline standardizes Sentinel-1 and Sentinel-2 observations into paired image patches and computes binary cloud masks for cloud-coverage analysis.

  • Full-scene images are divided into non-overlapping 256 × 256px2 patches after upsampling bands to 10m resolution.
  • Sentinel-1 data undergoes noise removal, radiometric calibration, orthorectification, and decibel transformation.
  • For ResNet inputs, Sentinel-1 VV and VH channels and Sentinel-2 patches are clipped and rescaled to specified numeric ranges.
  • The s2cloudless detector generates binary masks marking each pixel as cloud-free or cloud-covered.

III. METHODS

The paper introduces two cloud-removal models using SEN12MS-CR-TS: a sequence-to-point network that integrates multi-modal, multi-temporal observations, and a sequence-to-sequence network that predicts cloud-free image series. The latter uses internal learning to reconstruct noisy observations directly from target sequences.

  • III. METHODS: Two methods address distinct tasks: predicting one cloud-free image or a cloud-free time series from cloudy observations.The sequence-to-point model uses multi-modal inputs, while the sequence-to-sequence model preserves temporal output structure.
  • A. Multi-temporal multi-modal cloud removal: The sequence-to-point model processes each time point with Siamese ResNet branches, then applies temporal stacking and 3D convolutions.Its input is a sequence of paired Sentinel-1 and Sentinel-2 observations, and its output is one cloud-free Sentinel-2 image.
  • B. Internal Learning for sequence-to-sequence cloud removal: The sequence-to-sequence model uses a symmetric 3D U-Net-style encoder-decoder with skip connections to map Sentinel-1 sequences to cloud-free Sentinel-2 sequences.Unlike the sequence-to-point model, it preserves temporal information in its output.
  • B. Internal Learning for sequence-to-sequence cloud removal: Internal learning trains the sequence-to-sequence network from scratch on each target sequence without additional cloud-free training data.The procedure reconstructs noisy pixels by exploiting spatio-temporal regularities in the observations.
  • B. Internal Learning for sequence-to-sequence cloud removal: The sequence-to-sequence approach is related to low-rank and sparse decomposition because it separates regular signal structure from irregular cloud-related components.The paper describes land-cover correlations as low-rank structure and cloud appearance as a sparse component.

IV. EXPERIMENTS AND RESULTS

The experiments evaluate the proposed cloud-removal methods and ablations using defined quality measures and comparisons with baseline models. The section separates evaluation of sequence-to-point and sequence-to-sequence tasks.

  • IV. EXPERIMENTS AND RESULTS: The experiments assess proposed cloud-removal methods and their ablation variants.The evaluation uses measures of goodness and reports separate experiments for sequence-to-point and sequence-to-sequence tasks.
  • IV. EXPERIMENTS AND RESULTS: The sequence-to-sequence model is illustrated as a 3D encoder-decoder that maps multi-temporal Sentinel-1 inputs to multi-temporal multispectral Sentinel-2 outputs.An ablation instead uses Gaussian noise as input.
  • IV. EXPERIMENTS AND RESULTS: The experimental design includes baselines for the sequence-to-point cloud-removal task and reports outcomes for both proposed task settings.The section structure distinguishes baseline comparisons from sequence-to-point and sequence-to-sequence experiments.

A. Metrics

Performance is evaluated with pixel-level and image-level metrics that measure reconstruction discrepancy, signal quality, structural similarity, and spectral agreement. NRMSE is additionally separated across all, cloud-covered, and cloud-free pixels.

  • A. Metrics: The study evaluates NRMSE, PSNR, SSIM, and SAM to measure cloud-removal prediction quality.These metrics cover pixel discrepancy, signal-to-noise ratio, structural similarity, and spectral angle.
  • A. Metrics: SSIM captures perceived structural change, contrast, and luminance relative to the target image.Its formulation uses image means, standard deviations, covariance, and stabilizing constants.
  • A. Metrics: NRMSE measures average discrepancy between target and predicted pixels, while PSNR measures whole-image signal-to-noise ratio.The images use pixel values in [0, 1] with dimensions C = 3 and H = W = 256.
  • A. Metrics: SAM measures the spectral angle between bands of two multi-channel images.It complements image-wise measures by focusing on spectral agreement across channels.
  • A. Metrics: NRMSE is computed over all pixels, cloud-covered pixels, and cloud-free pixels to distinguish reconstruction from information preservation.Cloud masks identify pixels visible in neither or at least one input optical patch.

B. Baseline Methods

The baseline suite measures task difficulty, optical information recoverable across time, point-to-point cloud removal, and sequence-to-point performance. These comparisons test whether sophisticated learning is needed beyond simpler alternatives.

  • B. Baseline Methods: The least-cloudy baseline forwards the least-cloudy observation without modification to measure task difficulty.It is compared directly with the cloud-free target image.
  • B. Baseline Methods: The mosaicing baseline averages cloud-free pixels across time and assigns 0.5 where no clear-view observation exists.It measures how much information can be reconstructed from multi-spectral optical observations alone.
  • B. Baseline Methods: ResNet provides a point-to-point cloud-removal baseline based on a residual network trained for the relevant task.Parts of this model also serve as Siamese residual branches in the proposed architecture.
  • B. Baseline Methods: Together, the baselines test trivial solutions and contextualize more sophisticated deep-learning approaches trained on SEN12MS-CR-TS.They span least-cloudy forwarding, temporal mosaicing, point-to-point removal, and sequence-to-point removal.
  • B. Baseline Methods: STGAN is a sequence-to-point cloud-removal baseline using the Branched ResNet generator architecture.The proposed sequence-to-point network closely follows its design.

C. Sequence-to-Point Cloud Removal

The sequence-to-point model combines temporally adjacent optical observations with SAR information to reconstruct a cloud-free target image. On test samples spanning 0–100% cloud coverage, it generally outperforms simple baselines, while performance depends strongly on cloud coverage.

  • Architecture: The sequence-to-point networks initialize from a pre-trained single-temporal ResNet cloud-removal model and extend it through temporal concatenation.The models use a pre-trained cloud-removal network before combining temporal information.
  • Training: Perceptual loss uses a land-cover-pre-trained VGG16 network alongside pixel-wise loss to produce sharper reconstructions.The losses are weighted with λL1 = 100 and λperc = 1.
  • Evaluation: The proposed network outperforms baselines on most metrics, except PSNR, where mosaicing leads, and clear-pixel NRMSE, where least cloudy performs best.The comparison includes normalized root mean squared error, PSNR, SSIM, and SAM.
  • Evaluation: Evaluation covers cloud-free, partly cloudy, and heavily cloud-covered samples, including cases with no visible observation at any time point.The test split includes S2 observations across the complete 0–100% cloud-coverage range.
  • Evaluation: Reconstruction performance is strongly associated with cloud coverage, although its decrease with increasing coverage is not strictly monotonic.This dependence is reported for the sequence-to-point model with three inputs, SAR, and perceptual loss.

D. Sequence-to-Sequence Cloud Removal

The sequence-to-sequence model learns directly from each target image sequence to copy cloud-free pixels and inpaint cloud-covered ones. Experiments show that SAR guidance improves reconstruction over a noise-conditioned ablation and baseline methods, even with limited data.

  • Training approach: The model trains directly on the target image time series without external training data, learning to replicate clear pixels and inpaint cloud-covered pixels.Its objective combines reconstruction terms over the target sequence, with temporal batches of adjacent images.
  • Evaluation protocol: The evaluation protocol blends cloud-covered pixels from the most-cloudy source image with clear pixels from the least-cloud-covered target image.A cloud mask determines which source and target pixels are combined for the constructed input.
  • Results: SAR-conditioned sequence-to-sequence reconstruction outperforms the corresponding single-sensor ablation and the considered baselines.The comparison uses RMSE, PSNR, SSIM, and SAM.
  • Results: The internal learning approach reconstructs cloud-covered pixels from a very limited amount of data.The experiments also compare predictions conditioned on SAR with predictions using no meaningful prior information.

V. DISCUSSION

The discussion presents SEN12MS-CR-TS as a challenging, globally distributed dataset for multi-modal and multi-temporal cloud removal. It reports benefits from using time series and multiple sensors, while noting a size trade-off relative to the mono-temporal predecessor.

  • Dataset contribution: SEN12MS-CR-TS provides globally sampled, heterogeneous observations across seasons, including clear, filmy, and dense non-transparent clouds.The dataset is designed to represent a challenging range of cloud conditions for cloud-removal methods.
  • Evaluation: The proposed sequence-to-point and sequence-to-sequence methods use pixel-wise and image-wise metrics to evaluate cloud removal.The discussion frames these models as demonstrations of the dataset’s usefulness.
  • Findings: Time-series information facilitates cloudy-pixel reconstruction, and multi-sensor measurements further improve the quality of cloud-removed predictions.These observations support including both temporal and modality information in the dataset.
  • Dataset scope: SEN12MS-CR-TS is more than twice as large as SEN12MS-CR but contains about half as many regions of interest sampled across continents.The two datasets remain compatible because their holdout regions can be used across training, validation, and testing without geospatial overlap.
  • Dataset scope: SEN12MS-CR-TS uses non-overlapping adjacent samples, whereas SEN12MS-CR contains 25–50% overlap between neighboring patches.Both datasets have comparable cloud coverage and span semitransparent haze through thick dense clouds.

VI. CONCLUSION

The conclusion positions SEN12MS-CR-TS as a global, all-season benchmark for multi-modal and multi-temporal cloud removal. The experiments demonstrate sequence-to-point and sequence-to-sequence use cases, with benefits from temporal and SAR information.

  • Dataset: SEN12MS-CR-TS contains globally distributed Sentinel-1 and Sentinel-2 observations covering more than 80,000 km^2 of landcover across the year.Its regions of interest are large-sized and represent heterogeneous landcover.
  • Models: The paper demonstrates the dataset with sequence-to-point and sequence-to-sequence cloud-removal models.The sequence-to-sequence model preserves temporal information in cloud-removal reconstruction.
  • Findings: Both methods benefit from co-registered paired SAR measurements in the dataset.The conclusion links the experiments to the use of multi-modal information for reconstructing noisy observations.
  • Implications: The experiments highlight benefits from multi-modal and multi-temporal information for reconstructing noisy information.The dataset is made public to support future research in multi-modal and multi-temporal image reconstruction.

APPENDIX A TEMPORAL COINCIDENCE OF PAIRED OBSERVATIONS

Paired Sentinel-1 and Sentinel-2 full-scene observations are collected within a 14-day window, with an average temporal separation of 2.61 days.

  • 2.61 ± 2.41 days is the mean time difference between paired Sentinel-1 and Sentinel-2 observations, indicating close temporal proximity.The pairing interval is bounded by 14 days.
Loading 2201.09613v1…