Source-linked AI summary
Multi-Sensor Data Fusion for Cloud Removal in Global and All-Season Sentinel-2 Imagery
Patrick Ebel, Andrea Meraner, Michael Schmitt, Xiaoxiang Zhu
TL;DR
Cloud-removal methods have lacked large, heterogeneous data for testing generalization across regions, seasons, and cloud coverage. The paper releases a global co-registered SAR-optical dataset and proposes a cloud-conditioned generative model; real-data training consistently outperforms synthetic-data training on real observations.
Problem
Existing cloud-removal studies are often region-specific, while large-scale datasets spanning global geography, seasons, and varied cloud coverage are lacking.
Method
The paper releases a global dataset of cloudy, cloud-free, and co-registered SAR observations and proposes a generative model conditioned on cloudy optical data and cloud coverage.
Results
Networks trained on real observations consistently outperform models trained on synthetic observations when tested on real data.
Takeaways & Limitations
The results underline the need for carefully curated real-observation datasets and flexible cloud-removal approaches covering clear to densely clouded cases.
Takeaways & Limitations
For complete coverage by large, dense clouds, reconstructed information can mismatch the target, such as predicting urban-like landcover instead of agricultural areas.
Abstract
from arXiv · showhide
This work has been accepted by IEEE TGRS for publication. The majority of optical observations acquired via spaceborne earth imagery are affected by clouds. While there is numerous prior work on reconstructing cloud-covered information, previous studies are oftentimes confined to narrowly-defined regions of interest, raising the question of whether an approach can generalize to a diverse set of observations acquired at variable cloud coverage or in different regions and seasons. We target the challenge of generalization by curating a large novel data set for training new cloud removal approaches and evaluate on two recently proposed performance metrics of image quality and diversity. Our data set is the first publically available to contain a global sample of co-registered radar and optical observations, cloudy as well as cloud-free. Based on the observation that cloud coverage varies widely between clear skies and absolute coverage, we propose a novel model that can deal with either extremes and evaluate its performance on our proposed data set. Finally, we demonstrate the superiority of training models on real over synthetic data, underlining the need for a carefully curated data set of real observations. To facilitate future research, our data set is made available online
I. INTRODUCTION
The paper addresses limited geographic and seasonal generalization in cloud-removal research by releasing a global, heterogeneous SAR-optical dataset and proposing a model evaluated on real observations.
- Cloud-removal methods are often evaluated on narrowly defined regions, while land-cover statistics and cloud coverage vary across continents and seasons.
- The released dataset contains over 100,000 samples from more than 100 regions spanning all continents and meteorological seasons.
- The work uses co-registered SAR and optical observations because Sentinel-1 SAR is not affected by bad weather or lack of daylight.
- The proposed generative architecture achieves competitive performance on metrics of generated-image goodness and diversity.
- Experiments show that synthetic cloudy-image data is a poor substitute for real cloud-coverage data.
II. METHODS
The method performs SAR-optical cloud removal with a continuous cloud-probability mask, avoiding strict pixelwise alignment while guiding reconstruction toward cloudy regions.
- The model uses SAR-optical data fusion to reconstruct cloud-occluded optical information.
- The architecture integrates SAR with optical observations without requiring strict pixelwise one-to-one correspondences between cloudy and non-cloudy images.
- An auxiliary cloud-map regression loss encourages sparse reconstructions that minimize modifications to cloud-free areas.
- A continuous-valued cloud mask conditions the model and represents each pixel’s probability of being cloud-covered.
B. Architecture
The architecture uses cycle-consistent bidirectional generators with cloud-map conditioning, residual learning, and discriminators trained on non-cloudy optical imagery. A residual regressor promotes sparse changes outside cloudy regions, while the target-domain discriminator encourages cloud-free synthesis.
- Bidirectional translation: Two generators translate images bidirectionally between source domain S1 and target domain S2, following a cycle-consistent GAN architecture.Within-domain identity mappings preserve the input image’s sensor characteristics.
- Residual mapping: The generator learns a residual mapping from the cloudy S2 input, after inverse-hyperbolic-tangent preprocessing of long-skipped pixels.The residual formulation focuses the learned change on cloud-related content.
- Cloud-map conditioning: A regression layer predicts the cloud map from the residual maps, enforcing their relation to cloud conditioning and encouraging near-zero residuals over non-cloudy areas.The regressor uses a 3×3 convolution that maps the generated three-dimensional image to a single-channel map with few additional parameters.
- Adversarial supervision: The target-domain discriminator is conditioned on cloud-probability maps and receives unpaired non-cloudy optical images to learn cloud-free patch statistics.These learned statistics implicitly force the S1-to-S2 generator to synthesize cloud-free images.
C. Losses
The model uses cloud-probability-weighted losses to preserve cloud-free regions while recovering cloudy areas, combining adversarial, cycle-consistency, identity, and auxiliary objectives.
- Cloud-probability masks weight reconstruction so cloud-free regions remain unchanged while cloudy areas are recovered using S1 information.
- The total loss combines adversarial, cycle-consistent, identity, and auxiliary cloud-map regression terms.The stated weights are λadv = 5.0, λcyc = 10.0, λidt = 1.0, and λaux = 10.0.
- Cycle-consistency and identity losses are weighted pixel-wise with the cloud map to constrain mappings and limit unnecessary changes.
- The auxiliary loss enforces sparse learned residual maps so non-cloudy S2 pixels receive little or no adjustment.
- The GS1→S2 generator receives S1, the cloud map m, and S2, while S2 is long-skip forwarded through a learned residual map.
A. Data
SEN12MS-CR is a globally distributed, multi-season cloud-removal dataset pairing cloudy and cloud-free Sentinel-2 observations with Sentinel-1 data across diverse cloud conditions.
- SEN12MS-CR contains 169 non-overlapping regions of interest distributed across all continents and meteorological seasons.The regions average approximately 52 × 40 km2 of ground coverage.
- The dataset extends SEN12MS by adding co-registered cloudy Sentinel-2 images acquired close in time to the original cloud-free observations and Sentinel-1 data.
- Mean cloud coverage is 47.93 ± 36.08 percent, with coverage varying from nearly cloud-free to entirely cloud-covered samples.
- Cloud coverage is relatively uniformly distributed, with slightly more samples showing almost no clouds or complete cloud cover.
- Test imagery illustrates four seasons and diverse scenes, with each region yielding over 700 patches of 256 × 256 px2 observations.
B. Experiments and Results
The experiments vary supervision, compare against baselines, and test synthetic-data training on real observations while evaluating cloud-removal quality and diversity with complementary metrics.
- Three experiments vary supervised training, compare the proposed model with baselines, and assess whether synthetic-data training generalizes to real data.
- Improved precision measures generated-image quality, while improved recall measures generated-image diversity and target-distribution coverage.
- The F1 score combines precision and recall into a single measure using their harmonic mean.
- Synthetic-data evaluation additionally uses MAE, RMSE, PSNR, SSIM, and SAM because simulated pairs provide pixelwise correspondence.
- MAE and RMSE quantify pixel-level deviations, while PSNR, SSIM, and SAM assess reconstruction quality, structural differences, and spectral similarity.
2) Quantifying the benefits of paired data:
The study compares training with varying amounts of paired cloudy–cloud-free data and finds that additional paired supervision improves cloud-removal performance, with the fully paired model performing best on key metrics.
- Increasing the percentage of paired training data improves the resulting cloud-removal performance.
- The paired-training experiment combines pixel-wise L1 and perceptual feature and style losses with the cycle-consistent architecture.
- Model ours-100 performs best in terms of precision and F1 score among the evaluated baselines and models.
- Raw S1 and cloudy S2 observations provide lower-bound baselines, although raw observations perform relatively poorly except on recall.
3) Model ablation experiment:
The ablation study examines cloud-sensitive components and compares model performance with baseline observations, showing that the full model benefits from the cloud probability masks and cloud-sensitive loss.
- The ablation study tests the effectiveness of the cloud detection mechanism and local cloud-sensitive loss by omitting the cloud-sensitive terms.
- Model ours-100 performs best in terms of precision and F1 score, while the raw observations generally perform poorly except on recall.
- Direct comparison with cloud-free optical targets can disadvantage modalities such as SAR because their observations differ from the target modality.
4) Assessing the goodness of synthetic data:
The synthetic-data experiment evaluates Perlin-noise and copy-paste cloud simulations against real observations and shows that synthetic test performance overestimates real-data performance.
- Synthetic training data are evaluated using the ours-0 and ours-100 architectures on synthetic and real test data.
- The proposed simulation modifications aim to preserve real-observation spectral properties and cloud-coverage statistics while supporting semi-transparent through fully occluded clouds.
- Synthetic-trained models perform better on synthetic test data than on real test data for all simulation approaches.
- The performance drop from synthetic to real testing is smaller for copy-paste data than for Perlin-noise data, which more closely resembles real cloud statistics and spectral distributions.
- Models trained on real observations consistently outperform models trained on synthetic observations when tested on real data.
IV. DISCUSSION
The discussion presents a global multimodal data set and a cloud-removal model designed for varied cloud coverage, while identifying real-data training as important for performance on real observations.
- The work provides a large-scale global cloud-removal data set and a model for recovering cloud-covered optical information.
- The data set includes full available optical spectral bands and information from both modalities to support future research.
- The data set combines over a hundred-thousand paired cloudy, cloud-free, and co-registered SAR sample triplets.
- Cloud coverage is relatively uniformly distributed, with clear and widely or densely occluded images occurring with similar probability.
- The proposed network explicitly models cloud coverage to retain cloud-free information and recover information in widely or densely cloud-covered areas.
- The evaluation uses metrics covering prediction quality and target-distribution coverage, and real-data-trained networks outperform synthetic-data-trained networks on real observations.
APPENDIX A CLOUD DETECTION
The cloud detection pipeline produces continuous-valued cloud probability masks from cloudy optical observations, supporting cloud-aware reconstruction and evaluation.
- The pipeline provides continuous-valued cloud masks for exemplary cloudy optical observations.The masks are predicted by the paper’s cloud detection pipeline.
- Improved precision and recall compare real and generated cloud-free image distributions in a high-dimensional feature space.Samples are mapped through an auxiliary pretrained network before evaluating distributional similarity.
- The evaluation uses |Φ| = 7893 and k = 10 to account for overlapping neighboring samples when computing manifold-based metrics.The setting removes the paired target and eight overlapping samples from neighborhood calculations.
APPENDIX C CLOUD COVERAGE STATISTICS ON TEST SPLIT
The test split has cloud-coverage statistics resembling the complete data set, with approximately 50% average occlusion, while challenging reconstruction cases remain possible.
- The test split’s cloud-coverage distribution is considerably alike to that of the complete data set.Its histogram is less smooth because the test split is smaller.
- Strict correspondence between cloudy references and cloud-free targets may be difficult when landcover, atmosphere, acquisition time, or seasonality changes.The data set minimizes these differences through close-in-time observation selection, but synthetic data may be needed to guarantee correspondence.
- Approximately 50% average occlusion is observed in the test split.The figure reports the average cloud coverage for the test split.
- Dense complete cloud coverage can produce reconstructions that do not match the target image, including agricultural areas predicted as urban-like landcover.The paper identifies reconstruction of cloud-covered information as particularly challenging in this setting.