Source-linked AI summary
Cloud removal in remote sensing images using generative adversarial networks and SAR-to-optical image translation
Faramarz Naderi Darbaghshahi, Mohammad Reza Mohammadi, Mohsen Soryani
TL;DR
Cloud contamination makes optical-image reconstruction challenging, particularly for thick clouds. The paper uses two GANs to translate SAR images into optical images and then remove clouds, and reports better results than state-of-the-art methods on both tasks.
Problem
Cloud contamination removes information from optical remote-sensing images, and large cloud regions challenge spatial reconstruction while temporal methods require short-interval cloud-free references.
Source: §1 · pp. 1–2Method
The method chains two GANs, first translating SAR to optical imagery and then using the translated images for cloud removal, with DRIB-based generators.
Source: §1 · pp. 2–3, §2.1 · p. 3Results
The proposed method achieved the highest PSNR and SSIM values in both SAR-to-optical translation and cloud removal, and outperformed other state-of-the-art methods on SEN1-2.
Source: §4 · p. 12, §5 · p. 12Takeaways & Limitations
The authors report that the method removes clouds while keeping non-cloudy areas unchanged.
Source: §4 · p. 12
Abstract
from arXiv · showhide
Satellite images are often contaminated by clouds. Cloud removal has received much attention due to the wide range of satellite image applications. As the clouds thicken, the process of removing the clouds becomes more challenging. In such cases, using auxiliary images such as near-infrared or synthetic aperture radar (SAR) for reconstructing is common. In this study, we attempt to solve the problem using two generative adversarial networks (GANs). The first translates SAR images into optical images, and the second removes clouds using the translated images of prior GAN. Also, we propose dilated residual inception blocks (DRIBs) instead of vanilla U-net in the generator networks and use structural similarity index measure (SSIM) in addition to the L1 Loss function. Reducing the number of downsamplings and expanding receptive fields by dilated convolutions increase the quality of output images. We used the SEN1-2 dataset to train and test both GANs, and we made cloudy images by adding synthetic clouds to optical images. The restored images are evaluated with PSNR and SSIM. We compare the proposed method with state-of-the-art deep learning models and achieve more accurate results in both SAR-to-optical translation and cloud removal parts.
1. Introduction
Clouds complicate optical remote sensing, especially where reconstruction methods lack suitable auxiliary or temporal reference data. The paper proposes a two-GAN approach with DRIB generators and reports stronger results than state-of-the-art methods in both translation and cloud removal.
- Clouds cover around 67% of Earth’s surface, motivating cloud removal for optical remote-sensing applications.Source: §1 · p. 1
- Spatial methods can fail on large cloud regions, temporal methods require short-interval cloud-free references, and SAR can pass through clouds.Source: §1 · p. 2
- The model uses one GAN to learn SAR image structures and another to remove clouds while retaining cloud-free areas.Source: §1 · pp. 2–3
- The proposed model outperformed state-of-the-art deep-learning methods in both SAR-to-optical translation and cloud removal.Source: §1 · p. 3
- The generators use dilated residual inception blocks to expand receptive fields and help prevent information loss.Source: §1 · p. 3
- The model was trained and tested on SEN1-2 and compared with state-of-the-art deep-learning cloud-removal methods.Source: §1 · p. 3
2. Methods
The method separates SAR-to-optical translation from cloud removal, using the translated images as an auxiliary input for cloud reconstruction. Both generators use DRIB-based encoder–decoder networks and a combined adversarial, L1, and SSIM objective, trained and tested on SEN1-2 data.
2. Methods
- Two GANs separate SAR-to-optical translation from cloud removal, letting translation training use SAR–optical pairs without depending on limited co-registered cloudy/cloud-free pairs.Source: §2 · p. 3
2.1. Network Architecture
- Both generators use encoder–DRIB–decoder structures with eight DRIBs as bottlenecks and symmetrical encoder–decoder concatenations that provide local information to upsampled outputs.Source: §2.1 · p. 3
- DRIBs combine dilated convolutions, residual connections, and inception modules; this design halves generator parameters versus U-net while raising performance.Dilated convolutions expand receptive fields while preserving spatial resolution.Source: §2.1 · p. 5
2.2. Objective
- The authors add SSIM loss to the pix2pix loss and use L1 loss to reduce blurring and bring output images closer to target images; stated values are λ1 = 100 and λ2 = 100.Source: §2.2 · p. 6
2.3. Training and testing
- SEN1-2 provides 256 × 256-pixel co-registered Sentinel-1 SAR and Sentinel-2 optical patches; cloud removal uses thick synthetic clouds alpha-blended onto optical images.The translation GAN used 18,712 training and 4,678 test pairs; the cloud-removal GAN used 2,000 training and 500 test pairs.Source: §2.3 · p. 7
3. Experimental results
The section outlines the training details, evaluation metrics, comparisons, and investigation of DRIB and SSIM-loss effects.
- The section proceeds from training and evaluation metrics to model comparisons, then examines DRIBs and SSIM loss.Source: §3 · p. 7
3.1. Implementation details
The implementation specifies shared GAN hyperparameters and discriminator input sizes, alongside dataset examples showing optical, SAR, and cloudy images.
- Both GANs use a learning rate of 2−4, batch size 32, dropout 0.5, normally initialized weights, and Adam with β1 = 0.5 and β2 = 0.999.Source: §3.1 · p. 7
- Figure 5 shows optical images on the left, SAR images in the center, and cloudy images on the right.Source: Figure 5 · p. 8
- The cloud-removal discriminator takes 256 × 256 × 6 inputs, while its output-side inputs are 256 × 256 × 3.Source: §3.1 · p. 8
3.2. Metrics
The study evaluates SAR-to-optical translation and cloud removal using SSIM and PSNR.
- Both SAR-to-optical translation and cloud removal are evaluated with SSIM and peak signal-to-noise ratio (PSNR).Source: §3.2 · p. 9
- In the PSNR definition, MAXI denotes the maximum possible pixel value of the image.Source: §3.2 · p. 9
3.3. Performance comparison
The comparison evaluates the proposed method against prior approaches in SAR-to-optical translation and cloud removal, using quantitative tables and example outputs.
3.3. Performance comparison
- The performance comparison covers state-of-the-art approaches in both SAR-to-optical translation and cloud removal.Source: §3.3 · p. 9
3.3.1. SAR-to-optical image translation
- SAR-to-optical translation is compared with Bermudez et al. (2018), Enomoto et al. (2018), and Li et al. (2020), with quantitative results in Table 3.Figure 6 displays SAR inputs, each model's output, and target images in separate columns.Source: §3.3.1 · p. 9, Table 3 · p. 9, Figure 6 · p. 10
3.3.2. Cloud removal
- Cloud removal is compared with the SAR-and-optical fusion GANs of Grohnfeldt et al. (2018) and Ebel et al. (2020), with PSNR and SSIM reported in Table 4.PSNR is measured separately in cloudy and non-cloudy areas to assess cloud-removal performance and preservation of clear regions.Source: §3.3.2 · p. 9, Table 4 · p. 9
3.4. Ablation study
The ablation study examines the SAR-to-optical translation step, generator architecture, and SSIM loss. It reports that adding SSIM loss increases both PSNR and SSIM.
3.4.1. Effect of SAR-to-optical image translation step and DRIBs
- The architecture ablation replaced DRIBs with U-net, while the translation ablation supplied cloudy and SAR images directly to the cloud-removal generator.The authors report that the tested model parts affect output-image quality.Source: §3.4.1 · p. 10, Table 5 · p. 11
- Table 5 presents performance for different generator architectures, while Table 6 presents performance for different loss functions.The supplied passages do not state the tables’ individual cell values.Source: Table 5 · p. 11, Table 6 · p. 12
3.4.2. Effect of SSIM loss function
- Adding the SSIM loss function increased both PSNR and SSIM values.Table 6 reports the investigation using different loss functions.Source: §3.4.2 · p. 12, Table 6 · p. 12
4. Discussion
The discussion reports that the proposed model outperforms state-of-the-art methods on both SAR-to-optical translation and cloud removal. It attributes the performance to two GANs, network-structure changes, and SSIM loss.
- The proposed model achieved the highest PSNR and SSIM values in both SAR-to-optical translation and cloud removal.The discussion identifies Tables 3 and 4 as the supporting comparisons.Source: §4 · p. 12
- PSNR results indicate that the model removes clouds while keeping non-cloudy areas unchanged.Source: §4 · p. 12
- The discussion attributes performance to two GANs, network-structure changes, and adding SSIM loss to the classical pix2pix loss.It also states that Table 5 confirms DRIBs’ performance and that SSIM loss increased PSNR and SSIM in Table 6.Source: §4 · p. 12
5. Conclusions
The paper proposes a two-GAN cloud-removal model using SAR images and reports superiority over state-of-the-art methods on SEN1-2. It identifies real-cloud imagery and unpaired-image removal as future directions.
- On SEN1-2, the proposed method outperformed state-of-the-art methods in both SAR-to-optical translation and cloud removal.Source: §5 · p. 12
- The proposed cloud-removal model uses two GANs and SAR images.Source: §5 · p. 12
- Future work could use real cloudy images for cloud removal and classification, and address cloud removal from unpaired images.Source: §5 · p. 12