Source-linked AI summary
Filmy Cloud Removal on Satellite Imagery with Multispectral Conditional Generative Adversarial Nets
Kenji Enomoto, Ken Sakurada, Weimin Wang, Hiroshi Fukui, Masashi Matsuoka, Ryosuke Nakamura, Nobuo Kawaguchi
TL;DR
Clouds destabilize visible-light satellite observation, while longer wavelengths trade higher cloud penetrability for lower resolution and different appearance. The paper extends cGANs to multispectral McGANs, trains on synthesized cloud-obscured RGB images with NIR inputs, and confirms cloud removal when clouds are thin enough for NIR penetration. The method remains limited for clouds that NIR cannot penetrate and for synthetic clouds that differ from real clouds.
Problem
Clouds obscure visible-light satellite imagery, while longer-wavelength imagery has lower spatial resolution and differs considerably in appearance.
Method
McGANs extends cGAN input channels to multispectral imagery, using synthesized cloud-obscured RGB images and registered NIR inputs to predict RGB outputs.
Results
McGANs removes clouds and predicts color properly when clouds are thin enough to be penetrated by NIR.
Takeaways & Limitations
The four-band RGB-plus-NIR approach demonstrates feasibility for removing NIR-penetrable clouds from visible-light satellite images.
Takeaways & Limitations
Clouds too thick for NIR penetration are not removed, and Perlin-noise simulated clouds differ from real clouds.
Abstract
from arXiv · showhide
In this paper, we propose a method for cloud removal from visible light RGB satellite images by extending the conditional Generative Adversarial Networks (cGANs) from RGB images to multispectral images. Satellite images have been widely utilized for various purposes, such as natural environment monitoring (pollution, forest or rivers), transportation improvement and prompt emergency response to disasters. However, the obscurity caused by clouds makes it unstable to monitor the situation on the ground with the visible light camera. Images captured by a longer wavelength are introduced to reduce the effects of clouds. Synthetic Aperture Radar (SAR) is such an example that improves visibility even the clouds exist. On the other hand, the spatial resolution decreases as the wavelength increases. Furthermore, the images captured by long wavelengths differs considerably from those captured by visible light in terms of their appearance. Therefore, we propose a network that can remove clouds and generate visible light images from the multispectral images taken as inputs. This is achieved by extending the input channels of cGANs to be compatible with multispectral images. The networks are trained to output images that are close to the ground truth using the images synthesized with clouds over the ground truth as inputs. In the available dataset, the proportion of images of the forest or the sea is very high, which will introduce bias in the training dataset if uniformly sampled from the original dataset. Thus, we utilize the t-Distributed Stochastic Neighbor Embedding (t-SNE) to improve the problem of bias in the training dataset. Finally, we confirm the feasibility of the proposed network on the dataset of four bands images, which include three visible light bands and one near-infrared (NIR) band.
1. Introduction
Cloud cover destabilizes visible-light satellite observation, while longer wavelengths improve cloud visibility but reduce spatial resolution and differ substantially in appearance. The paper addresses this trade-off with McGANs, which uses multispectral inputs to generate cloud-free visible-light imagery.
- 1. Introduction: Clouds make ground monitoring unstable in visible-light satellite images.
- 1. Introduction: Longer wavelengths improve visibility through clouds but reduce spatial resolution and produce imagery that differs from visible light.
- 1. Introduction: McGANs extends cGAN input channels to multispectral images and predicts RGB images from cloud-obscured RGB and registered NIR inputs.
- 1. Introduction: The training data combine simulated clouds over ground-truth RGB images with color correction to address unavailable paired cloud-free and cloud-obscured scenes.
- 1. Introduction: t-SNE sampling reduces dataset bias caused by the high proportion of forest and sea images, and feasibility is evaluated on four-band imagery.
2. Related Work
Prior work uses cloud-penetrating sensing and image-restoration networks, but these approaches face resolution, interpretability, or modality limitations. The paper combines visible-light color information with longer-wavelength penetrability, first evaluating RGB-plus-NIR cloud removal.
- 2. Related Work: SAR improves cloud visibility but retains lower spatial resolution than visible-light imagery and is difficult to interpret directly.
- 2. Related Work: RGB and RGB-NIR dehazing methods use color assumptions or NIR guidance because NIR penetrates fog better than visible light.
- 2. Related Work: GANs use generator–discriminator training, while DCGANs introduce convolutional networks and batch normalization to improve stability.
- 2. Related Work: cGAN-based restoration commonly combines U-Net generators with PatchGAN discriminators to preserve spatial features and model high-frequency details.
- 2. Related Work: The proposed direction combines visible-light color with longer-wave penetrability, beginning with RGB and NIR inputs before future SAR integration.
3. Dataset Generation for Cloud Removal
The dataset uses WorldView-2 imagery and synthetic clouds to create paired training inputs and targets, with color correction reducing tonal variation. t-SNE-based sampling then counteracts category imbalance by distributing training images across feature space.
- 3. Dataset Generation for Cloud Removal: 37,000 training patches of 256×256 pixels were extracted from eight comparatively cloudless WorldView-2 images.The source images have 0.5 m/pixel spatial resolution and 20,000 × 20,000 dimensions.
- 3.1. Synthesis of cloud-obscured images: Perlin-noise clouds are alpha-blended onto RGB images and color-corrected to synthesize cloud-obscured inputs aligned with ground truth.
- 3.1. Synthesis of cloud-obscured images: Color correction greatly improves variation in color tone between synthesized and original imagery.
- 3.2. Uniformization of the dataset with t-SNE: t-SNE sampling addresses overfitting risk caused by the dataset’s predominance of sea and forest images.
- 3.2. Uniformization of the dataset with t-SNE: 4096-dimensional AlexNet features are mapped into 2D with t-SNE, after which 2000 images are uniformly sampled from the feature space.
- 3.2. Uniformization of the dataset with t-SNE: The ImageNet-derived AlexNet features are selected for t-SNE, and grid-based sampling improves uniformity in the training data.
4. Multispectral conditional Generative Adversarial Networks (McGANs)
McGANs extends cGANs to four-channel RGB-NIR inputs and jointly predicts a cloud-free RGB image with a cloud mask. Its design uses multispectral conditioning, channel-wise L1 supervision, and an architecture detailed in the network specifications.
- McGANs extends cGANs to four-channel RGB-NIR inputs for cloud removal and outputs both a predicted RGB image and a grayscale cloud mask.The input and output channels are normalized to [−1, 1].
- The multispectral design is intended to merge visible-light color information with longer-wavelength information for cloud removal.The paper describes this extension as a way to combine RGB and longer-wavelength inputs.
- The McGANs architecture is specified through convolution, batch-normalization, activation, dropout, and filter-configuration components.The architecture is presented in the network diagram and tabulated specification.
- The generator is trained with a conditional GAN objective augmented by an L1 loss to encourage less-blurred predictions.The L1 term uses channel weights and compares the predicted result with the four-channel target.
5. Evaluation Results
Experiments evaluate McGANs on synthesized and real cloud-obscured satellite images, comparing multispectral inputs with NIR-only and RGB-only alternatives. The results support cloud removal and color prediction when NIR penetrates the cloud, while identifying failures for opaque clouds and cloud-like white objects.
- For synthesized clouds, Figure 7 compares obscured RGB inputs, NIR images, McGANs predictions, ground truth, and predicted cloud masks.The figure presents the input, output, reference, and mask images in columns.
- McGANs predicts more natural RGB results than NIR-only networks because NIR-only outputs differ in color from the ground truth.Real-cloud examples compare McGANs with NIR-GANs.
- McGANs does not remove clouds that are too thick for NIR penetration, leaving those regions unpredicted.The paper also notes erroneous cloud-mask recognition for overlapping white objects.
- Compared with RGB-only input, adding NIR produces more natural results for clouds penetrable only by NIR.The difference is smaller for thin clouds partly penetrated by visible light.
- McGANs removes clouds and predicts color properly when the cloud is thin enough to be penetrated by NIR.This is the paper’s reported overall evaluation conclusion.
6. Conclusion
The proposed method extends cGANs to multispectral inputs for thin-cloud removal, using synthesized training data and category-balanced sampling. Experiments support removal when clouds are penetrated in NIR images, while the authors identify realism, geographic diversity, and thicker-cloud handling as remaining challenges.
- The method extends cGANs to multispectral images for thin-cloud removal.
- Experimental results support removing visible-light clouds when they are penetrated in NIR images.
- Simulated clouds are used to construct paired cloud-obscured inputs and cloud-free ground truth for training.
- t-SNE sampling is introduced to reduce overfitting to categories in the biased training dataset.
- The authors note that Perlin-noise clouds differ from real clouds and that more varied images are needed for different areas.
- Future work targets FIR and SAR inputs and removal of all clouds in visible-light images.