PaperlayerView exact revision on arXiv

Source-linked AI summary

Cloud Removal for Remote Sensing Imagery via Spatial Attention Generative Adversarial Network

Heng Pan

arXiv:2009.13015v2eess.IVcs.CV

TL;DR

Clouds reduce the availability of optical satellite imagery, and cloud removal is a necessary preprocessing step. SpA GAN applies local-to-global spatial attention to focus on cloud areas. It achieves the best reported PSNR and SSIM among the compared models on both RICE and RICE2.

  • Problem

    Clouds reduce optical satellite-image availability, making removal necessary before image analysis, while neural-network use for remote-sensing cloud removal remains relatively limited.

    Source: §I · p. 1, CONCLUSION · p. 6

  • Method

    SpA GAN uses a generative adversarial network with local-to-global spatial attention to identify cloud areas and enhance their information recovery.

    Source: §I · p. 2

  • Results

    30.232 dB PSNR and 0.954 SSIM; SpA GAN achieved the best PSNR and SSIM among the compared models on RICE, and also led on RICE2.

    Source: D. Results and analysis · pp. 5–6

  • Takeaways & Limitations

    The reported comparisons indicate that spatial attention helps observe cloud areas and improve cloud-removal performance; generated images retain ground-object geometry and spatial information visually.

    Source: D. Results and analysis · pp. 5–6

Abstract

from arXiv · show

Optical remote sensing imagery has been widely used in many fields due to its high resolution and stable geometric properties. However, remote sensing imagery is inevitably affected by climate, especially clouds. Removing the cloud in the high-resolution remote sensing satellite image is an indispensable pre-processing step before analyzing it. For the sake of large-scale training data, neural networks have been successful in many image processing tasks, but the use of neural networks to remove cloud in remote sensing imagery is still relatively small. We adopt generative adversarial network to solve this task and introduce the spatial attention mechanism into the remote sensing imagery cloud removal task, proposes a model named spatial attention generative adversarial network (SpA GAN), which imitates the human visual mechanism, and recognizes and focuses the cloud area with local-to-global spatial attention, thereby enhancing the information recovery of these areas and generating cloudless images with better quality...

I. INTRODUCTION

Clouds impair optical remote-sensing imagery, making cloud removal a necessary preprocessing step. SpA GAN applies local-to-global spatial attention to cloud removal and is reported to outperform existing models on RICE by PSNR and SSIM.

I. INTRODUCTION

  • Clouds reduce the availability of optical satellite imagery, so cloud removal is an indispensable preprocessing step before image analysis.The imagery is used in areas including national defense security, environmental science, and weather monitoring.Source: §I · p. 1
  • Thin clouds retain some ground-feature information and can be addressed from one image, whereas thick-cloud information is lost and often requires multitemporal data.The paper focuses on removing thin clouds from a single optical remote-sensing image; cloud shadows often appear with thick clouds.Source: §I · p. 1
  • Cloud removal can also be framed as image denoising, with clouds treated as noise relative to surface objects.The introduction notes that methods for removing fog, raindrops, watermarks, and shadows offer relevant design references.Source: §I · p. 1
  • SpA GAN combines a generative adversarial network with spatial attention [2] [3] to focus on cloud areas and improve their information recovery.The paper describes this local-to-global attention as imitating human visual attention and generating better-quality cloudless images.Source: §I · p. 2
  • On the open-source RICE dataset, SpA GAN achieved the best PSNR and SSIM among the compared cloud-removal models.The paper reports these comparisons as evidence that spatial attention improves cloud-removal image quality.Source: §I · p. 2

A. Cloud removal

The section reviews traditional and learned cloud-removal approaches, introduces GANs, and describes adapting a GAN to cloud removal with spatial attention.

A. Cloud removal

  • Traditional cloud-removal methods use pixel correction or image priors, while their designed models can have limited performance and HOT can overcorrect or distort color.HOT uses correlation between blue and red bands; DCP infers a cloud model from dark-channel statistics.Source: A. Cloud removal · p. 2
  • Learned cloud-removal approaches include McGANs trained with synthetic clouds and near-infrared bands, cycle-GAN-based Cloud GAN, and methods fusing SAR imagery.Cloud GAN maps cloudy images to cloudless images and back toward the original input; SAR imagery is described as unaffected by climatic conditions.Source: A. Cloud removal · p. 2

B. Generative adversarial networks

  • A GAN pairs a generator that learns the target-sample distribution with a discriminator that judges whether samples come from real data.The two networks train in competition: the generator seeks outputs the discriminator cannot distinguish from training samples, while the discriminator distinguishes real from generated data.Source: B. Generative adversarial networks · p. 2, §III · p. 2

III. METHODOLOGY

  • For cloud removal, the paper modifies the primal GAN and proposes a spatial-attention GAN.The stated minimax setup uses real samples x, random noise z, discriminator D, and generated output G(z).Source: §III · p. 2

A. Generator

SPANet generates the cloud-removed image through feature extraction, four spatial attentive blocks, and reconstruction, with attention maps guiding cloud removal.

A. Generator

  • SPANet processes input features through three standard residual blocks, four spatial attentive blocks, two residual blocks, and a final convolution to generate the cloud-removed image.The figure caption says the four attentive blocks identify clouds progressively in four stages and the final residual blocks reconstruct a clean background.Source: A. Generator · p. 2, Figure 3 · p. 3
  • Spatial attention maps assign each pixel a continuous attention value, indicating the spatial distribution of clouds for subsequent removal steps.The SAM uses two rounds of four-direction recurrent processing to aggregate local and then non-local context into a global perceptual feature map.Source: A. Generator · p. 3
  • Each spatial attentive residual block removes clouds through negative residuals guided by the attention map.The attentive block contains three SARBs and one SAM connected in parallel.Source: A. Generator · pp. 2–3, Figure 3 · p. 3

B. Discriminator

SpA GAN uses a convolutional discriminator to classify images as real or generator-produced, while its training objective combines conditional-GAN, reconstruction, and attention losses.

B. Discriminator

C. Loss

  • The total SpA GAN objective combines conditional-GAN, pixelwise L1 reconstruction, and attention-loss terms.The L1 term measures reconstructed-pixel accuracy; the attention term uses the spatial-attention map and a binary cloud-area image.Source: C. Loss · p. 3
  • The L1 term measures pixelwise accuracy by comparing the generator's predicted result with the ground truth.Each channel's loss weight is set to 1.Source: C. Loss · p. 3
  • The attention loss uses the module's attention map and a binary cloud-area image derived from the cloudy and cloudless images.The attention map is generated by the spatial attentive module.Source: C. Loss · p. 3

D. Discuss

SpA GAN uses spatial attention in its generative network to accumulate global image information and guide cloud removal and model training.

D. Discuss

  • The attention map accumulates global image information during the generative process.Each location learns from its four-direction connected pixels, with information spreading through the image.Source: D. Discuss · p. 4
  • Spatial information propagates through four-direction connected pixels until locations learn global image information.Source: D. Discuss · p. 4
  • The attention map guides both the spatial-attention residual blocks during cloud removal and the model's training through the loss function.Source: D. Discuss · p. 4

IV. EXPERIMENTS

The RICE dataset provides paired cloudy and cloudless remote-sensing images in two subsets, with cloud masks additionally supplied for RICE2.

A. Datasets

  • RICE contains RICE1 and RICE2, providing paired cloudy and cloudless samples, with cloud masks in RICE2.Fig. 5 shows cloudy samples in the first row and cloudless samples in the second; the first two columns are RICE1 and the last two are RICE2.Source: A. Datasets · p. 4, Figure 5 · p. 4
  • RICE1 has 500 paired 512×512 cloudy and cloudless samples collected from Google Earth.The cloudy/cloudless pairs are obtained by setting whether the cloud layer is displayed.Source: A. Datasets · p. 4
  • RICE2 contains 736 groups of 512×512 images, each with a cloudy image, a cloudless reference, and a cloud mask.The reference image is manually selected from the same location, with its acquisition less than 15 days from the cloudy image.Source: A. Datasets · p. 4

B. Evaluation metric

The paper evaluates generated cloudless-image quality and cloud-removal ability using PSNR and SSIM.

B. Evaluation metric

  • PSNR and SSIM are used to measure generated cloudless-image quality and the model's cloud-removal ability.Source: B. Evaluation metric · p. 4

a) PSNR: PSNR is the most widely and most common used objective measurement for evaluating image quality, whose calculation formula is:

PSNR evaluates image quality using the mean squared error between an image and its reference. The section gives the MSE calculation and describes how to interpret PSNR values.

a) PSNR

b) SSIM: SSIM is an evaluation metirc that measures the similarity of two images through three aspects: brightness, contrast, and structure, whose formula are (8), (9), (10) respectively:

SSIM combines luminance, contrast, and structure comparisons to measure image similarity. Its value ranges from 0 to 1, with 1 indicating identical images.

b) SSIM

C. Settings

The experiments use separate training and testing splits for RICE1 and RICE2, with fixed training settings. SpA GAN is compared with conditional GAN and cycle GAN.

C. Settings

  • RICE1 uses 400 training images and 100 testing images, while RICE2 uses 588 training images and 148 testing images.Source: C. Settings · p. 4
  • Training uses a learning rate of 0.0004, minibatch size 1, and 200 epochs.Source: C. Settings · p. 4
  • SpA GAN is compared with conditional GAN and cycle GAN for cloud removal.Source: C. Settings · p. 4

D. Results and analysis

On RICE1 and RICE2, SpA GAN achieved the best reported PSNR and SSIM among the three compared models. The visual comparisons report stronger detail and consistency for SpA GAN on thick-cloud RICE2 images, while RICE1 outputs retain ground-object structure in a mostly thin-cloud setting.

D. Results and analysis

  • On mostly thin-cloud RICE1 images, every generated result retained the ground objects' geometric structure and spatial information.The section notes that ground-object information is not completely lost in these thin-cloud examples.Source: D. Results and analysis · p. 5, Figure 6 · p. 5
  • 30.232 dB PSNR and 0.954 SSIM were SpA GAN's best results on RICE1, significantly outperforming conditional GAN and cycle GAN.The quantitative comparison is reported in Table I.Source: D. Results and analysis · p. 5, Table I · p. 5
  • On thick-cloud RICE2 images, SpA GAN retained more detail and consistency and was visually closest to ground truth, unlike the fuzzy or spatially disrupted alternatives.Cycle GAN removed white cloud areas but disrupted spatial continuity; conditional GAN preserved continuity but produced fuzzy areas with poorly restored detail.Source: D. Results and analysis · p. 5, Figure 8 · p. 6
  • SpA GAN's attention heatmap assigns more attention to redder areas and less to bluer areas, illustrating the cloud-area attention highlighted in the results and conclusion.The conclusion reports that SpA GAN had the best PSNR and SSIM on the public RICE dataset compared with conditional GAN and cycle GAN.Source: Figure 7 · p. 5, D. Results and analysis · p. 6, CONCLUSION · p. 6
  • 28.368 dB PSNR and 0.906 SSIM were best on RICE2; PSNR rose 2.982 dB over conditional GAN and 4.458 dB over cycle GAN.The section also reports SSIM increases of 0.095 over conditional GAN and 0.113 over another model, but the latter baseline is not specified in the supplied passage.Source: D. Results and analysis · p. 6, Table II · p. 6
Loading 2009.13015v2…