PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep learning based cloud detection for medium and high resolution remote sensing images of different sensors

Zhiwei Li, Huanfeng Shen, Qing Cheng, Yuhao Liu, Shucheng You, Zongyi He

arXiv:1810.05801v3cs.CV

TL;DR

Cloud detection is necessary for precise optical satellite-image applications, but existing methods are often sensor-specific and can face calibration and bright-surface challenges. The paper introduces MSCFF, a deep-learning method that fuses multi-scale features for cloud and cloud-shadow detection across multiple sensors. The reported results show better discrimination of clouds from bright land surfaces and more refined masks than compared methods.

  • Problem

    Existing cloud-detection methods are often designed for specific satellite imagery, while radiometric calibration is not always accurate, particularly for sensors lacking onboard calibration.

  • Method

    MSCFF uses a deep-learning multi-scale convolutional feature-fusion architecture to extract clouds and cloud shadows from multiple types of satellite imagery.

  • Results

    MSCFF performs better than compared methods at distinguishing clouds from bright land surfaces and generating more refined cloud and cloud-shadow masks.

  • Takeaways & Limitations

    MSCFF shows promise for practical cloud detection across multiple types of medium- and high-resolution remote-sensing images.

  • Takeaways & Limitations

    The approach must address imagery for which radiometric calibration is not accurately implemented, especially from sensors without onboard calibration capacity.

Abstract

from arXiv · show

Cloud detection is an important preprocessing step for the precise application of optical satellite imagery. In this paper, we propose a deep learning based cloud detection method named multi-scale convolutional feature fusion (MSCFF) for remote sensing images of different sensors. In the network architecture of MSCFF, the symmetric encoder-decoder module, which provides both local and global context by densifying feature maps with trainable convolutional filter banks, is utilized to extract multi-scale and high-level spatial features. The feature maps of multiple scales are then up-sampled and concatenated, and a novel multi-scale feature fusion module is designed to fuse the features of different scales for the output. The two output feature maps of the network are cloud and cloud shadow maps, which are in turn fed to binary classifiers outside the model to obtain the final cloud and cloud shadow mask. The MSCFF method was validated on hundreds of globally distributed optical satellite images, with spatial resolutions ranging from 0.5 to 50 m, including Landsat-5/7/8, Gaofen-1/2/4, Sentinel-2, Ziyuan-3, CBERS-04, Huanjing-1, and collected high-resolution images exported from Google Earth. The experimental results show that MSCFF achieves a higher accuracy than the traditional rule-based cloud detection methods and the state-of-the-art deep learning models, especially in bright surface covered areas. The effectiveness of MSCFF means that it has great promise for the practical application of cloud detection for multiple types of medium and high-resolution remote sensing images. Our established global high-resolution cloud detection validation dataset has been made available online.

1. Introduction

Cloud detection is essential for optical satellite-image applications, yet existing methods are often sensor-specific and can struggle with bright surfaces, thin clouds, and limited calibration. The paper proposes MSCFF as a more general deep-learning approach for multi-source imagery and reports improved cloud and cloud-shadow extraction.

  • Clouds and cloud shadows reduce the availability and precise application of optical satellite imagery.
  • Most rule-based cloud detectors are designed for particular satellite imagery, making separate methods inefficient as satellite sources increase.Fmask is a prominent Landsat method based on cloud-physical-property thresholds.
  • Traditional rule-based methods can mistake bright non-cloud objects for clouds and omit thin clouds, especially with limited spectral information.
  • Previous deep-learning cloud detectors often achieve high accuracy but are mainly validated in local regions and on specific imagery types.
  • MSCFF uses deep learning to extract clouds and cloud shadows from multiple satellite-image types collected globally across varied land-cover conditions.
  • MSCFF performs better than compared methods at distinguishing clouds from bright land surfaces and generating more refined cloud and cloud-shadow masks.

2. Methodology

MSCFF is a two-stage, fully convolutional cloud-detection pipeline that trains on labeled image patches and predicts separate cloud and cloud-shadow maps. Its encoder-decoder extracts multi-scale features, while feature fusion combines them for pixel-wise outputs and external binarization.

  • MSCFF comprises model-training and model-testing stages for processing multiple types of satellite imagery.Training uses images with corresponding cloud and cloud-shadow labels; testing predicts cloud and cloud-shadow maps for image patches.
  • Training minimizes a loss computed from model predictions and binary labels, with image patches clipped to 256 × 256 pixels.
  • During testing, separate cloud and cloud-shadow feature maps are binarized externally and merged into one cloud and cloud-shadow mask.
  • The encoder-decoder extracts multi-scale, high-level spatial features, using residual units and dilated convolutions to improve trainability and effectiveness.
  • The multi-scale fusion module uses convolutional features from different scales to improve cloud-detection accuracy.
  • The network combines a symmetric encoder-decoder module with a multi-scale feature fusion module.The modules contain convolutional, pooling, and deconvolutional layers.

A. The basic convolutional layer

MSCFF uses convolutional layers to extract feature maps and produces cloud and cloud-shadow outputs through a fully convolutional architecture. Its CBRR blocks combine convolutions, batch normalization, ReLU activation, and residual units.

  • A. The basic convolutional layer: A convolutional layer transforms m × n × c input feature maps into m′ × n′ × k output maps using k filters.Each filter has size r × r × c, and the output is computed as Y_i = W_i ∗ X + b_i.
  • A. The basic convolutional layer: Batch normalization and element-wise ReLU activation are applied to convolutional outputs except those of the last layer.These operations follow the convolutional feature extraction stage in MSCFF.
  • A. The basic convolutional layer: MSCFF uses CBRR blocks containing convolutional layers, batch normalizations, ReLU activations, and a residual unit.The convolutional layers use 3 × 3 filters with zero padding to preserve feature-map height and width.
  • A. The basic convolutional layer: The fully convolutional MSCFF architecture uses a symmetric encoder-decoder to extract multi-scale features, then concatenates and fuses six up-sampled scales.The final convolutional layer outputs cloud and cloud-shadow maps for external pixel-wise binary classification.

B. The dilated convolutional layer

MSCFF introduces dilated convolutions to expand receptive fields and capture broader context without changing feature-map size. This design addresses the computational burden of increasing network depth or filter size.

  • B. The dilated convolutional layer: Increasing network depth or filter size enlarges receptive fields but substantially increases computational burden and training time.This motivates the use of dilated convolutional layers in MSCFF.
  • B. The dilated convolutional layer: Dilated convolutions aggregate multi-scale contextual information while preserving resolution and can expand receptive fields exponentially with depth.Prior work is cited as reporting better classification performance than non-dilated counterparts without increasing model depth or complexity.
  • B. The dilated convolutional layer: MSCFF uses dilation factors 2, 4, 4, and 2 in the last two encoder and first two corresponding decoder CBRR blocks.The convolutions use 3 × 3 kernels with the stated dilation factors.
  • B. The dilated convolutional layer: Dilated convolutions let MSCFF extract features from a larger field without changing feature-map size, preserving spatial information for feature fusion.The preserved spatial information is used in subsequent multi-scale fusion operations.

C. The pooling layer

MSCFF combines pooling and dilated convolution in the encoder to obtain strong multi-scale features while retaining detail for the decoder. Pooling reduces spatial dimensions through maximum selection.

  • C. The pooling layer: Pooling extracts maximum or average values from local feature-map regions, reducing spatial dimensions without reducing depth.MSCFF uses maximum pooling for encoder down-sampling.
  • C. The pooling layer: MSCFF uses three 2 × 2 pooling layers with a two-pixel stride in the encoder.These layers down-sample feature maps by the maximum operator.
  • C. The pooling layer: Pooling is applied after the first three CBRR blocks, while dilated convolutions replace basic convolutions in the encoder’s last two CBRR blocks.This arrangement combines down-sampling with dilated processing at later encoder stages.
  • C. The pooling layer: The combined pooling and dilated-convolution design provides multi-scale features and preserves detail for the decoder.The paper connects this combination with achieving better subsequent processing.

D. The deconvolutional layer

MSCFF decodes encoder features by up-sampling and reconstructing them, then fuses multiple scales to produce cloud and cloud-shadow maps. Training uses mean-squared error, and thresholding converts outputs into masks.

  • D. The deconvolutional layer: Deconvolutional layers up-sample and reconstruct feature maps, recovering the initial encoder feature-map shape.Deconvolution serves as the decoding layer of the convolutional encoder.
  • D. The deconvolutional layer: The symmetric encoder-decoder connects corresponding encoding and decoding layers with skip connections and element-wise summation.Skip connections retain spatial details and support training of very deep networks.
  • D. The deconvolutional layer: Decoder feature maps from six scales are up-sampled to the input size, concatenated, and fused by the final convolutional layer.Feature maps closer to the output provide higher-level semantic features, while earlier maps retain more spatial detail.
  • D. The deconvolutional layer: MSCFF outputs cloud and cloud-shadow maps with values intended to range from 0 to 1, followed by external binary classification.Mean-squared error is used to focus training on generating these maps for later binarization.
  • D. The deconvolutional layer: During testing, overlapping patch predictions use maximal values, and a threshold converts the maps into cloud and cloud-shadow masks.The default threshold is 0.5, after which the masks are merged with cloud given higher priority.
  • D. The deconvolutional layer: Both multi-scale feature fusion and residual units improve cloud-detection accuracy, while increasing filter counts yields only slight gains but significantly higher resource use.The default filter setting is reported to balance accuracy and computational efficiency.

3. Experimental data

The experiments use four diverse cloud and cloud-shadow datasets spanning multiple sensors, spatial resolutions, land-cover types, and global regions. A newly created high-resolution dataset addresses the lack of suitable public data for deep-model training and testing.

  • Data preparation: Some L7_Irish and L8_Biome images were removed after visual inspection because their cloud or cloud-shadow labels contained intolerable errors.For HRC_WHU, only clouds were labeled because high-resolution cloud shadows were rare and difficult to select accurately.
  • Datasets: Four datasets cover Landsat-7, Landsat-8, Gaofen-1, and high-resolution imagery with spatial resolutions from 0.5 to 30 m.The datasets include 166 L7_Irish scenes, 92 L8_Biome scenes, 108 GF1_WHU scenes, and 150 HRC_WHU images.
  • HRC_WHU: HRC_WHU was created because suitable public high-resolution cloud detection datasets were unavailable for deep-model training and testing.It contains 150 RGB images at 0.5–15 m resolution, with expert-digitized reference masks.
  • HRC_WHU: The HRC_WHU high-resolution validation dataset was made available online for cloud detection research and benchmarking.The dataset was established in this study and released through the Wuhan University MSCFF website.
  • Datasets: The datasets represent globally distributed imagery collected across different land-cover types and cloud conditions.L8_Biome scenes are clustered into eight biomes, while GF1_WHU scenes cover global land-cover types with varying cloud conditions.
  • Data preparation: Scenes were split into 80% training and 20% testing within each dataset, with training and testing imagery kept completely independent.Training samples were clipped from training scenes, whereas test scenes were processed patch by patch.

4. Experimental results

MSCFF was evaluated against rule-based and deep-learning cloud detection methods on multiple sensor datasets and land-cover types. It generally achieved the highest reported accuracies, with particular advantages on bright surfaces and high-resolution imagery, while retaining a smaller model size than the deep baselines.

  • Evaluation setup: MSCFF was compared with Fmask, MFC, PRS, DeepLab, and DCN using dataset-specific models because sensors have different spectral bands and spatial resolutions.DeepLab and DCN were trained from scratch with the same training data used for MSCFF.
  • Landsat results: 94.45% and 94.96% overall accuracy were achieved by MSCFF on tested Landsat-7 and Landsat-8 scenes, versus 91.71% and 89.59% for Fmask.MSCFF also achieved the highest mIoU and F-score values among compared methods for both Landsat datasets.
  • High-resolution results: 0.903 mIoU was achieved by MSCFF on high-resolution images, compared with 0.856 for DeepLab and 0.877 for DCN.The authors report that MSCFF was more effective for cloud detection in high-resolution images.
  • Land-cover analysis: MSCFF improved snow/ice cloud-detection accuracy from 87.05% to 90.20% on Landsat and from 65.39% to 92.17% on Gaofen-1 relative to the cited rule-based methods.It achieved the highest accuracies over most tested land-cover types.
  • Comparative performance: MSCFF produced fewer mask errors and the highest accuracies among DeepLab, DCN, and MSCFF, especially when bright surfaces resembled clouds.The deep models were more effective than traditional methods on bright surfaces; MSCFF was strongest among the deep models.
  • Efficiency: MSCFF processed a whole Landsat-8 image in less than a minute in GPU mode and used significantly fewer parameters than DeepLab and DCN.Its time efficiency was slightly lower than DeepLab, but the authors report higher accuracy and a much smaller model size.

5. Application extension and limitations

The pre-trained MSCFF model extends across multiple satellite image types and resolutions, but its performance is bounded by input characteristics, receptive-field limits, and insufficient high-resolution cloud-shadow training data.

  • 5.1. Application extension: MSCFF is less sensitive than traditional rule-based methods to image light-condition changes, while contrast- and color-balanced inputs can improve mask accuracy.For low-contrast imagery, the implementation applies linear transformation before inference; image stretching may also be considered after normalization.
  • 5.1. Application extension: The pre-trained MSCFF model generates visually satisfactory cloud masks across Gaofen-2, Ziyuan-3, Gaofen-1, Sentinel-2A, Landsat-5, and Gaofen-4 imagery spanning 4–50 m resolutions.The model was trained on globally distributed Gaofen-1 WFV data and applied to other satellite images.
  • 5.1. Application extension: A single method can handle many satellite image types with different spatial resolutions, although transfer works best for imagery with similar spectral settings.Model fine-tuning with a small learning rate and limited labels can improve performance on new image types.
  • 5.2. Limitations: MSCFF may fail under complex surface conditions because scale relationships between clouds and shadows vary across spatial resolutions.Expanding the receptive field through larger patches or deeper networks may help, but requires device support and increases network complexity.
  • 5.2. Limitations: Limited receptive fields and patch sizes can cause MSCFF to misclassify large bright objects such as snow when thermal infrared input is unavailable.The issue is rare but may occur in high-altitude snow- or ice-covered imagery, where bright-surface cores resemble clouds.
  • 5.2. Limitations: Cloud-shadow detection is unreliable when the model trained on medium-resolution imagery is applied to high-resolution images, where shadows can be faint or resemble building shadows.More high-resolution training samples with sufficient cloud-shadow labels are needed; increasing receptive field and input patch size also raises computational complexity.

6. Conclusions

MSCFF combines multi-scale convolutional features to detect clouds and cloud shadows across medium- and high-resolution imagery from different sensors. It outperformed compared methods across datasets, particularly over bright surfaces, while cloud-shadow detection and large-area shadow delineation remain limitations.

  • 6. Conclusions: MSCFF targets cloud and cloud-shadow detection in medium- and high-resolution satellite images from different sensors.The method combines multi-scale convolutional features within an encoder-decoder architecture.
  • 6. Conclusions: Feature fusion and residual network units both improve model performance.The conclusion attributes performance gains to these two architectural components.
  • 6. Conclusions: MSCFF achieves higher accuracy than compared methods across different datasets, especially in areas covered by bright surfaces.The reported comparison includes rule-based and other detection methods in the broader evaluation.
  • 6. Conclusions: MSCFF was applied to extract clouds and cloud shadows from more than 10 types of satellite imagery, supporting practical use across multiple imagery types.The conclusion describes this cross-sensor application as promising for practical cloud-detection use.
  • 6. Conclusions: Cloud-shadow detection remains difficult because shadows can resemble mountain shadows, building shadows, and water bodies, especially with limited spectral information.The authors identify cloud shadow as more difficult than cloud detection and plan to improve its accuracy.
  • 6. Conclusions: Limited input patches and CNN receptive fields may cause large-area cloud shadows to be identified fragmentarily.The authors suggest combining CNNs with object-based image analysis as a possible improvement.
  • 6. Conclusions: The authors plan better training strategies for classes with insufficient samples and intend to generalize MSCFF to more imagery types using a single model.Cloud shadow is cited as an example of a class with insufficient training samples.
Loading 1810.05801v3…