PaperlayerView exact revision on arXiv

Source-linked AI summary

Missing Data Reconstruction in Remote Sensing image with a Unified Spatial-Temporal-Spectral Deep Convolutional Neural Network

Qiang Zhang, Qiangqiang Yuan, Chao Zeng, Xinghua Li, Yancong Wei

arXiv:1802.08369v1cs.CV

TL;DR

Remote sensing images often contain missing information from sensor malfunctions and atmospheric conditions, while many existing methods address only individual reconstruction tasks. The paper proposes STS-CNN, a unified deep CNN using spatial, temporal, and spectral auxiliary data, and reports effectiveness across three missing-information tasks in simulated and real-data experiments. Thick-cloud removal still produces some spectral distortion and blurring.

  • Problem

    Remote sensing images suffer missing information, and most existing reconstruction methods are independently designed for limited conditions or a single task.

  • Method

    STS-CNN uses a unified deep convolutional neural network with spatial, spectral, and temporal multi-source data to reconstruct missing information.

  • Results

    STS-CNN addresses Aqua MODIS band 6 dead lines, Landsat ETM+ SLC-off, and thick cloud removal, with simulated and real-data experiments reporting high effectiveness.

  • Takeaways & Limitations

    A single STS-CNN framework can be applied across the three listed remote sensing missing-information reconstruction tasks using multi-source supplementary data.

  • Takeaways & Limitations

    When temporal information is used for thick-cloud removal, the method produces some spectral distortion and blurring.

Abstract

from arXiv · show

Because of the internal malfunction of satellite sensors and poor atmospheric conditions such as thick cloud, the acquired remote sensing data often suffer from missing information, i.e., the data usability is greatly reduced. In this paper, a novel method of missing information reconstruction in remote sensing images is proposed. The unified spatial-temporal-spectral framework based on a deep convolutional neural network (STS-CNN) employs a unified deep convolutional neural network combined with spatial-temporal-spectral supplementary information. In addition, to address the fact that most methods can only deal with a single missing information reconstruction task, the proposed approach can solve three typical missing information reconstruction tasks: 1) dead lines in Aqua MODIS band 6; 2) the Landsat ETM+ Scan Line Corrector (SLC)-off problem; and 3) thick cloud removal. It should be noted that the proposed model can use multi-source data (spatial, spectral, and temporal) as the input of the unified framework. The results of both simulated and real-data experiments demonstrate that the proposed model exhibits high effectiveness in the three missing information reconstruction tasks listed above.

I. INTRODUCTION

Remote sensing imagery frequently contains missing information, while existing reconstruction methods are often limited to specific information sources or individual tasks. STS-CNN unifies spatial, temporal, and spectral auxiliary data in a deep CNN framework for three reconstruction problems.

  • Satellite sensor conditions and atmospheric effects produce missing information such as dead pixels and thick cloud cover in remote sensing images.
  • Existing methods use spatial, spectral, temporal, or combined information, but most operate independently and address only specific reconstruction tasks or conditions.
  • STS-CNN learns an end-to-end nonlinear mapping from missing data and auxiliary data to intact data using a deep convolutional neural network.
  • The unified framework combines spatial, spectral, and temporal supplementary information to improve recovery accuracy and consistency across different reconstruction tasks.
  • STS-CNN addresses dead lines in Aqua MODIS band 6, Landsat ETM+ SLC-off gaps, and thick cloud removal, with simulated and real experiments reporting stronger evaluation and visual results than mainstream methods.
  • Spatial-based methods include interpolation, exemplar-based, PDE-based, variational, and learning-based approaches, but large or complex texture areas limit their precision.

B. Spectral-Based Methods

Spectral and temporal auxiliary information can support missing-data reconstruction, but each approach has important scope constraints. Spectral methods cannot address thick cloud cover, while temporal methods face temporal differences and registration errors.

  • High spatial correlation among multispectral or hyperspectral bands enables spectral information to support reconstruction of missing data.
  • Spectral-based methods can recover missing spectral data accurately by exploiting correlations between different spectral bands.
  • Spectral methods cannot handle thick cloud cover because cloud causes the absence of all spectral bands to different degrees.
  • Temporal-based methods use observations of the same region at different times and can address situations including thick cloud and ETM+ SLC-off gaps.
  • Temporal differences and registration errors between multitemporal images negatively affect the precision of recovered regions.

D. Spatial-Temporal-Spectral-Based Methods

Prior reconstruction methods often target individual tasks or rely on linear models, while STS-CNN combines CNN-based nonlinear representation with joint spatial-temporal-spectral information for three task types.

  • Most reconstruction methods are developed independently for a single missing-information recovery task.
  • Linear models may struggle with highly nonlinear relationships among multi-source remote sensing images and complex degradation models.
  • STS-CNN combines a deep CNN with joint spatial-temporal-spectral information to reconstruct Aqua MODIS band 6 dead lines, Landsat SLC-off gaps, and thick cloud cover.
  • CNNs extract internal image features through locally connected feature maps, learned weights, biases, and nonlinear activation functions.
  • The CNN uses forward propagation, back-propagation, and gradient descent to update parameters and learn relationships between labeled and reconstructed data.

B. The Whole Framework Description

STS-CNN is designed as a unified framework for recovering different forms of missing information using an auxiliary spectral or temporal image alongside degraded spatial data.

  • STS-CNN is intended to recover dead pixels and remove thick cloud within one framework.
  • The model learns a nonlinear relationship between missing-region input 1y, auxiliary input 2y, and the original image x through a converged loss.

C. The Proposed STS-CNN Reconstruction Framework

The proposed framework fuses degraded spatial data with complementary spectral or temporal information before reconstruction. Its inputs vary by task while sharing a unified CNN framework.

  • STS-CNN accepts missing spatial data together with complementary spectral or temporal data as two network inputs.
  • For Aqua MODIS band 6 dead lines, the missing spectral image is paired with intact spectral data as auxiliary information.
  • For SLC-off gaps and thick-cloud removal, the degraded temporal image is paired with another temporal image as auxiliary information.
  • The two input streams are convolved separately and their 30-feature-map outputs are concatenated into a 3×3×60 representation.

2) Multi-Scale Convolutional Feature Extraction Unit

STS-CNN extracts contextual features at multiple spatial scales and enlarges receptive fields with dilated convolutions to support reconstruction across varied object sizes and regions.

  • The multi-scale unit applies simultaneous 3×3, 5×5, and 7×7 convolutions to extract multi-context features.
  • Multi-scale extraction addresses contextual variation caused by ground objects having different sizes across non-local regions.
  • Each multi-scale convolution produces 20 channels, which are concatenated into a single 60-channel feature map for later processing.
  • Dilated convolutions enlarge the receptive field while preserving the convolution kernel size, using different dilation factors across ranges.

4) Boosting of the Spatial-Temporal-Spectral Information

The model boosts spatial and spectral/temporal information through dedicated transfer paths and residual learning, using auxiliary data to reconstruct missing regions while preserving intact content.

  • 4) Boosting of the Spatial-Temporal-Spectral Information: A dedicated structure transfers spatial and spectral/temporal information through multiple network layers.The design is introduced to maintain and boost information transmission in the proposed method.
  • 4) Boosting of the Spatial-Temporal-Spectral Information: The residual image between the intact label and missing-data input is transferred before the loss function to preserve spatial information.This residual corresponds to the constructed missing regions because intact input and output regions are largely the same.
  • 4) Boosting of the Spatial-Temporal-Spectral Information: r_i = y_i^1 − x_i represents the difference between the missing-data image and the original undamaged image.The residual is concentrated mainly in missing regions, while values outside them are close to zero.
  • 4) Boosting of the Spatial-Temporal-Spectral Information: The model uses mean squared error as its loss for training image pairs with missing-data, intact, and spectral or temporal auxiliary inputs.The network parameters are optimized over a collection of N training image pairs.
  • 4) Boosting of the Spatial-Temporal-Spectral Information: A filled version of input 1, together with input 2 and a mask, is passed onward to enhance auxiliary spectral/temporal features and reduce spectral distortion.Multi-layer data transfer is used to strengthen spectral/temporal information in subsequent network layers.

5) Skip Connection

Skip connections address optimization difficulties in the deep CNN, while separate task-specific training data and settings support reconstruction across the proposed applications.

  • 5) Skip Connection: Skip connections pass previous-layer features to posterior layers, helping preserve image details and reduce vanishing gradients.The structure is used because increasing network depth can make training harder through gradient vanishing or exploding.
  • 5) Skip Connection: The same framework is trained independently with corresponding training data for each reconstruction task.Separate specific models are trained for the different missing-information reconstruction tasks.
  • 5) Skip Connection: For dead lines in Aqua MODIS band 6, Terra MODIS imagery provides similar training data, including 600 cropped 400×400×7 images divided into 40×40 patches.A separate Terra MODIS example is used to test simulated and real dead-line reconstruction.
  • 5) Skip Connection: ETM+ SLC-off and thick-cloud reconstruction use 16 temporal Landsat TM images from 2001.10.7 to 2002.5.4 arranged as temporal pairs.The images have size 1720×2040×6 and 30-m spatial resolution, with 100×100 training patches.
  • 5) Skip Connection: Training uses SGD with an initial learning rate of 0.01 for 100 epochs, reducing the learning rate by a factor of 0.1 every 20 epochs.The supplied passage also states that the network uses the Caffe framework.
  • 5) Skip Connection: The experiments compare task-specific methods using PSNR, SSIM, CC, mPSNR, mSSIM, and SAM evaluation indexes.The compared algorithms differ across dead-line, SLC-off, and thick-cloud reconstruction tasks.

B. Simulated Experiments

Simulated experiments evaluate STS-CNN on Terra MODIS dead lines, Landsat SLC-off, thick-cloud removal, and their combination, with reported visual and quantitative advantages over comparison methods.

  • B. Simulated Experiments: Terra MODIS band 6 dead-line recovery is evaluated with five methods, including STS-CNN, using PSNR, SSIM, and CC.The simulated Terra experiment serves as an approximate evaluation for Aqua MODIS because the two sensors have similar design patterns.
  • B. Simulated Experiments: Compared with LF, HMLLSF, and WCLF, STS-CNN is evaluated against stripe noise arising from complex relationships between MODIS bands.The comparison includes enlarged recovery regions and quantitative results in Table I.
  • B. Simulated Experiments: STS-CNN performs better than state-of-the-art methods for simulated Landsat ETM+ SLC-off recovery and reduces spectral distortion.The comparative methods produce discontinuous detail features, while STS-CNN achieves superior quantitative assessment results.
  • B. Simulated Experiments: With both SLC-off and thick cloud, STS-CNN performs better than comparison methods in spectral-distortion reduction and quantitative assessment.The experiment evaluates LLHM, NSPI, WLR, and STS-CNN using mSSIM, mPSNR, CC, and SAM.
  • B. Simulated Experiments: For simulated thick-cloud removal, STS-CNN performs well in reducing spectral distortion and achieves strong quantitative assessment results.The experiment compares STS-CNN with LLHM, mNSPI, WLR, STWR, and AWTC using Table III.

4) Simulated TM Data with Both Cloud and SLC-Off

The combined SLC-off and thick-cloud simulation tests reconstruction under large missing areas, where STS-CNN outperforms comparison methods in visual completeness and quantitative assessment.

  • 4) Simulated TM Data with Both Cloud and SLC-Off: The combined simulation evaluates reconstruction outputs from LLHM, NSPI, WLR, and STS-CNN for Landsat TM data with SLC-off and cloud cover.Quantitative evaluations use mSSIM, mPSNR, CC, and SAM.
  • 4) Simulated TM Data with Both Cloud and SLC-Off: LLHM and mNSPI cannot completely recover cloud-covered regions, while WLR produces texture discontinuity in large missing areas.These limitations are reported for remote sensing data containing both SLC-off and thick cloud.
  • 4) Simulated TM Data with Both Cloud and SLC-Off: STS-CNN performs better than the comparison methods in reducing spectral distortion and in the quantitative assessment.The reported comparison concerns the combined SLC-off and thick-cloud reconstruction task.
  • 4) Simulated TM Data with Both Cloud and SLC-Off: In real Terra MODIS band 6 recovery, all methods produce reasonable overall outcomes, but comparative methods retain stripe noise in enlarged regions.The real-data comparison includes LF, HMLLSF, WCLF, AWTC, and STS-CNN.

2) SLC-Off ETM+ Images

Real-data experiments show STS-CNN reconstructs Landsat ETM+ SLC-off gaps with less stripe noise and more continuous detail than comparison methods. In thick-cloud removal, it reduces spectral distortion and better restores texture details.

  • SLC-Off ETM+ Images: STS-CNN recovered Landsat ETM+ SLC-off gaps with the least stripe noise among the compared methods.Unlike LLHM, NSPI, and WLR, it did not require LPRM to complete residual gaps.
  • SLC-Off ETM+ Images: The proposed method preserved detail texture while producing more consistent and continuous reconstructions for dead pixels.
  • Cloud Removal of TM Images: In real TM cloud-removal experiments, competing methods showed spectral distortion within the largest cloud areas.
  • Cloud Removal of TM Images: STS-CNN reduced spectral distortion and reconstructed cloud-region texture better than WLR and STWR.Its spectral distortion was less than LLHM, mNSPI, and AWTC, while texture details were better than WLR and STWR.

1) Analysis of the Proposed Network Components

Component analyses evaluate multi-scale feature extraction, dilated convolution, and spatial-temporal/spectral information boosting, while registration-error tests assess robustness. The reported results support the network design and show STS-CNN degrades more slowly than comparison methods as misregistration increases.

  • Analysis of the Proposed Network Components: The ablation analysis compared network variants with and without multi-scale extraction, dilated convolution, and spatial-temporal/spectral information boosting.
  • Analysis of the Proposed Network Components: Dilated convolution improved reconstruction accuracy by 0.2/0.4 dB in the two simulated experiments.
  • Effects of Image Registration Errors: With increasing image registration errors, STS-CNN had the lowest degradation rate among the compared methods across the reported recovery measures.The evaluation used mSSIM, mPSNR, CC, and SAM with registration errors from 0–5 pixels.
  • Conclusions: The paper concludes that STS-CNN improves reconstruction accuracy and visual perception across simulated and real-data experiments.The framework addresses dead lines, ETM+ SLC-off gaps, and thick-cloud removal using spatial, spectral, and temporal information.
  • Conclusions: Thick-cloud removal can still produce spectral distortion and blurring, motivating future use of prior constraints to improve spectral fidelity and texture details.
Loading 1802.08369v1…