PaperlayerView exact revision on arXiv

Source-linked AI summary

Missing Data Reconstruction in Remote Sensing image with a Unified Spatial-Temporal-Spectral Deep Convolutional Neural Network

Qiang Zhang, Qiangqiang Yuan, Chao Zeng, Xinghua Li, Yancong Wei

arXiv:1802.08369v1cs.CV

TL;DR

Remote sensing images often contain missing information from sensor malfunctions and atmospheric conditions, while many existing methods address only individual reconstruction tasks. The paper proposes STS-CNN, a unified deep CNN using spatial, temporal, and spectral auxiliary data, and reports effectiveness across three missing-information tasks in simulated and real-data experiments. Thick-cloud removal still produces some spectral distortion and blurring.

  • Problem

    Remote sensing images suffer missing information, and most existing reconstruction methods are independently designed for limited conditions or a single task.

    Source: §I · p. 1

  • Method

    STS-CNN uses a unified deep convolutional neural network with spatial, spectral, and temporal multi-source data to reconstruct missing information.

    Source: §I · pp. 1–2

  • Results

    STS-CNN addresses Aqua MODIS band 6 dead lines, Landsat ETM+ SLC-off, and thick cloud removal, with simulated and real-data experiments reporting high effectiveness.

    Source: §I · p. 2, §V · p. 13

  • Takeaways & Limitations

    A single STS-CNN framework can be applied across the three listed remote sensing missing-information reconstruction tasks using multi-source supplementary data.

    Source: §I · p. 1, §V · p. 13

    When temporal information is used for thick-cloud removal, the method produces some spectral distortion and blurring.

    Source: §V · p. 13

Abstract

from arXiv · show

Because of the internal malfunction of satellite sensors and poor atmospheric conditions such as thick cloud, the acquired remote sensing data often suffer from missing information, i.e., the data usability is greatly reduced. In this paper, a novel method of missing information reconstruction in remote sensing images is proposed. The unified spatial-temporal-spectral framework based on a deep convolutional neural network (STS-CNN) employs a unified deep convolutional neural network combined with spatial-temporal-spectral supplementary information. In addition, to address the fact that most methods can only deal with a single missing information reconstruction task, the proposed approach can solve three typical missing information reconstruction tasks: 1) dead lines in Aqua MODIS band 6; 2) the Landsat ETM+ Scan Line Corrector (SLC)-off problem; and 3) thick cloud removal. It should be noted that the proposed model can use multi-source data (spatial, spectral, and temporal) as the input of the unified framework. The results of both simulated and real-data experiments demonstrate that the proposed model exhibits high effectiveness in the three missing information reconstruction tasks listed above.

I. INTRODUCTION

Remote sensing imagery frequently contains missing information, while existing reconstruction methods are often limited to specific information sources or individual tasks. STS-CNN unifies spatial, temporal, and spectral auxiliary data in a deep CNN framework for three reconstruction problems.

  • Satellite sensor conditions and atmospheric effects produce missing information such as dead pixels and thick cloud cover in remote sensing images.Source: §I · p. 1
  • Existing methods use spatial, spectral, temporal, or combined information, but most operate independently and address only specific reconstruction tasks or conditions.Source: §I · p. 1
  • STS-CNN learns an end-to-end nonlinear mapping from missing data and auxiliary data to intact data using a deep convolutional neural network.Source: §I · pp. 1–2
  • The unified framework combines spatial, spectral, and temporal supplementary information to improve recovery accuracy and consistency across different reconstruction tasks.Source: §I · p. 2
  • STS-CNN addresses dead lines in Aqua MODIS band 6, Landsat ETM+ SLC-off gaps, and thick cloud removal, with simulated and real experiments reporting stronger evaluation and visual results than mainstream methods.Source: §I · pp. 1–2
  • Spatial-based methods include interpolation, exemplar-based, PDE-based, variational, and learning-based approaches, but large or complex texture areas limit their precision.Source: A. Spatial-Based Methods · p. 2

B. Spectral-Based Methods

Spectral and temporal auxiliary information can support missing-data reconstruction, but each approach has important scope constraints. Spectral methods cannot address thick cloud cover, while temporal methods face temporal differences and registration errors.

  • High spatial correlation among multispectral or hyperspectral bands enables spectral information to support reconstruction of missing data.Source: B. Spectral-Based Methods · p. 2
  • Spectral-based methods can recover missing spectral data accurately by exploiting correlations between different spectral bands.Source: B. Spectral-Based Methods · p. 2
  • Spectral methods cannot handle thick cloud cover because cloud causes the absence of all spectral bands to different degrees.Source: B. Spectral-Based Methods · p. 2
  • Temporal-based methods use observations of the same region at different times and can address situations including thick cloud and ETM+ SLC-off gaps.Source: C. Temporal-Based Methods · pp. 2–3
  • Temporal differences and registration errors between multitemporal images negatively affect the precision of recovered regions.Source: C. Temporal-Based Methods · p. 3

D. Spatial-Temporal-Spectral-Based Methods

Prior reconstruction methods often target individual tasks or rely on linear models, while STS-CNN combines CNN-based nonlinear representation with joint spatial-temporal-spectral information for three task types.

B. The Whole Framework Description

STS-CNN is designed as a unified framework for recovering different forms of missing information using an auxiliary spectral or temporal image alongside degraded spatial data.

C. The Proposed STS-CNN Reconstruction Framework

The proposed framework fuses degraded spatial data with complementary spectral or temporal information before reconstruction. Its inputs vary by task while sharing a unified CNN framework.

2) Multi-Scale Convolutional Feature Extraction Unit

STS-CNN extracts contextual features at multiple spatial scales and enlarges receptive fields with dilated convolutions to support reconstruction across varied object sizes and regions.

4) Boosting of the Spatial-Temporal-Spectral Information

The model boosts spatial and spectral/temporal information through dedicated transfer paths and residual learning, using auxiliary data to reconstruct missing regions while preserving intact content.

4) Boosting of the Spatial-Temporal-Spectral Information

  • A dedicated structure transfers spatial and spectral/temporal information through multiple network layers.The design is introduced to maintain and boost information transmission in the proposed method.Source: 4) Boosting of the… · p. 6
  • The residual image between the intact label and missing-data input is transferred before the loss function to preserve spatial information.This residual corresponds to the constructed missing regions because intact input and output regions are largely the same.Source: 4) Boosting of the… · p. 6
  • r_i = y_i^1 − x_i represents the difference between the missing-data image and the original undamaged image.The residual is concentrated mainly in missing regions, while values outside them are close to zero.Source: 4) Boosting of the… · p. 6
  • The model uses mean squared error as its loss for training image pairs with missing-data, intact, and spectral or temporal auxiliary inputs.The network parameters are optimized over a collection of N training image pairs.Source: 4) Boosting of the… · p. 6
  • A filled version of input 1, together with input 2 and a mask, is passed onward to enhance auxiliary spectral/temporal features and reduce spectral distortion.Multi-layer data transfer is used to strengthen spectral/temporal information in subsequent network layers.Source: 4) Boosting of the… · p. 6

5) Skip Connection

Skip connections address optimization difficulties in the deep CNN, while separate task-specific training data and settings support reconstruction across the proposed applications.

5) Skip Connection

  • Skip connections pass previous-layer features to posterior layers, helping preserve image details and reduce vanishing gradients.The structure is used because increasing network depth can make training harder through gradient vanishing or exploding.Source: 5) Skip Connection · p. 6, Figure 9 · p. 6
  • The same framework is trained independently with corresponding training data for each reconstruction task.Separate specific models are trained for the different missing-information reconstruction tasks.Source: A. Experimental Settings · p. 6
  • For dead lines in Aqua MODIS band 6, Terra MODIS imagery provides similar training data, including 600 cropped 400×400×7 images divided into 40×40 patches.A separate Terra MODIS example is used to test simulated and real dead-line reconstruction.Source: A. Experimental Settings · p. 6
  • ETM+ SLC-off and thick-cloud reconstruction use 16 temporal Landsat TM images from 2001.10.7 to 2002.5.4 arranged as temporal pairs.The images have size 1720×2040×6 and 30-m spatial resolution, with 100×100 training patches.Source: A. Experimental Settings · p. 7
  • Training uses SGD with an initial learning rate of 0.01 for 100 epochs, reducing the learning rate by a factor of 0.1 every 20 epochs.The supplied passage also states that the network uses the Caffe framework.Source: A. Experimental Settings · p. 7
  • The experiments compare task-specific methods using PSNR, SSIM, CC, mPSNR, mSSIM, and SAM evaluation indexes.The compared algorithms differ across dead-line, SLC-off, and thick-cloud reconstruction tasks.Source: A. Experimental Settings · p. 7

B. Simulated Experiments

Simulated experiments evaluate STS-CNN on Terra MODIS dead lines, Landsat SLC-off, thick-cloud removal, and their combination, with reported visual and quantitative advantages over comparison methods.

B. Simulated Experiments

  • Terra MODIS band 6 dead-line recovery is evaluated with five methods, including STS-CNN, using PSNR, SSIM, and CC.The simulated Terra experiment serves as an approximate evaluation for Aqua MODIS because the two sensors have similar design patterns.Source: 1) Simulated Dead Lines in Terra MODIS… · p. 7, Table I · p. 7
  • Compared with LF, HMLLSF, and WCLF, STS-CNN is evaluated against stripe noise arising from complex relationships between MODIS bands.The comparison includes enlarged recovery regions and quantitative results in Table I.Source: 1) Simulated Dead Lines in Terra MODIS… · pp. 7–8
  • STS-CNN performs better than state-of-the-art methods for simulated Landsat ETM+ SLC-off recovery and reduces spectral distortion.The comparative methods produce discontinuous detail features, while STS-CNN achieves superior quantitative assessment results.Source: 2) Simulated ETM+ SLC-Off in Landsat… · p. 8, Table II · p. 8
  • With both SLC-off and thick cloud, STS-CNN performs better than comparison methods in spectral-distortion reduction and quantitative assessment.The experiment evaluates LLHM, NSPI, WLR, and STS-CNN using mSSIM, mPSNR, CC, and SAM.Source: Table IV · p. 9
  • For simulated thick-cloud removal, STS-CNN performs well in reducing spectral distortion and achieves strong quantitative assessment results.The experiment compares STS-CNN with LLHM, mNSPI, WLR, STWR, and AWTC using Table III.Source: 3) Simulated Cloud Removal of Landsat… · pp. 8, 10, Table III · p. 9

4) Simulated TM Data with Both Cloud and SLC-Off

The combined SLC-off and thick-cloud simulation tests reconstruction under large missing areas, where STS-CNN outperforms comparison methods in visual completeness and quantitative assessment.

4) Simulated TM Data with Both Cloud and SLC-Off

  • The combined simulation evaluates reconstruction outputs from LLHM, NSPI, WLR, and STS-CNN for Landsat TM data with SLC-off and cloud cover.Quantitative evaluations use mSSIM, mPSNR, CC, and SAM.Source: 4) Simulated TM Data with Both Cloud… · p. 10
  • LLHM and mNSPI cannot completely recover cloud-covered regions, while WLR produces texture discontinuity in large missing areas.These limitations are reported for remote sensing data containing both SLC-off and thick cloud.Source: 4) Simulated TM Data with Both Cloud… · p. 10
  • STS-CNN performs better than the comparison methods in reducing spectral distortion and in the quantitative assessment.The reported comparison concerns the combined SLC-off and thick-cloud reconstruction task.Source: 4) Simulated TM Data with Both Cloud… · p. 10
  • In real Terra MODIS band 6 recovery, all methods produce reasonable overall outcomes, but comparative methods retain stripe noise in enlarged regions.The real-data comparison includes LF, HMLLSF, WCLF, AWTC, and STS-CNN.Source: 1) Dead Lines in Aqua MODIS Band 6 · p. 10

2) SLC-Off ETM+ Images

Real-data experiments show STS-CNN reconstructs Landsat ETM+ SLC-off gaps with less stripe noise and more continuous detail than comparison methods. In thick-cloud removal, it reduces spectral distortion and better restores texture details.

SLC-Off ETM+ Images

  • STS-CNN recovered Landsat ETM+ SLC-off gaps with the least stripe noise among the compared methods.Unlike LLHM, NSPI, and WLR, it did not require LPRM to complete residual gaps.Source: 2) SLC-Off ETM+ Images · p. 10
  • The proposed method preserved detail texture while producing more consistent and continuous reconstructions for dead pixels.Source: 2) SLC-Off ETM+ Images · p. 11

Cloud Removal of TM Images

  • In real TM cloud-removal experiments, competing methods showed spectral distortion within the largest cloud areas.Source: 3) Cloud Removal of TM Images · p. 11
  • STS-CNN reduced spectral distortion and reconstructed cloud-region texture better than WLR and STWR.Its spectral distortion was less than LLHM, mNSPI, and AWTC, while texture details were better than WLR and STWR.Source: 3) Cloud Removal of TM Images · p. 11

1) Analysis of the Proposed Network Components

Component analyses evaluate multi-scale feature extraction, dilated convolution, and spatial-temporal/spectral information boosting, while registration-error tests assess robustness. The reported results support the network design and show STS-CNN degrades more slowly than comparison methods as misregistration increases.

Analysis of the Proposed Network Components

Effects of Image Registration Errors

Conclusions

  • The paper concludes that STS-CNN improves reconstruction accuracy and visual perception across simulated and real-data experiments.The framework addresses dead lines, ETM+ SLC-off gaps, and thick-cloud removal using spatial, spectral, and temporal information.Source: §V · p. 13
  • Thick-cloud removal can still produce spectral distortion and blurring, motivating future use of prior constraints to improve spectral fidelity and texture details.Source: §V · p. 13

Related papers

Loading 1802.08369v1…