PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery

Lichao Mou, Lorenzo Bruzzone, Xiao Xiang Zhu

arXiv:1803.02642v1cs.CV

TL;DR

Change detection requires modeling nonlinear temporal relationships in bi-temporal multispectral imagery while exploiting useful spectral and spatial information. The paper proposes an end-to-end ReCNN that combines CNN-based spectral-spatial extraction with RNN-based temporal modeling. Experiments on real multispectral images report competitive performance against conventional and recurrent baselines.

  • Problem

    Temporal correlations in bi-temporal imagery are inherently nonlinear, while change detection requires joint spectral-spatial-temporal feature learning.

  • Method

    ReCNN combines convolutional layers, a recurrent sub-network, and fully connected layers to learn features and predict change labels end to end.

  • Results

    ReCNN achieves competitive performance on real multispectral images against conventional change-detection models and a spectral-temporal RNN algorithm.

  • Takeaways & Limitations

    The framework jointly captures spectral, spatial, and temporal information from bi-temporal multispectral images and can be applied to other remote-sensing sequence-prediction problems.

Abstract

from arXiv · show

Change detection is one of the central problems in earth observation and was extensively investigated over recent decades. In this paper, we propose a novel recurrent convolutional neural network (ReCNN) architecture, which is trained to learn a joint spectral-spatial-temporal feature representation in a unified framework for change detection in multispectral images. To this end, we bring together a convolutional neural network (CNN) and a recurrent neural network (RNN) into one end-to-end network. The former is able to generate rich spectral-spatial feature representations, while the latter effectively analyzes temporal dependency in bi-temporal images. In comparison with previous approaches to change detection, the proposed network architecture possesses three distinctive properties: 1) It is end-to-end trainable, in contrast to most existing methods whose components are separately trained or computed; 2) it naturally harnesses spatial information that has been proven to be beneficial to change detection task; 3) it is capable of adaptively learning the temporal dependency between multitemporal images, unlike most of algorithms that use fairly simple operation like image differencing or stacking. As far as we know, this is the first time that a recurrent convolutional network architecture has been proposed for multitemporal remote sensing image analysis. The proposed network is validated on real multispectral data sets. Both visual and quantitative analysis of experimental results demonstrates competitive performance in the proposed mode.

I. INTRODUCTION

Change detection supports important multitemporal earth-observation applications, but modeling temporal correlations in bi-temporal imagery is difficult because observations are inherently nonlinear. The paper proposes ReCNN to learn joint spectral-spatial-temporal representations end to end.

  • Change detection identifies land-cover differences across time for applications including urban expansion, disaster assessment, resource management, and land-use monitoring.
  • Atmospheric scattering, light-scattering mechanisms, and intra-class variability make temporal correlation modeling inherently nonlinear, motivating sophisticated task-driven learning methods.
  • ReCNN combines CNN and RNN components to learn joint spectral-spatial-temporal features from bi-temporal multispectral data in an end-to-end framework.
  • The CNN transforms paired 3D multispectral patches into abstract spectral-spatial representations, while the RNN models temporal dependency and predicts changed, unchanged, or change-type labels.
  • The proposed architecture learns informative features directly from multispectral data without hand-crafted visual features or preprocessing steps.
  • Its RNN component models temporal correlation through a task-driven end-to-end approach and produces labels for the image sequence.

II. METHODOLOGY

The ReCNN methodology combines convolutional feature extraction, recurrent temporal modeling, and fully connected classification in one differentiable network. It processes paired patches sequentially and predicts change labels from the recurrent representation.

  • ReCNN consists of a convolutional sub-network, a recurrent sub-network, and fully connected layers trained jointly with back-propagation using one loss function.
  • The network receives paired multispectral images or patches acquired at two times over the same geographical area, with labels for changed, unchanged, or change-type categories.
  • Separate identical convolutional branches process the T1 and T2 patches to produce spectral-spatial feature representations for recurrent processing.
  • The recurrent sub-network updates and retains hidden-state information from the first patch before processing the second patch with that state.
  • Fully connected layers use the sequence representation to predict the input patches’ label, and applying the network across pixels produces a change-detection map.

B. Spectral-Spatial Feature Extraction via the Convolutional Sub-Network

The convolutional sub-network is designed to learn spectral-spatial features while preserving image structure and controlling model size. It uses paired branches and dilated convolutions to incorporate broader context efficiently.

  • The network uses trainable feature generation because image-algebra and transformation methods may not capture task-specialized discriminative features.
  • Fully connected 1D models can require many learnable parameters and treat multispectral imagery as vectors, ignoring its 2D spatial structure.
  • CNNs automatically learn contextual 2D spatial and spectral features while reducing parameters through local connections and weight sharing.
  • Dilated convolution enlarges the network’s field of view, with dilation rate 1 corresponding to traditional convolution.
  • The convolutional sub-network processes paired 5×5 multispectral patches through two identical branches before passing their features to the recurrent sub-network.
  • Using dilated convolution exponentially enlarges the field of view with linearly increasing parameters, reducing parameters while increasing effective context.

C. Modeling Temporal Dependency by the Recurrent Sub-Network

The recurrent sub-network models temporal dependency in bi-temporal multispectral imagery using recurrent hidden states, with fully connected RNN, LSTM, and GRU variants. It uses the final recurrent representation to predict change labels.

  • Recurrent modeling: RNNs model dependent sequential inputs through recurrent hidden states, unlike feedforward networks that implicitly treat inputs as independent.This makes recurrent models suitable for processing temporal sequences in change detection.
  • Fully connected RNN: A fully connected RNN updates its hidden state from convolutional feature vectors and can model the conditional probability of the second image representation given the first.The bi-temporal images are treated as sequential data rather than as a difference or stacked image.
  • Prediction: The recurrent layer outputs a sequence of hidden states, but ReCNN uses only the final state as input to fully connected layers for label prediction.The resulting labels can be generated across the image by applying the network to all pixels.
  • Model comparison: Figure 3 compares the through-time graphical models of fully connected RNN, LSTM, and GRU and labels their principal gates, memory contents, and activations.The caption identifies LSTM output, forget, and input gates, and GRU reset and update gates.
  • LSTM: LSTM maintains memory cells whose contents are selectively forgotten and updated through input and forget gates.Output gates control how much memory content is exposed, while the new memory contents are computed from current features and the previous hidden state.
  • GRU: GRU interpolates between the previous and candidate activations using update gates, while reset gates control how much previous-state information is retained.Unlike LSTM, GRU directly exposes whole state values at each time step.

D. Network Training

The network is trained in TensorFlow with Nesterov Adam, using fixed optimizer settings, a small learning rate, Glorot initialization, and task-specific final-layer activations.

  • Optimization: The network is trained with Nesterov Adam because it shows faster convergence for this task than SGD with momentum or Adam.The learning rate is 2e-04, with β1 = 0.9, β2 = 0.999, ϵ = 1e−08, and schedule decay 0.004.
  • Network setup: Glorot uniform initialization is used, with sigmoid for binary detection and softmax for multi-class detection in the final fully connected layer.Training uses a single NVIDIA GeForce GTX TITAN with 12 GB of GPU memory.

A. Data Description

The experiments use two Landsat ETM+ multispectral data sets with six bands and 30 m spatial resolution. The Taizhou data cover city expansion and other labeled change classes, while the Eppalock scene contains four multi-class change types.

  • Data and preprocessing: Both data sets were acquired by the Landsat ETM+ sensor, contain six bands at 30 m spatial resolution, and are normalized to [0,1] after radiance conversion.The preprocessing converts original digital numbers into absolute radiance before normalization.
  • Taizhou data set: The Taizhou data set contains two 400 × 400 images from March 2000 and February 2003, with changes mainly involving city expansion.Its manually annotated multi-class samples include unchanged area, city expansion, and soil change classes.

1) Taizhou Data:

The Eppalock Lake experiments evaluate binary and multi-class change detection using standard accuracy criteria and comparisons with classical, supervised, unsupervised, and recurrent methods.

  • Data set: The Eppalock Lake data set contains four multi-class change types, including unchanged regions, city expansion, water loss, and soil change.The images were acquired in February 1991 and March 2009, and both contain 602 × 631 pixels.
  • Evaluation criteria: The experiments use overall accuracy and the Kappa coefficient to evaluate change detection maps.Overall accuracy measures correctly classified bi-temporal pixels, while Kappa measures agreement with ground truth after correcting for chance agreement.
  • Compared methods: CVA, PCA, MAD, and IRMAD are unsupervised change detection methods based on image algebra or transformed feature representations.IRMAD extends MAD with an iterative scheme, while PCA is noted for computational simplicity.
  • Compared methods: The compared recurrent variants include a fully connected recurrent sub-network, a GRU-based sub-network, and an LSTM-based sub-network.The SVM baseline uses an RBF kernel with C and γ selected by five-fold cross-validation.
  • Compared methods: Binary experiments compare CVA, PCA, MAD, IRMAD, and RNN, while multi-class experiments compare decision tree, SVM, and RNN with the proposed network.K-means automatically selects thresholds for unsupervised methods in the binary task.

C. Analysis of Recurrent Sub-network: Comparisons between Fully Connected RNN, LSTM, and GRU

The recurrent-subnetwork analysis compares fully connected RNN, LSTM, and GRU variants within ReCNN. Across the reported experiments, the LSTM variant performs best, while the three variants remain close overall.

  • Recurrent architecture: Fully connected RNN, LSTM, and GRU architectures share additive recurrent updates, unlike conventional feedforward CNNs.LSTM and GRU preserve current content while adding new content; LSTM uses three gates.
  • Results: ReCNN-LSTM outperforms ReCNN-FC and ReCNN-GRU on all reported overall accuracy and Kappa indexes across both data sets.The paper reports this pattern for both binary and multi-class change detection.
  • Results: 0.38% OA and 0.0122 Kappa are the Taizhou binary-task gains of ReCNN-LSTM over ReCNN-FC.Against ReCNN-GRU on the same task and data set, the gains are 0.06% OA and 0.0021 Kappa.
  • Results: The three ReCNN variations perform closely to each other, while the gated ReCNN-LSTM and ReCNN-GRU slightly outperform ReCNN-FC on both data sets and tasks.This indicates only modest separation among the recurrent-subnetwork variants in the reported experiments.

D. Analysis of Spatial Component: RNN vs ReCNN-LSTM

The spatial-component analysis compares spectral-temporal RNN with spectral-spatial-temporal ReCNN-LSTM. ReCNN-LSTM improves quantitative accuracy and removes noisy scattered detections in the reported maps.

  • Quantitative comparison: 2.23% OA and 0.0708 Kappa are the Taizhou binary-task gains of ReCNN-LSTM over RNN.For Eppalock Lake, the corresponding binary gains are 3.46% OA and 0.071 Kappa.
  • Quantitative comparison: 2.56% OA and 0.0905 Kappa are the Taizhou multi-class gains of ReCNN-LSTM over RNN.For Eppalock Lake, the corresponding multi-class gains are 2.36% OA and 0.036 Kappa.
  • Visual comparison: RNN produces noisy scatter points in the change detection map, whereas ReCNN-LSTM eliminates noisy scattered points of wrong detection.The comparison concerns the visual quality of spectral-temporal versus spectral-spatial-temporal detection maps.
  • Model-size comparison: The compared RNN architectures use 128 recurrent units each when model size is evaluated.The figure compares total trainable parameters across the different RNN architectures.

E. Comparison with Other Approaches

ReCNN combines convolutional and recurrent processing for joint spectral-spatial-temporal feature extraction and achieves competitive change-detection performance on real multispectral images. Quantitative comparisons show gains over established competitors, while future work targets reduced dependence on labeled data.

  • 3.59% and 0.1279 improvements in OA and Kappa over IRMAD were achieved on Taizhou, while Eppalock lake gains were 7.4% and 0.1554.
  • 4.14% and 2.56% OA increases over SVM and RNN were obtained on Taizhou, compared with 2.84% and 2.36% on Eppalock lake.
  • ReCNN integrates CNN and RNN components to extract joint spectral-spatial-temporal features from bi-temporal multispectral images and predict change types.
  • Experiments on real multispectral images demonstrate competitive performance against conventional change-detection models and a spectral-temporal RNN algorithm.
  • Future work proposes semi-supervised ReCNN architectures using small amounts of labeled data together with larger amounts of unlabeled data.
Loading 1803.02642v1…