Source-linked AI summary
DuPLO: A DUal view Point deep Learning architecture for time series classificatiOn
Roberto Interdonato, Dino Ienco, Raffaele Gaetano, Kenji Ose
TL;DR
The paper addresses how to represent dependencies in Satellite Image Time Series more completely. It proposes DuPLO, combining CNN and RNN branches, and reports better quantitative and qualitative results than state-of-the-art classification methods on two real-world study sites.
Problem
The paper addresses how to represent dependencies in Satellite Image Time Series more completely.
Method
DuPLO combines CNN and RNN branches to produce a more diverse and complete representation of Satellite Image Time Series data.
Results
DuPLO achieves better quantitative and qualitative results than state-of-the-art classification methods on two real-world study sites.
Takeaways & Limitations
Combining CNN and RNN branches provides a more diverse and complete representation for Satellite Image Time Series classification.
Takeaways & Limitations
The authors report that the RNN module overfits and identify extending the proposed approach as future work.
Abstract
from arXiv · showhide
Nowadays, modern Earth Observation systems continuously generate huge amounts of data. A notable example is represented by the Sentinel-2 mission, which provides images at high spatial resolution (up to 10m) with high temporal revisit period (every 5 days), which can be organized in Satellite Image Time Series (SITS). While the use of SITS has been proved to be beneficial in the context of Land Use/Land Cover (LULC) map generation, unfortunately, machine learning approaches commonly leveraged in remote sensing field fail to take advantage of spatio-temporal dependencies present in such data. Recently, new generation deep learning methods allowed to significantly advance research in this field. These approaches have generally focused on a single type of neural network, i.e., Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), which model different but complementary information: spatial autocorrelation (CNNs) and temporal dependencies (RNNs). In this work, we propose the first deep learning architecture for the analysis of SITS data, namely \method{} (DUal view Point deep Learning architecture for time series classificatiOn), that combines Convolutional and Recurrent neural networks to exploit their complementarity. Our hypothesis is that, since CNNs and RNNs capture different aspects of the data, a combination of both models would produce a more diverse and complete representation of the information for the underlying land cover classification task. Experiments carried out on two study sites characterized by different land cover characteristics (i.e., the \textit{Gard} site in France and the \textit{Reunion Island} in the Indian Ocean), demonstrate the significance of our proposal.
1. Introduction
Satellite Image Time Series support land-cover analysis by exposing temporal behavior, but conventional methods often ignore temporal dependencies. DuPLO addresses this gap by combining CNNs and RNNs to model complementary spatial and temporal information, with experiments showing effectiveness across two study sites.
- Motivation: SITS can improve LULC discrimination by capturing temporal behaviors that single images cannot represent.Their use has expanded across applications including ecology, agriculture, mobility, health, risk monitoring, and land management.
- Problem: Standard machine-learning approaches applied to stacked SITS images assume spectral bands and timestamps are independent, ignoring temporal dependencies.This limitation is identified for methods such as Random Forest and SVM.
- Deep learning approaches: CNNs model spatial autocorrelation, whereas RNNs capture temporal correlations in multidimensional time series.The two model families therefore address different aspects of SITS data.
- Related work: Previous SITS deep-learning studies generally focused on either convolutional or recurrent networks rather than combining both.Existing work included CNN-based land-cover classification and RNN-based change detection or optical and radar SITS analysis.
- Contribution: DuPLO combines CNN and RNN branches to exploit their complementary information and produce a more diverse, complete representation for SITS classification.The paper presents DuPLO as the first deep-learning architecture combining these approaches for satellite image time series.
- Evaluation: Experiments on the Gard site in France and Reunion Island demonstrate effectiveness against state-of-the-art approaches for land-cover mapping on SITS data.Quantitative and qualitative results also favor combining CNN and RNN models over using a single neural-network model.
2. DuP LO: A DUal view Point deep Learning architecture for time series classificatiOn
DuPLO analyzes Sentinel-2 satellite image time series by combining CNN and RNN branches that extract complementary spatial and temporal information. The CNN processes the series as a stacked image, while the RNN branch uses shallow spatial processing before modeling temporal dependencies.
- Architecture: DuPLO processes Sentinel-2 image time series through separate CNN and RNN branches that provide complementary information for classification.The CNN and RNN branches independently summarize extracted knowledge before their outputs are combined.
- CNN Branch: The CNN branch treats the complete time series as a stacked image and progressively increases feature-map capacity through convolutional layers.The series length is represented by T, the number of timestamps; the network is inspired by VGG-style architectures.
- Training: The proposed deep architecture may be difficult to train and can suffer from overfitting, so dropout is added after batch normalization with a 0.4 drop rate.A 0.4 drop rate randomly deactivates 40% of neurons at each propagation step.
- RNN Branch: The RNN branch first applies a shallow CNN to each timestamp patch, then feeds the resulting 64-dimensional feature sequence into a GRU with attention.This design extracts spatial-neighborhood information before modeling temporal behavior and combines information across timestamps.
- RNN Branch: The GRU uses reset and update gates to combine current and previous information, enabling hidden units to capture dependencies over different time scales.An attention mechanism learns weights over the GRU outputs so contributions from different timestamps can be combined into one feature vector.
3. Data
The study uses Sentinel-2 time series from Reunion Island and Gard, with different acquisition periods, spatial extents, spectral inputs, and ground-truth inventories. Ground truth is rasterized at 10 m resolution and spans 13 classes in Reunion Island and 8 in Gard.
- Reunion Island contains 34 Sentinel-2 images acquired between April 2016 and May 2017, while Gard contains 37 images acquired between December 2015 and January 2017.
- Cloudy observations are filled using linear multi-temporal interpolation, and polygon ground truth is manually corrected before rasterization at Sentinel-2 resolution.
- Each pixel includes four surface reflectances plus NDVI, yielding five variables per image in the time series.
- The study sites differ substantially in spatial extent: Reunion Island is 6 656 × 5 913 pixels and Gard is 4 822 × 6 748 pixels.
- Ground truth comprises 322 748 pixels across 13 Reunion Island classes and 1 157 260 pixels across 8 Gard classes.
4. Experiments
Experiments evaluate DuPLO through ablations, quantitative comparisons with Random Forest and LSTM, and qualitative land-cover-map analysis. Across both benchmarks, DuPLO generally outperforms competing methods, while RF(DuPLO) often gives the best average performance.
- The evaluation combines component ablations, global and per-class quantitative comparisons, and qualitative analysis of land-cover maps.
- Ablation Analysis: The convolutional and recurrent branches achieve similar Accuracy, F-Measure, and Kappa values when evaluated separately.
- Ablation Analysis: DuPLO outperforms the variant without auxiliary classifiers and variants using only the convolutional or recurrent branch on both datasets.
- Comparative Analysis: DuPLO outperforms the competing state-of-the-art methods on both benchmarks, while RF(DuPLO) provides the best average Accuracy, F-Measure, and Kappa performances.
- Comparative Analysis: On the Reunion benchmark, the framework achieves the best classification results for nine of thirteen land-cover categories, with gains of 8, 9, and 20 points for selected classes.
- Comparative Analysis: On Gard, DuPLO improves by more than 27 percentage points over the best state-of-the-art method, while qualitative maps show sharper and more spatially coherent boundaries.
5. Conclusion
DuPLO combines CNN and RNN branches to learn complementary representations of optical SITS for land-cover classification. Evaluations on two real-world sites report better quantitative and qualitative results than state-of-the-art methods, while future work targets multisource data integration.
- Architecture: The architecture obtains descriptors designed to discriminate different land-cover classes from optical SITS data.
- Architecture: DuPLO combines CNN and RNN branches processing the same information to produce diverse representations for land-cover classification.The branch features are concatenated, and the framework is learned end-to-end from scratch.
- Results: DuPLO achieves better quantitative and qualitative results than state-of-the-art optical-SITS classification methods on two real-world study sites.Visual inspection of land-cover maps also supports the effectiveness of the strategy.
- Future work: Future work will extend the approach to multisource scenarios combining optical and radar SITS, such as Sentinel-2 and Sentinel-1.