Self-attention for raw optical Satellite Time Series Classification

Marc Rußwurm; Marco Körner

Source-linked AI summary

Self-attention for raw optical Satellite Time Series Classification

Marc Rußwurm, Marco Körner

arXiv:1910.10536v3Oct 23, 2019cs.LGeess.IVstat.ML

TL;DR

The paper addresses how to exploit growing Earth observation archives without relying heavily on region- or problem-specific preprocessing. It compares convolutional, recurrent, self-attention, and Random Forest models on raw and preprocessed Sentinel 2 time series, finding that preprocessing generally improves performance while Transformers and recurrent models are stronger than convolutional models on raw data.

Problem
Earth observation data is growing rapidly, but many remote-sensing methods still require preprocessing or explicit feature design, limiting use of raw sensory data without expert knowledge.
Method
The study compares convolutional, recurrent, self-attention, and Random Forest models for crop type classification on raw and preprocessed Sentinel 2 time series.
Results
Preprocessing improved classification performance across models, while Transformers and recurrent networks outperformed convolutional networks on noisy raw time series.
Takeaways & Limitations
Robust classification of raw satellite time series without region-specific expert knowledge is identified as important for quantitatively exploiting newly published satellite data.
Takeaways & Limitations
The evaluation used one type of preprocessed data, so comparisons with models using distinct model-specific preprocessing are needed to confirm the findings generally.

Abstract

from arXiv · show

The amount of available Earth observation data has increased dramatically in the recent years. Efficiently making use of the entire body information is a current challenge in remote sensing and demands for light-weight problem-agnostic models that do not require region- or problem-specific expert knowledge. End-to-end trained deep learning models can make use of raw sensory data by learning feature extraction and classification in one step solely from data. Still, many methods proposed in remote sensing research require implicit feature extraction through data preprocessing or explicit design of features. In this work, we compare recent deep learning models on crop type classification on raw and preprocessed Sentinel 2 data. We concentrate on the common neural network architectures for time series, i.e., 1D-convolutions, recurrence, a shallow random forest baseline, and focus on the novel self-attention architecture. Our central findings are that data preprocessing still increased the overall classification performance for all models while the choice of model was less crucial. Self-attention and recurrent neural networks, by their architecture, outperformed convolutional neural networks on raw satellite time series. We explore this by a feature importance analysis based on gradient back-propagation that exploits the differentiable nature of deep learning models. Further, we qualitatively show how self-attention scores focus selectively on few classification-relevant observations.

1. Introduction

Earth observation archives are expanding faster than they are being used, while preprocessing remains a major barrier to exploiting raw satellite data. The paper evaluates convolution, recurrence, and self-attention models on raw and preprocessed data for crop type identification.

Sentinel 2 provides imagery at up to 10 m resolution across 13 spectral bands every two to five days.
Only 7.6% of published Earth observation images in 2018 were downloaded, leaving 12 out of 13 images unused.
Preprocessing is common in remote sensing, including atmospheric correction and manual or automatic cloud filtering.
Deep learning can jointly learn feature extraction and classification from data through gradient backpropagation.
The study presents a large-scale comparison of convolutional, recurrent, and self-attention models on raw and preprocessed satellite data for crop type identification.
It also provides quantitative and qualitative analyses of self-attention in deep learning models for satellite time series classification.

2. Related Work

Remote-sensing time-series methods have traditionally separated feature extraction from classification, often using temporal statistics, fitted phenological curves, or domain-specific algorithms. Neural networks instead learn feature extraction and classification jointly from the provided data.

Multitemporal satellite data supports vegetation applications, but limited availability and computational demands have hindered broad use of temporal information.
Traditional workflows calculate vegetation indices and temporal statistics before applying classifiers such as Random Forests or Support Vector Machines.
TimeSat fits non-symmetric Gaussian curves whose parameters represent phenological characteristics and can serve as classification features.
CCDC and related methods are tailored toward specific data types, whereas artificial neural networks learn feature extraction and classification jointly from the provided data.

3. Method

The method formalizes deep models as differentiable mappings trained by minimizing prediction loss, then introduces convolution, recurrence, attention, and self-attention for temporal feature extraction.

General framework: Deep learning models map input time series to class predictions through cascaded differentiable layers whose parameters minimize an objective comparing labels and predictions.Optimization uses mini-batch stochastic gradient descent with batch-averaged gradients and a learning-rate-controlled step size.
Recurrence: Recurrent layers incorporate previous time-step context, while LSTM gates regulate forgetting, writing, and output through an additional cell-state matrix.GRUs use the same gated principle with fewer gates and weights than LSTMs.
Convolution: Convolutional layers correlate a time series with kernels, extracting features from a fixed temporal neighborhood whose receptive field depends on kernel size and depth.The generic operation is H = φ(X ∗ Θ).
Attention: Attention extracts features through weighted sums of value observations, with scores produced from key-query products and normalized by softmax.The attention matrix generates output vectors in parallel across the sequence.
Self-attention: Self-attention uses equal input and output dimensions, derives keys, queries, and values from the same input through linear transformations, and scales scores by √Dh.The scaling is included for improved gradient backpropagation.
Soft-attention: DuPLO’s soft-attention is a special attention case with values equal to the input, input-derived queries, and a learned fixed key matrix.The paper compares this simplified soft-attention with self-attention in a dedicated experiment.

4. Models

The study evaluates four neural-network topologies and a Random Forest baseline, covering recurrent, self-attention, residual-convolutional, temporal-convolutional, and hybrid architectures.

Overview: The study evaluates four neural-network topologies using convolution, recurrence, or self-attention as temporal feature-extraction mechanisms.These topologies are evaluated experimentally in the following section.
LSTM-RNN: The LSTM-RNN stacks bidirectional LSTM layers, concatenates the final forward and backward hidden states, and classifies them with a softmax output layer.The number of recurrent layers and dropout are tuned.
Transformer: The Transformer adds positional encoding, applies stacked multihead self-attention blocks with dense layers, and globally max-pools the final representation before classification.Skip connections and layer normalization are used throughout the blocks.
MS-ResNet: MS-ResNet uses convolutional residual connections adapted to time series, combining branch representations before a final softmax classifier.Its design requires fixed sequence length T = 512, obtained by nearest-neighbor interpolation.
TempCNN: TempCNN is a lightweight model with three sequential 1D convolutional layers followed by regularization and fully connected classification layers.Its tunable parameters include hidden dimensionality, kernel sizes, and dropout probability.
DuPLO: DuPLO combines a CNN stream for spatial features with a CNN-GRU stream for temporal characteristics and aggregates their representations through soft-attention.The architecture was adapted by treating time series as single-pixel images.
Random Forest: The Random Forest baseline augments raw reflectance inputs with NDVI, NDWI, BI, IRECI, and EVI spectral indices.The augmentation reflects the usual reliance on feature extraction for strong shallow-model performance.

5. Training Details

Deep-learning models are trained with Adam-based mini-batch stochastic gradient descent, while learning-rate and weight-decay settings are selected through hyperparameter search.

Optimization: The LSTM-RNN, MS-ResNet, and TempCNN use mini-batch stochastic gradient descent with Adam momentum parameters β1 = 0.9 and β2 = 0.98.Learning rate and weight decay are sampled from log-uniform ranges and selected through hyperparameter tuning; the Transformer uses a learning-rate scheduler.

6. Model Selection

Model selection samples candidate hyperparameters, trains models on part of the training partition, and evaluates them on a validation set.

Search space: The search varies architecture and optimization hyperparameters, including hidden dimensions, layer counts, attention heads, kernel sizes, dropout, warmup steps, learning rate, and weight decay.The available hyperparameters differ across models and datasets.
Selection procedure: Candidate settings are sampled from a hyperparameter space, trained on a subset of the training partition, and evaluated on a validation set.This procedure determines the optimal parameter set used for model selection.

7. Data

The study evaluates crop-type identification using spatially separated Bavarian regions and raw or preprocessed Sentinel 2 time series. It describes the satellite data, parcel-level feature construction, labels, and preprocessing differences.

7.1. Study Area and Labels: Crop-type identification uses temporal phenology signals to distinguish vegetation classes across three spatially separate Bavarian regions.The regions are Hollfeld, Krumbach, and northeastern Bavarian Forest, approximately 100 km apart.
7.1. Study Area and Labels: The study labels follow a long-tailed distribution with more than 269 crop categories, while 15, 26, and 62 categories cover 90%, 95%, and 99% of parcels.Categories were aggregated and selected with StMELF and GAF AG for the ministry’s objectives.
7.2.1. Raw Dataset: Raw Sentinel 2 data consists of top-of-atmosphere L1C reflectances, mean-aggregated within each parcel into 13-band feature vectors at each observation time.Cloud coverage produces positive reflectance peaks that dominate the raw signal and hinder visual interpretation.
7.2.2. Preprocessed Dataset: Preprocessing enables visual interpretation of growth onset and cutting events but introduces repeated values through temporal interpolation and cloud removal.Raw observations retain most measured-signal information, while atmospheric and cloud noise obscures phenological events.
7.2.2. Preprocessed Dataset: The preprocessed dataset applies atmospheric correction, cloud-free temporal selection, vegetative-period focus, and cloud masking, making it easier to classify.A separate cloud classification model identifies and filters cloudy observations.

8. Experiments and Results

The experiments compare recurrent, self-attention, convolutional, and random-forest models on raw and preprocessed Sentinel 2 crop-classification data. Preprocessing generally improves performance, while recurrence and self-attention show particular advantages on noisy raw series and selectively use temporal observations.

Quantitative Model Evaluation: The evaluation compares LSTM-RNN, Transformer, MS-ResNet, DuPLO, TempCNN, and random forest across preprocessed and raw Sentinel 2 datasets.Experiments cover 23-class land-use and 12-class land-cover categorizations, using separately tuned hyperparameters.
Quantitative Model Evaluation: Manual preprocessing improved classification performance for all models, while model differences were generally less important on preprocessed data.The authors attribute the advantage partly to preprocessing exposing classification-relevant events before end-to-end learning.
Quantitative Model Evaluation: Random forest was competitive on preprocessed data but fell behind deep learning models on raw data, with weaker kappa and F1 performance.The reported kappa deficit was about 0.15 for the 23-class variant and 0.1 to 0.2 for the 12-class variant.
Qualitative Model Evaluation: On raw 12-class data, classification accuracies did not vary significantly across deep learning models, so no clear model advantage was visible qualitatively.The comparison used predictions for land-cover classes from a held-out spatial block.
Temporal Feature Importance by Gradient Backpropagation: Gradient analysis found that LSTM-RNN and Transformer predictions relied on comparatively few cloud-free observations, whereas convolutional models used broader temporal information.Cloud-induced positive reflectance peaks generally had vanishing gradients for recurrent and Transformer models, while convolutional models sometimes retained cloudy observations.
Self-Attention and Feature Analysis: Transformer self-attention heads focused on distinct events and avoided strong cloud-related peaks, while DuPLO soft attention assigned nonzero scores across neighboring time points.The attention analysis also reports progressively more separable learned embeddings for classes such as corn and rapeseed.

9. Conclusion

The study quantitatively and qualitatively analyzed self-attention alongside recurrence, convolution, and a random-forest baseline on raw and preprocessed Sentinel 2 time series. Preprocessing reduced the importance of architecture, while feature embeddings became more separable through Transformer self-attention layers and robust raw-data classification remains a future priority.

The authors analyzed self-attention quantitatively and qualitatively across models using self-attention, recurrence, convolution, and a random-forest baseline.
All models performed equally well on preprocessed data, and the random-forest baseline achieved competitive overall accuracy.
Preprocessing made model architecture less critical than it was for raw Sentinel 2 time series.
Each self-attention layer increased the separability of features extracted by the Transformer network.
Robust classification of raw time series without region-specific expert knowledge is identified as a future requirement for exploiting daily satellite data.