Source-linked AI summary
Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series
Charlotte Pelletier, Geoffrey I. Webb, Francois Petitjean
TL;DR
Accurate, up-to-date land-cover mapping from SITS requires methods that use temporal information more explicitly than traditional classifiers. This paper extensively evaluates TempCNNs for SITS classification and finds them more accurate than RF and RNNs while providing architecture guidelines and visual map assessment.
Problem
Traditional SITS classifiers perform well but do not explicitly use temporal relationships, motivating evaluation of deep learning methods for large, long SITS.
Method
The paper extensively studies TempCNNs, applying convolutions in the temporal domain and evaluating architectures, regularization, batch sizes, spectral features, and pooling choices.
Results
TempCNN architectures outperform RF and RNN algorithms by 1 to 3 % in experiments on 46 Formosat-2 images, with visual analysis showing good land-cover mapping quality.
Takeaways & Limitations
TempCNNs are a strong learner for SITS classification, but pooling layers and the use of both temporal and spectral dimensions require careful study.
Takeaways & Limitations
Salt-and-pepper noise in the produced maps indicates that spatial information should be considered alongside spectral and temporal dimensions.
Abstract
from arXiv · showhide
New remote sensing sensors now acquire high spatial and spectral Satellite Image Time Series (SITS) of the world. These series of images are a key component of classification systems that aim at obtaining up-to-date and accurate land cover maps of the Earth's surfaces. More specifically, the combination of the temporal, spectral and spatial resolutions of new SITS makes possible to monitor vegetation dynamics. Although traditional classification algorithms, such as Random Forest (RF), have been successfully applied for SITS classification, these algorithms do not make the most of the temporal domain. Conversely, some approaches that take into account the temporal dimension have recently been tested, especially Recurrent Neural Networks (RNNs). This paper proposes an exhaustive study of another deep learning approaches, namely Temporal Convolutional Neural Networks (TempCNNs) where convolutions are applied in the temporal dimension. The goal is to quantitatively and qualitatively evaluate the contribution of TempCNNs for SITS classification. This paper proposes a set of experiments performed on one million time series extracted from 46 Formosat-2 images. The experimental results show that TempCNNs are more accurate than RF and RNNs, that are the current state of the art for SITS classification. We also highlight some differences with results obtained in computer vision, e.g. about pooling layers. Moreover, we provide some general guidelines on the network architecture, common regularization mechanisms, and hyper-parameter values such as batch size. Finally, we assess the visual quality of the land cover maps produced by TempCNNs.
1. Introduction
Accurate, up-to-date land cover maps are important for environmental monitoring, resource management, disaster prevention, modeling, and policy. New satellite image time series expose temporal information that traditional pixel-level classifiers do not explicitly use, motivating deep-learning approaches such as TempCNNs.
- Accurate land cover maps support environmental research, climate-change monitoring, resource management, disaster prevention, modeling systems, and public policy.
- New satellite constellations acquire high-resolution satellite image time series that provide valuable data for regional and continental land-cover, vegetation, and crop mapping.
- Random Forests and Support Vector Machines applied to stacked multispectral pixels are oblivious to acquisition order, losing temporal behavior relevant to some land-cover classes.
- Hand-crafted temporal features and temporal-similarity nearest-neighbor methods offer alternatives, but reported benefits are limited or computationally prohibitive.
- RNNs explicitly model temporal correlations but may be harder and slower to train for whole-series labels because back-propagation depends on series length.
- The paper extensively studies TempCNNs, evaluates temporal convolutions and spectral features, and explores architecture choices against RFs and RNNs.
2. Temporal Convolutional Neural Networks
TempCNNs apply shared convolutional filters along multivariate time series, combining temporal feature extraction with dense classification layers. The paper introduces a baseline architecture and discusses training, regularization, and architectural choices for SITS classification.
- 2.1. General Principles: Each neural-network layer computes a weighted combination of the previous layer’s outputs, adds biases, and applies a nonlinear activation function.The studied networks use ReLU activations, which are defined as ReLU(z) = max(0, z).
- 2.1. General Principles: Training minimizes a cost function based on prediction errors, using cross-entropy for multiclass classification and the Softmax probability assigned to the true class.The cost is described as the average loss over training instances.
- 2.2. Temporal Convolutions: Temporal convolution applies filters along a time series to detect signal patterns at different positions while sharing parameters across the input.A gradient filter produces high positive responses for sharp increases and negative responses for decreases.
- 2.3. Proposed network architecture: The architecture study examines network width, depth, filter size, pooling, and regularization rather than claiming an exhaustively optimized architecture.Regularization includes batch normalization, dropout at 0.5, L2 weight decay at 10^-6, and a validation split of 5%.
- 2.3. Proposed network architecture: A TempCNN receives a multivariate time series, applies three convolutional layers, then a dense layer and Softmax output for class probabilities.The experimental baseline uses 64-unit convolutional layers, a 256-unit dense layer, and filter size 5.
- 2.3. Proposed network architecture: Training uses Adam with batch size 32 for up to 20 epochs, with early stopping applied during optimization.The influence of batch size on accuracy and training time is evaluated separately.
3. Material and Methods
The study uses Formosat-2 SITS and labeled land-cover references, with preprocessing, feature construction, normalization, and RF/RNN benchmarks for TempCNN experiments.
- 3.1. Optical Satellite Data: The study area covers 24 km × 24 km near Toulouse, France, where approximately 60% of the soil is arable.
- 3.1. Optical Satellite Data: The dataset contains 46 Formosat-2 images acquired at 8-meter resolution during 2006, with acquisitions mainly concentrated in summer.
- 3.1. Optical Satellite Data: Three bands—Near-Infrared, Red, and Green—are used, while blue is discarded because it is sensitive to atmospheric artifacts.
- 3.2. Reference Data: Reference data define 13 classes spanning winter crops, summer crops, natural classes, and urban surfaces.
- 3.2. Reference Data: Reference polygons are split into 60% training and 40% testing, with five repeated splits whose results are averaged.
- 3.3.2. Feature extraction: The experiments compare NDVI, spectral bands, and combined spectral-band/index feature vectors for TempCNN models.
4. Experimental Results
The experiments evaluate how spectral and temporal guidance, architecture choices, and regularization affect TempCNN classification performance against benchmark models.
- Experimental design: Six experiments examine spectral and temporal guidance, filter size, pooling, model width and depth, regularization, and batch size.
- 4.1. Spectral and temporal guidance: The comparison includes RF, RNN, fully connected, and TempCNN models under four levels of spectral and temporal guidance.
- 4.1. Spectral and temporal guidance: CNN overall accuracy increases as more types of spectral and temporal guidance are added, regardless of the feature vector.
- 4.1. Spectral and temporal guidance: When spectral bands are included, TempCNNs exceed RNN, fully connected, and RF models by 1–3% in overall accuracy, with lower variation.
- 4.1. Spectral and temporal guidance: Adding spectral indexes to spectral bands does not improve accuracy for the traditional or deep-learning algorithms evaluated.
4.2. Influence of the filter size
The filter-size experiment relates temporal convolution reach to classification accuracy under two-day regular sampling.
- 4.2. Influence of the filter size: The tested filter sizes f = {3, 5, 9, 17, 33} correspond to convolution reaches of 2, 4, 8, 16, and 32 days.
- 4.2. Influence of the filter size: A reach of 8 days achieves the maximum overall accuracy, with similar accuracy at a reach of 4 days.
- 4.2. Influence of the filter size: The appropriate convolution reach depends mainly on the temporal patterns to be abstracted at the given temporal resolution.
4.3. Are local and global temporal pooling layers important?
Pooling layers generally reduce TempCNN accuracy in this SITS task, contrasting with common image-classification findings because temporal feature location and amplitude matter.
- Pooling configurations: The experiment compares local max-pooling, local average pooling, global average pooling, and their combinations across reaches of 2–32 days.
- Results: Pooling-layer models almost always achieve lower overall accuracy than the no-pooling baseline.
- Results: Global average pooling causes the largest accuracy decrease because its extreme dimensionality reduction reduces the final convolution representation.
- Results: Local max- and average-pooling produce similar results, with performance tending to decline as reach increases.
- Results: For reaches below nine days, local pooling is comparable to no pooling, with a slight improvement for local average pooling at a four-day reach.
- Interpretation: Unlike image classification, local pooling does not generally help because removing temporal peak location can hinder discrimination between shifted crop profiles.
4.4. How big and deep model for our data?
TempCNN accuracy is stable across a wide range of model sizes, while two or three convolutional layers provide the best depth setting in the tested architectures.
- Width influence or the bias-variance trade-off: 93.28% OA at 50M parameters versus 93.69% at 2.5M demonstrates robustness to large changes in model width.The standard deviation increases with parameter count, but most 50M-parameter results remain between 91% and 95% accuracy.
- Width influence or the bias-variance trade-off: About 2.5M parameters with three convolutional layers and one dense layer offers a good bias-variance trade-off for this dataset.
- Width influence or the bias-variance trade-off: With more training data, a similar architecture is likely to work well conservatively; with less data, a smaller architecture may be chosen because results are stable overall.
- Depth influence: Two or three convolutional layers achieve the highest accuracy with the lowest standard deviation at approximately 2.5M parameters.Inappropriate combinations of depth and units can underestimate TempCNN performance and may require expensive cross-validation or meta-learning to optimize.
4.5. How to control overfitting?
The study evaluates four mechanisms for controlling overfitting in TempCNNs and finds dropout to be the most influential individual mechanism.
- Dropout alone produces an OA value close to that obtained with all four regularization mechanisms.The other mechanisms are weight regularization, a validation set, and batch normalization.
- Validation-set use and weight decay appear less useful for regularizing the network than dropout.
4.6. What values are used for the batch size?
Across the tested batch sizes, batch size changes training time but not learned-model accuracy, supporting larger batches when memory permits.
- Table 6 reports training time and averaged Overall Accuracy over five folds for five batch sizes.
- All tested batch sizes produce comparable OA values, while batch size influences training time.The study evaluates batch sizes of 8, 16, 32, 64, and 128 for the same model.
- Large batch sizes can speed training when sufficient memory storage is available.
4.7. Visual analysis
RF and TempCNN maps appear visually similar overall but differ substantially in some class boundaries and objects; both remain sensitive to salt-and-pepper noise.
- The visual comparison uses blue and green areas, with RF and TempCNN maps shown alongside red disagreement images.The areas measure 3.7 km × 3.6 km, or 465 pixels × 450 pixels.
- Disagreement maps reveal strong RF–TempCNN differences in land-cover delineations and at the object level.The compared objects include crops, urban areas, and forests.
- RF spreads the majority urban class, causing over-detection especially for mixing pixels.
- RF confuses an urban area with sunflower crop in one object-level disagreement.
- Both classifiers are sensitive to salt-and-pepper noise, which may be addressed through post-processing or spatial information in the framework.
5. Conclusion
TempCNNs performed strongly for SITS classification, while experiments identified architectural recommendations and a remaining need to incorporate spatial information.
- TempCNN architectures outperformed RF and RNN algorithms by 1 to 3% on 46 Formosat-2 images.The evaluation used an extensive set of experiments on a series of 46 Formosat-2 images.
- TempCNNs accurately mapped land cover without overrepresenting majority classes in visual analysis.
- Global pooling layers harmed SITS classification, so the authors recommend studying pooling choices carefully and favoring local average global pooling.The authors also report that global pooling drastically reduces trainable parameters.
- Manually calculated spectral features such as NDVI did not seem to improve TempCNN models, so the authors recommend not using them.
- Salt-and-pepper noise indicates that SITS classification should incorporate spatial information alongside spectral and temporal dimensions.The authors identify accounting for the spatial dimension as future work.
Abbreviations
The manuscript defines abbreviations for pooling methods, neural-network models, spectral indices, accuracy, remote-sensing data, and classifiers.
- AP, GAP, and MP denote Average Pooling, Global Average Pooling, and Max pooling.
- CNN, RNN, and TempCNN denote Convolutional Neural Network, Recurrent Neural Network, and Temporal Convolutional Neural Network.
- IB, NDVI, and NDWI denote Brilliance Index, Normalized Difference Vegetation Index, and Normalized Difference Water Index.
- OA denotes Overall Accuracy, while RF and SVM denote Random Forests and Support Vector machine.
- SB and SITS denote Spectral Band and Satellite Image Time Series.