Source-linked AI summary
Land Cover Classification via Multi-temporal Spatial Data by Recurrent Neural Networks
Dino Ienco, Raffaele Gaetano, Claire Dupaquier, Pierre Maurel
TL;DR
Efficient analysis of satellite image time series remains an open remote-sensing problem, with limited use of temporal deep learning for land-cover classification. The paper evaluates LSTM-based recurrent models across pixel- and object-based datasets and tests their learned representations with standard classifiers. The results show competitive or superior performance, especially for weak or mixed classes, while LSTM features improve standard classifiers.
Problem
Efficiently analyzing satellite image time series for land-cover classification remains an open challenge, and few studies use recurrent temporal deep-learning approaches for remote-sensing time series.
Method
The study evaluates LSTM-based recurrent classification on two SITS datasets covering object- and pixel-based prediction, and uses the LSTM representation as input to standard classifiers.
Results
LSTM models are competitive with or outperform classical approaches, particularly for low-represented or highly mixed classes, while LSTM-derived features improve standard classifiers.
Takeaways & Limitations
LSTM-based classification can support both pixel- and object-based land-cover analysis and can provide useful learned representations for standard classifiers.
Abstract
from arXiv · showhide
Nowadays, modern earth observation programs produce huge volumes of satellite images time series (SITS) that can be useful to monitor geographical areas through time. How to efficiently analyze such kind of information is still an open question in the remote sensing field. Recently, deep learning methods proved suitable to deal with remote sensing data mainly for scene classification (i.e. Convolutional Neural Networks - CNNs - on single images) while only very few studies exist involving temporal deep learning approaches (i.e Recurrent Neural Networks - RNNs) to deal with remote sensing time series. In this letter we evaluate the ability of Recurrent Neural Networks, in particular the Long-Short Term Memory (LSTM) model, to perform land cover classification considering multi-temporal spatial data derived from a time series of satellite images. We carried out experiments on two different datasets considering both pixel-based and object-based classification. The obtained results show that Recurrent Neural Networks are competitive compared to state-of-the-art classifiers, and may outperform classical approaches in presence of low represented and/or highly mixed classes. We also show that using the alternative feature representation generated by LSTM can improve the performances of standard classifiers.
I. INTRODUCTION
The paper addresses the challenge of analyzing satellite image time series for land cover classification by evaluating LSTM-based recurrent models on multi-temporal spatial data. It studies both pixel- and object-based settings and also tests LSTM-learned representations with standard classifiers.
- Analyzing satellite image time series efficiently remains an open challenge in remote sensing.
- Temporal profiles from satellite image time series can help distinguish land-cover classes, but standard approaches often treat stacked-image features independently.
- The study evaluates LSTM recurrent models for land-cover classification using multi-temporal spatial data from satellite image time series.
- Experiments cover object-oriented classification on the three-date THAU series and pixel-based prediction on the 23-date REUNION ISLAND series.
- The study also evaluates the last LSTM-derived representation as input to standard classifiers.
A. Long-Short Term Memory
RNNs model temporal dependencies by feeding previous outputs into the next time step, while LSTM units use memory states and gates to learn long-term dependencies. The paper presents their structure through unfolded recurrent connections and information-flow mechanisms.
- RNNs explicitly manage temporal dependencies by combining the previous output with the next input at each time step.
- Figure 1 depicts a recurrent unit alongside its unfolded structure across time.
- Figure 2 represents directed connections, information-flow paths, and operations that combine or transform information within the LSTM unit.
- LSTM is an RNN architecture designed to learn long-term dependencies that earlier RNNs struggled to capture because of vanishing and exploding gradients.
- An LSTM processes a variable-length sequence of feature vectors indexed by timestamps.
- The LSTM unit combines memory and hidden states with input, forget, and output gates to regulate information flow across timestamps.
B. LSTM-Based Time Series Classification
The classification architecture places a SoftMax layer on top of the LSTM representation to produce multi-class predictions. SoftMax outputs a normalized probability distribution across the target classes.
- A SoftMax layer is stacked on the LSTM unit to perform the final multi-class prediction.
- The SoftMax layer has one neuron per target class and produces values that form a probability distribution summing to 1.
C. Representation Learning with LSTM for time series data
The paper evaluates LSTM as a feature extractor by using its final hidden-state vector as a new representation for standard classifiers. This tests whether learned temporal features improve classification beyond direct LSTM prediction.
- The last hidden-state vector produced by the LSTM is used as a new data representation for standard machine-learning classifiers.
- The learned representation is assessed by training standard classifiers on the extracted LSTM features.
III. DATA
The study evaluates the proposal on two remote-sensing datasets, including an object-based THAU dataset derived from multi-temporal very-high-resolution imagery. THAU objects are segmented, statistically represented, visually labeled, and organized into land-cover classes.
- Dataset scope: The experiments use two remote-sensing datasets to assess the proposal's generality.One dataset contains spatial objects with limited temporal depth; the other is pixel-based and richer in spectral and temporal resolution.
- THAU imagery: The THAU dataset uses Pléiades very-high-resolution imagery at 2 m resolution from July and September 2012 and March 2013.The study site is the THAU Basin in southern France and covers 42,000 ha, with 70% land area.
- Object representation: Multi-temporal segmentation produces a consistent object layer, with each object represented by per-date means and standard deviations from four bands and NDVI.The features use blue, green, red, near-infrared, and NDVI information.
- Labels and classes: Visual inspection retains 15,196 THAU objects labeled into 11 land-cover classes.The class cardinalities are reported in Table I.
- Dataset summary: Table I reports the land-cover classes and their cardinalities for the THAU time-series dataset.The table organizes the retained objects by class and relative class size.
B. REUNION ISLAND dataset
The REUNION ISLAND dataset represents each pixel through a multi-temporal feature stack, with dataset information reported in Table II.
- Pixel representation: Each pixel at each timestamp is represented by reflectances plus three indices.The representation combines spectral measurements with derived indices.
- Dataset summary: The REUNION ISLAND dataset is summarized in Table II.The supplied passage identifies the table as the dataset report.
- Dataset identity: Table II is labeled REUNION ISLAND.The caption identifies the dataset associated with the table.
IV. EXPERIMENTAL RESULTS
The experimental section reports settings and discusses results for the two SITS datasets introduced earlier.
- Experimental scope: The section reports the experimental settings used in the study.It introduces the procedures used to evaluate the presented methods.
- Results scope: The section discusses results obtained on two SITS datasets.The datasets were presented previously in Section III.
- Dataset linkage: The results discussion concerns the SITS datasets described earlier in the paper.The section connects the evaluation to the data introduced in Section III.
A. Experimental Settings
The study compares an LSTM-based time-series classifier with RF and SVM, and also evaluates LSTM-learned representations for standard classifiers. Performance is assessed with cross-validation and multiple metrics suited to imbalanced datasets.
- Compared methods: The LSTM-based Time Series Classification model is compared with Random Forest and Support Vector Machine approaches.The comparison uses standard classifiers commonly applied to multi-temporal spatial data.
- Representation evaluation: The study evaluates whether the representation learned by the LSTM improves standard-classifier performance.RF and SVM are trained using both original data and the LSTM-learned representation.
- Classifier settings: Random Forest uses 400 trees with maximum depth 10, while SVM uses an RBF kernel with complexity 100 and gamma 0.01.These settings are applied to both original data and the LSTM-based representation.
- Data preprocessing: The source imagery is provided by THEIA and preprocessed with CNES's MACCS level 2A processor.The processor supplies atmospheric, environmental, geometric, and cloud-mask corrections.
- Validation and metrics: Five-fold cross-validation evaluates Global Accuracy, Kappa, average F-Measure, and per-class F-Measure.Multiple measures are used because both time-series datasets are unbalanced.
B. Results and Discussions
Across the THAU and REUNION ISLAND datasets, LSTM-based classification performs competitively with or better than standard methods, especially for underrepresented and difficult classes, while learned representations improve standard classifiers. The proposal uses one LSTM layer, leaving deeper architectures for future work.
- THAU dataset: 74.63% average F-Measure: the LSTM-based classifier outperforms RF and SVM on all three reported THAU metrics.The second-best method, SVM(LSTM), reaches 73.31% average F-Measure.
- THAU dataset: LSTM clearly outperforms standard approaches on THAU classes with few reference samples while remaining similar or slightly better on well-represented classes.Standard approaches sometimes completely miss classes (1) and (4); learned representations also improve RF and SVM on critical classes.
- REUNION ISLAND dataset: On REUNION ISLAND, the LSTM-based classifier performs better than standard methods for Accuracy and F-Measure, while SVM(LSTM) achieves the highest F-Measure.Classifiers trained on LSTM-learned features outperform counterparts trained on the original feature space.
- Interpretation: LSTM methods are particularly effective for low-represented and highly mixed agricultural classes whose temporal patterns vary across practices.These classes may be detectable only in short time-series portions where crop conditions are comparable.
- Scope: The experiments use one LSTM layer; stacking additional layers is left for future research.The stated objective is to assess the suitability of RNN methods for SITS data.
V. CONCLUSION
The framework applies LSTM-based recurrent modeling to both object-based and pixel-based land-cover classification. It remains competitive with classical approaches, improves predictions for weak classes, and can provide learned representations for standard classifiers.
- Conclusion: The framework efficiently handles both pixel-based and object-based classification across two SITS datasets.The conclusion reports validation on two different datasets.
- Conclusion: The framework is competitive with classical approaches and improves prediction quality for weak classes in unbalanced datasets.The conclusion characterizes this improvement as a key advantage.
- Conclusion: The LSTM-based model can act as a feature extractor whose learned representation improves standard classifiers on SITS data.This extends the model’s use beyond direct LSTM-based classification.