Source-linked AI summary
Improvement in Land Cover and Crop Classification based on Temporal Features Learning from Sentinel-2 Data Using Recurrent-Convolutional Neural Network (R-CNN)
Vittorio Mazzia, Aleem Khaliq, Marcello Chiaberge
TL;DR
Accurate crop maps support agricultural management, but pixel-based classification of broad multispectral time series can be demanding for conventional methods. The paper combines recurrent and convolutional networks to classify multi-temporal Sentinel-2 imagery, reporting 96.5% overall accuracy across 15 classes and outperforming the tested non-deep-learning classifiers.
Problem
Accurate crop maps support agricultural decision-making, while SVM classification of broad multispectral time series can be resource-consuming.
Source: §1 · p. 2, §2.2 · p. 3Method
Pixel R-CNN combines an RNN to learn temporal correlations from multi-temporal Sentinel-2 data with a CNN to analyze crop patterns.
Source: §1 · p. 2, §4.1 · p. 6Results
96.5% overall accuracy and kappa=0.914 were reported for 15 classes, and Pixel R-CNN outperformed the tested non-deep-learning classifiers.
Source: §6 · p. 16, §5.5 · p. 16Takeaways & Limitations
The architecture automates feature extraction by learning time correlations, reducing manual feature engineering and modeling of crop phenological stages.
Source: §6 · p. 17Direct quantitative comparison with other studies is difficult because a multi-temporal benchmark dataset with ground truth across the considered land-cover types was unavailable.
Source: §5.4 · p. 15
Abstract
from arXiv · showhide
The increasing spatial and temporal resolution of globally available satellite images, such as provided by Sentinel-2, creates new possibilities for researchers to use freely available multi-spectral optical images, with decametric spatial resolution and more frequent revisits for remote sensing applications such as land cover and crop classification (LC&CC), agricultural monitoring and management, environment monitoring. Existing solutions dedicated to cropland mapping can be categorized based on per-pixel based and object-based. However, it is still challenging when more classes of agricultural crops are considered at a massive scale. In this paper, a novel and optimal deep learning model for pixel-based LC&CC is developed and implemented based on Recurrent Neural Networks (RNN) in combination with Convolutional Neural Networks (CNN) using multi-temporal sentinel-2 imagery of central north part of Italy, which has diverse agricultural system dominated by economic crop types. The proposed methodology is capable of automated feature extraction by learning time correlation of multiple images, which reduces manual feature engineering and modeling crop phenological stages. Fifteen classes, including major agricultural crops, were considered in this study. We also tested other widely used traditional machine learning algorithms for comparison such as support vector machine SVM, random forest (RF), Kernal SVM, and gradient boosting machine, also called XGBoost. The overall accuracy achieved by our proposed Pixel R-CNN was 96.5%, which showed considerable improvements in comparison with existing mainstream methods. This study showed that Pixel R-CNN based model offers a highly accurate way to assess and employ time-series data for multi-temporal classification tasks.
1 Introduction
Land-cover and crop maps support agricultural management and policy decisions, and freely available Sentinel-2 data offers 10 m imagery with frequent revisits. The paper proposes a pixel-based RNN–CNN approach to learn temporal and crop-pattern features for classification.
1 Introduction
- Accurate crop maps and spatial crop distributions support agricultural decision-making, resource management, and food-security policy.The introduction links these maps to the needs of governments, policymakers, and farmers.Source: §1 · p. 2
- Sentinel-2 offers freely available multispectral imagery at up to 10 m resolution with a five-day revisit time using twin satellites.The paper identifies these characteristics as an opportunity for remote-sensing applications.Source: §1 · p. 2
- Multi-temporal imagery can outperform mono-temporal mapping, but temporal-feature engineering and derived indices can add complexity and data volume.The introduction also notes that imagery from key phenological stages may suffice for crop-area estimation.Source: §1 · p. 2
- Land-cover and crop classification includes both pixel-based and object-based approaches, with object-based methods segmenting imagery into homogeneous regions.The introduction identifies object-based mapping as an existing approach to producing crop maps.Source: §1 · p. 2
- The paper proposes an RNN–CNN architecture for pixel-based land-cover and crop classification, with the RNN learning temporal correlations and the CNN analyzing crop patterns.The authors present it as a way to automate feature extraction from multi-temporal Sentinel-2 imagery.Source: §1 · p. 2
2 Related work
Related work describes the challenges of representing crop temporal patterns and reviews pixel-based classification alongside recurrent and convolutional neural networks. It motivates learning representations from sequential and image data rather than relying only on hand-designed features.
2.1 Temporal feature representation
- Temporal-feature methods range from vegetation-index metrics and curve fitting to Markov models, but often require manual design amid variability and uncertain conditions.The review argues for an approach that uses sequential information to extract temporal patterns.Source: §2.1 · p. 3
2.2 Pixel based crops classification
- For pixel-based mapping, reviewed SVM classifiers achieved about 75% overall accuracy, while neural-network classifiers achieved almost 74%; SVMs can be resource-consuming on broad multispectral time series.The review also notes that random forests require derived features for effective use.Source: §2.2 · p. 3
2.3 Recurrent neural network (RNN)
- RNNs are used for sequence learning, while LSTM and GRU architectures address difficulties training over many time steps.Remote-sensing examples apply RNNs to spectral correlation and LSTMs to combined spectral-temporal representations.Source: §2.3 · p. 3
2.4 Convolutional neural network (CNN)
- CNNs support image classification, and remote-sensing studies use 2D convolutions to extract spatial features and multiple convolutional layers to learn spatial and spectral features.The review reports that 2D spatial convolution performed better than 1D spectral-domain convolution in crop classification.Source: §2.4 · pp. 3–4
3 Study area and data
The study uses Sentinel-2 imagery and LUCAS-derived reference data for a diverse agricultural area near Carpi in Emilia-Romagna, Italy. The selected area covers about 2,640 km² and includes major crop fields such as maize, lucerne, barley, wheat, and vineyards.
3 Study area and data
- The study area near Carpi covers about 2,640 km² and includes major fields of maize, lucerne, barley, wheat, and vineyards.Most farmers in the area practice single cropping.Source: §3 · p. 4
- LUCAS-2015 microdata and technical documentation supplied crop and land-cover reference information, with pixels selected manually by overlaying imagery and survey data.The paper used QGIS to import and inspect the spatial microdata.Source: §3 · p. 4, Figure 2 · p. 6
- The experiments used ten co-registered Sentinel-2 images from July 2015 to July 2016, selecting 10 m bands and derived NDVI after cloud screening and visual inspection.The scenes were corrected to Bottom of Atmosphere Level 2A imagery.Source: §3 · p. 4, Table 1 · p. 4, Table 2 · p. 5
- Figure 1 locates the Carpi study site and marks LUCAS-2015 ground-truth locations on an August 2015 Sentinel-2 RGB composite.The figure caption identifies the coordinates as WGS84.Source: Figure 1 · p. 5
4 Convolutional and recurrent neural networks for pixel-based crops classification
Pixel R-CNN maps each multi-temporal, multispectral pixel to class probabilities through recurrent temporal encoding, hierarchical convolutional feature extraction, and a final classifier. Its training uses supervised cross-entropy, adaptive learning-rate procedures, and dropout.
4.1 Formulation
- Pixel R-CNN maps multi-temporal, multispectral pixels through recurrent encoding, hierarchical convolutional feature extraction, and a K-class probability output.Source: §4.1 · p. 6
4.1.1 Time correlation representation
- Peephole LSTM cells use gated long- and short-term states to encode temporal sequences, then TimeDistributedDense preserves their multidimensional structure.Source: §4.1.1 · pp. 8–9
4.1.2 Temporal patterns extraction
- After reshaping the sequence representation, two shared-filter convolutional layers progressively reduce dimensions and hierarchically extract abstract temporal patterns.Source: §4.1.2 · pp. 9–10
4.1.3 Multiclass classification
- Flattened convolutional features are mapped through learned class scores and softmax to probabilities for the K classes.Source: §4.1.3 · p. 10
4.2 Training
- Supervised categorical cross-entropy trains the mapping; AMSGrad, cosine-annealed learning rates, and dropout between LSTM and TimeDistributedDense are also used.Source: §4.2 · p. 11
5 Experimental Results and discussion
The experiments prepare labeled pixel time series, inspect their structure with PCA, and evaluate the Pixel R-CNN architecture on land-cover and crop classification. They also compare it with conventional classifiers and examine class-level outcomes and internal activations.
5.1 Training data
- The 92,116 pixel samples were arranged as 9 time steps by 5 spectral bands, split 60%/40% with stratified sampling, then standardized.The tensor dimensions were (92116, 9, 5); the training and test tensors had shapes (55270, 9, 5) and (36846, 9, 5).Source: §5.1 · p. 11, Figure 8 · p. 12
5.2 Dataset Visualization
- The first three PCA components preserve 64.5% of dataset variance; most classes show substantial intra-class variation, while explained-variance growth slows at higher dimensions.The authors suggest reducing time steps would not significantly affect overall results.Source: §5.2 · p. 12, Figure 9 · p. 13
5.3 Experimental settings
- The 30,936-parameter model used a 32-unit peephole LSTM, associated with about 0.8% higher overall accuracy than standard LSTM, followed by two CNN layers and a 15-class SoftMax.The TimeDistributed Dense block reshapes the LSTM output into a 9x9 matrix for the CNN stack.Source: §5.3 · p. 13
5.4 Classification
- OA=96.5% and Kappa=0.914 across 15 classes; Figure 11 shows the final classified map, a zoomed region, and its Sentinel-2 RGB composite.The authors report significant improvement over mainstream methods.Source: §5.4 · p. 14, Figure 11 · p. 14
- Grassland was weakest at PA = 65% and UA = 63%; Apple had PA 86% and UA = 68%, with Peer and Vineyard pixels sometimes classified as Apple.The major grassland confusions were Lucerne and Vineyard.Source: §5.4 · pp. 14–15
- Direct comparison with prior studies is constrained because a multi-temporal benchmark dataset is unavailable and studies differ in sample counts, study extents, and evaluated class counts.The authors therefore describe an overview of recent studies rather than an equal-footing quantitative comparison.Source: §5.4 · p. 15
- For four different artificial-class samples, activations at the TimeDistributedDense output are already similar despite visibly different inputs.The figure visualizes neuron activations, with darker values close to zero.Source: Figure 12 · p. 16, §5.4 · p. 16
5.5 Non deep learning classifiers
- OA=79.6% for SVM, 77.5% for RF, 76.8% for Kernel SVM, and 77.2% for XGBoost, versus Pixel R-CNN's OA=96.5%.The authors note that learning temporal and spectral correlations over large datasets is challenging for traditional methods.Source: §5.5 · p. 16
6 Conclusion
The conclusion presents Pixel R-CNN, a recurrent-convolutional model using multitemporal Sentinel-2 imagery for land-cover and crop classification. It reports 96.5% overall accuracy and kappa=0.914 across 15 classes, alongside automated temporal feature extraction.
6 Conclusion
- 96.5% overall accuracy and kappa=0.914 were achieved for 15 classes, with significant improvement over mainstream methods.Table 6 compares Pixel R-CNN with non-deep-learning classifiers.Source: §6 · p. 16, Table 6 · p. 17
- SVM, RF, kernel SVM, and XGBoost were less effective, especially when temporal-feature extraction was key to classification accuracy.Source: §6 · p. 17
- Pixel R-CNN automates feature extraction by learning temporal correlations across images, reducing manual feature engineering and modeling of crop phenological stages.Source: §6 · p. 17