Source-linked AI summary
Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention
Vivien Sainte Fare Garnot, Loic Landrieu, Sebastien Giordano, Nesrine Chehata
TL;DR
Large-scale crop classification needs efficient use of satellite image time series, while existing convolutional-recurrent approaches are poorly matched to coarse-resolution parcels and costly at continental scale. The paper uses unordered pixel-set encoders for spatial representation and adapted self-attention for temporal encoding. It reports better classification than re-implemented alternatives, processing over 4 times faster, memory reduced by over 70%, and releases an open-access annotated benchmark.
Problem
Automated crop-type classification matters for large-scale agricultural monitoring, but hybrid convolutional-recurrent architectures do not adapt well to coarse-resolution parcels and large variable satellite time series.
Method
The paper represents parcel images as unordered pixel sets and adapts the Transformer architecture with self-attention for end-to-end time-sequence embedding.
Results
The method achieves better classification than other re-implemented methods, with acceleration over 4 times and memory imprint reduced by over 70% compared with best-performing literature approaches.
Takeaways & Limitations
Pixel-set spatial encoding and self-attention provide a more computationally efficient approach to object-based satellite time-series classification, supported by a new annotated benchmark.
Abstract
from arXiv · showhide
Satellite image time series, bolstered by their growing availability, are at the forefront of an extensive effort towards automated Earth monitoring by international institutions. In particular, large-scale control of agricultural parcels is an issue of major political and economic importance. In this regard, hybrid convolutional-recurrent neural architectures have shown promising results for the automated classification of satellite image time series.We propose an alternative approach in which the convolutional layers are advantageously replaced with encoders operating on unordered sets of pixels to exploit the typically coarse resolution of publicly available satellite images. We also propose to extract temporal features using a bespoke neural architecture based on self-attention instead of recurrent networks. We demonstrate experimentally that our method not only outperforms previous state-of-the-art approaches in terms of precision, but also significantly decreases processing time and memory requirements. Lastly, we release a large open-access annotated dataset as a benchmark for future work on satellite image time series.
1 Introduction
The paper targets automated agricultural parcel monitoring by replacing convolutional-recurrent designs with pixel-set spatial encoders and self-attention temporal encoding. It reports higher precision, faster processing, lower memory use, and releases an annotated benchmark dataset.
- Motivation: High-quality satellite data and abundant agricultural labels motivate automated crop-type classification for large-scale policy monitoring.The EU needs yearly crop maps for Common Agricultural Policy subsidies, while Sentinel-2 and farmer declarations provide suitable data sources.
- Problem: Hybrid convolutional-recurrent architectures are identified as poorly adapted to key characteristics of satellite parcel classification.The paper focuses on coarse spatial resolution and the resulting mismatch with convolution-based texture extraction.
- Spatial Encoding of Parcels: Pixel-set encoders represent medium-resolution parcels as unordered pixel sets, providing an alternative to convolutional spatial encoding.The approach is motivated by the limited textural information captured at 10m-per-pixel resolution and by variable parcel sizes.
- Temporal Encoding of Satellite Time Series: An adapted self-attention architecture encodes satellite time series while using acquisition dates to address inconsistent temporal sampling.The approach is designed for large-scale monitoring, where parallelism is valuable and one year of EU agricultural observations amounts to 25Tb.
- Results: The method improves classification precision, accelerates processing by over 4 times, and reduces memory imprint by over 70% versus the best-performing literature approaches.These results establish a new state-of-the-art for large-scale agricultural parcel classification.
- Dataset: The paper releases the first open-access Sentinel-2 crop-classification dataset with ground-truth labels.The dataset is intended to support benchmarking of satellite image time-series classification.
2 Related Work
Related work spans handcrafted-feature models, deep convolutional and recurrent architectures, attention-based models, and temporal convolutions. The paper positions its approach against limitations involving variable sampling, sequence length, spatial resolution, and computational cost.
- Problem Formulation: Object-based approaches can exploit parcel-level spatial homogeneity and shape, whereas pixel-based approaches do not require parcel borders beforehand.This distinction frames the paper’s focus on object-level agricultural parcel classification.
- Traditional Machine Learning: Traditional crop classification commonly uses Random Forest or Support Vector Machine models with handcrafted spectral and phenological features.NDVI and meteorological information are examples of engineered inputs used in earlier work.
- Traditional Machine Learning: Concatenating spatial and spectral features across dates is poorly suited to geographically broad data with variable acquisition times, cloud cover, and sequence lengths.These conditions motivate methods that model temporal evolution more explicitly.
- Convolutional and Recurrent Approaches: Hybrid convolutional-recurrent models became state-of-the-art by jointly learning spatial and temporal structures in satellite image series.ConvLSTM-based parcel classification also showed the ability to detect and ignore cloud-obstructed observations.
- Attention-Based Approach: Transformer self-attention has been applied to pixel-based classification with performance comparable to recurrent models and similar robustness to cloud-obstructed observations.The paper extends this direction to end-to-end sequence-to-embedding learning for object-level classification.
- Purely Convolutional Approach: Temporal convolutions can process some classification tasks faster than recurrent networks, but deeper architectures may be needed for long-term dependencies.Fixed temporal-CNN architectures also cannot directly accommodate sequences with different lengths or acquisition patterns.
3 Methods
The architecture replaces convolutional spatial processing with a Pixel-Set Encoder and replaces recurrent temporal processing with a self-attention-based Temporal Attention Encoder. These components produce a single spatio-temporal embedding for classification.
- 3.1 Spatial Encoder: The Pixel-Set Encoder represents each parcel observation as an unordered set of sampled pixels rather than relying on CNN texture features.It computes learned statistical descriptors of the parcel’s spectral distribution and is inspired by PointNet and Deep Sets.
- 3.1 Spatial Encoder: Each sampled pixel passes through a shared MLP, after which pooling produces permutation-invariant parcel statistics combined with geometric features.The geometric features include perimeter, pixel count, cover ratio, and perimeter-to-surface ratio.
- 3.1 Spatial Encoder: Concatenating the pooled mean and standard deviation across sampled pixels gave the best pooling results.Repeated pixels should be removed before pooling for parcels smaller than the sampling size to obtain unbiased estimates.
- 3.2 Temporal Attention Encoder: The temporal encoder adapts Transformer self-attention to satellite image time series, enabling parallel processing instead of recurrent sequential computation.The design uses query-key compatibility and multiple attention heads to process temporal embeddings.
- 3.2 Temporal Attention Encoder: Positional encodings use the number of days since the first observation rather than the observation index to account for inconsistent temporal sampling.The temporal embeddings are formed from spatial embeddings with positional encodings, while key-query pairs are computed for attention.
- 3.2 Temporal Attention Encoder: A single master query per attention head selects useful dates, and concatenated head outputs are processed by MLP3 into one temporal embedding.The classifier embeds all observations in parallel with a shared PSE, processes the resulting sequence with the temporal encoder, and applies MLP4 to produce class logits.
4 Numerical Experiments
The experiments evaluate pixel-set and temporal-attention design choices for agricultural parcel classification, comparing them with re-implemented alternatives under matched conditions. The proposed architecture improves classification while reducing processing time and memory requirements.
- Dataset: The evaluation uses Sentinel-2 sequences with 10 spectral bands, covering 191,703 parcels observed on 24 dates in southern France.The area of interest spans 12,100 km2 and contains diverse crop types and terrain conditions.
- Data Preparation: Pixel-set data preserve parcel information without resizing, save up to 70% disk space versus patches, and require geometric features to be precomputed.Pixels are stored in arbitrary order as tensors of size T × C × N, where N varies by parcel.
- Main Results: The proposed architecture outperforms other deep-learning models by 0.4 OA points and 3 to 9 mIoU points, while providing a four-fold speed-up over convolution-based methods.Training disk usage decreases by over 70%, and inference-only disk usage decreases by close to 90%.
- Ablation Studies: The PSE increases mIoU by 1.7 points relative to CNN+TAE and reaches 48.9 mIoU when replaced by precomputed means and standard deviations.The authors interpret this as evidence that PSE-learned spectral descriptors are more meaningful than simple statistics or convolutional features.
- Ablation Studies: Removing geometric features decreases mIoU by 0.9 points, while replacing the TAE with a GRU decreases mIoU by 3.6 points.The TAE also trains faster through parallelization, and adding positional encoding to the GRU reduces the performance gap to 2.2 mIoU points.
- Ablation Studies: Replacing handcrafted geometric features with a CNN over parcel masks provides minimal gains, so the additional step is removed for simplicity.
Conclusion
The paper frames object-based satellite time-series classification through unordered pixel sets and Transformer-based temporal encoding. On a new annotated agricultural-parcel benchmark, the method achieves better classification with lower processing and memory requirements.
- The method represents medium-resolution satellite images as unordered pixel sets and uses a fitting encoder for object-based classification.
- Temporal sequences are embedded with an adapted Transformer architecture using a master-query forming strategy and end-to-end learning.
- On the new open-access annotated benchmark, the method produces better classification than all other re-implemented methods.
- The network is several times faster and more memory-efficient than other state-of-the-art methods, including convolutional-recurrent hybrids.
- The results suggest attention-based temporal models and set-based encoders are promising directions for satellite time series and analogous vision tasks.
Supplementary Material
Supplementary materials document model configurations, processing-time measurements, and qualitative prediction analyses. The figures and tables provide implementation and error-analysis context for the main experiments.
- Model Configurations: The supplementary hyperparameter table lists configurations for competing architectures, with a shared decoder specification across models.
- Processing Time: The processing-time table compares methods on batches of 128 parcels and separates training and inference-related timing measures.Its caption distinguishes disk usage, full training time, preprocessing plus inference time, and dataset sizes before and after preprocessing.
- Processing Time: Processing time is dominated by loading for most methods, except the Transformer, which processes precomputed means.
- Qualitative Evaluation: The confusion matrix reports parcel counts relative to each class population, while the prediction visualization marks correct parcels green and false predictions red.
- Qualitative Evaluation: Many errors are assigned to Meadows, the most represented class, and Winter Durum Wheat is difficult to distinguish from Winter Cereal because of similar phenology.