Source-linked AI summary
Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders
Marc Rußwurm, Marco Körner
TL;DR
Many LULC methods do not exploit readily available temporal EO data because sequential observations require scalable processing that is robust to noise and often involve cumbersome preprocessing. This paper adapts convolutional recurrent sequence encoders to Sentinel 2 image sequences for crop classification. Using unfiltered TOA reflectance data, it reports state-of-the-art accuracies across many crop classes and indicates that cloud filtering can be learned jointly with classification.
Problem
Multi-temporal EO classification needs methods that can process abundant sequential observations robustly and scalably without relying on cumbersome region-specific preprocessing.
Method
The paper uses convolutional recurrent sequence encoders to aggregate variable-length Sentinel 2 image sequences into one classification decision.
Results
State-of-the-art crop-classification accuracies were achieved with unfiltered, atmospherically uncorrected TOA observations across a large number of crop classes.
Takeaways & Limitations
The results indicate that cloud masking can be learned jointly with classification, reducing the need for explicit cloud-related labels or prior cloud classification.
Takeaways & Limitations
The experimental setup used point-wise classification of neighboring pixels and therefore could not produce areal classification maps.
Abstract
from arXiv · showhide
Earth observation (EO) sensors deliver data with daily or weekly temporal resolution. Most land use and land cover (LULC) approaches, however, expect cloud-free and mono-temporal observations. The increasing temporal capabilities of today's sensors enables the use of temporal, along with spectral and spatial features. Domains, such as speech recognition or neural machine translation, work with inherently temporal data and, today, achieve impressive results using sequential encoder-decoder structures. Inspired by these sequence-to-sequence models, we adapt an encoder structure with convolutional recurrent layers in order to approximate a phenological model for vegetation classes based on a temporal sequence of Sentinel 2 (S2) images. In our experiments, we visualize internal activations over a sequence of cloudy and non-cloudy images and find several recurrent cells, which reduce the input activity for cloudy observations. Hence, we assume that our network has learned cloud-filtering schemes solely from input data, which could alleviate the need for tedious cloud-filtering as a preprocessing step for many EO approaches. Moreover, using unfiltered temporal series of top-of-atmosphere (TOA) reflectance data, we achieved in our experiments state-of-the-art classification accuracies on a large number of crop classes with minimal preprocessing compared to other classification approaches.
1. Introduction
Recent data and machine-learning advances make multi-temporal Earth observation increasingly feasible, while sequence-to-sequence methods provide a model for processing temporal information. This work adapts sequence encoders to EO and applies them to crop classification across two seasons.
- 1. Introduction: Temporal information can characterize LULC classes alongside spectral and spatial features, but many classification tasks have historically focused on only the latter two.Vegetation classes are especially associated with phenological changes in spectral reflectivity.
- 1. Introduction: Technological advances have made high-resolution multi-temporal EO data more available and feasible to store, transmit, and process.Machine learning and GPU hardware also support learning complex relationships directly from observations.
- 1. Introduction: Sequence-to-sequence learning offers a framework for transforming variable-length input sequences into intermediate representations, motivating temporal EO processing.The paper draws this analogy from machine translation, text summarization, and speech recognition.
- 1. Introduction: The work adapts sequence encoders from sequence-to-sequence learning to EO, visualizes recurrent gate activations, and performs crop classification over two seasons.These are identified as the paper’s main contributions.
2. Related Work
Earlier multi-temporal crop-classification approaches commonly relied on preprocessing, engineered features, and region-specific expert knowledge, creating procedural and scalability challenges. The paper situates convolutional recurrent sequence encoders as a data-driven alternative for sequential EO classification.
- 2. Related Work: Traditional crop-classification methods use atmospheric correction, vegetation indices, phenological features, or expert knowledge to improve feature separability.Examples of expert inputs include agro-meteorological data, selected observation dates, and classification rules.
- 2. Related Work: These approaches can achieve good accuracy in respective areas of interest but are difficult to transfer across regions and scale because they require supervised processing decisions.The cited decisions include product selection, visual inspection, and parameter tuning.
- 2. Related Work: Data-driven deep learning is presented as potentially suitable for approximating phenological changes governed by complex biochemical processes observed indirectly through surface reflectivity.The paper contrasts this with manually designing a functional model for the relationship.
- 2. Related Work: Recurrent architectures are described as better suited than convolutional networks for sequential tasks because they iteratively process sequential information.Prior EO applications include change detection, sea-level anomaly identification, and land-cover classification.
- 2. Related Work: The proposed approach builds on prior crop-classification experiments by combining convolutional recurrent layers with a many-to-one sequence-encoder scheme.The method is positioned as an adaptation of sequence modeling for crop classification.
3. Methodology
The paper adapts recurrent sequence encoders to multi-temporal EO imagery, using convolutional recurrent layers and bidirectional processing to encode image sequences into classification maps. This redesign addresses limitations of prior point-wise, mono-directional approaches and supports cloud-unfiltered TOA inputs.
- Sequential encoders: Sequence-to-sequence learning represents an entire variable-length observation series with a final cell state rather than producing predictions at every time step.This many-to-one encoding motivates the proposed temporal aggregation strategy.
- Recurrent architectures: Convolutional recurrent layers process raster observations while preserving neighborhood relationships through convolutional kernels.The convolutional weights operate on image tensors with height, width, and spectral depth.
- Prior work: Prior experiments found recurrent architectures well suited to extracting temporal features from multi-temporal EO imagery, but their setup required cloudy-pixel labels and introduced later-observation bias.These limitations motivated the bidirectional, sequence-level redesign.
- Proposed network: Compared with prior work, the model produces one areal classification map for the full series instead of point-wise predictions at every observation.The redesign also avoids extra post-processing to aggregate time-step labels.
- Proposed network: The proposed sequence-level output treats atmospheric perturbations as temporal noise, enabling evaluation on atmospherically uncorrected TOA reflectance without prior cloud classification.The paper investigates this behavior by visualizing recurrent activations on cloudy and non-cloudy observations.
- Proposed network: The proposed encoder passes each image sequence in both forward and reversed order, concatenates the final states, and projects them to class-specific softmax maps.Shared weights across the two passes reduce bias toward later observations.
4. Dataset
The evaluation uses a large agricultural area north of Munich, partitioned spatially into independent training, validation, and evaluation blocks. Sentinel 2 time series from two growing seasons are paired with field-parcel crop labels aggregated into 17 classes.
- Area of interest: The area of interest covers 102 km × 42 km north of Munich, Germany, and contains 137 k field parcels.It is further tiled into squared blocks of 3.84 km × 3.84 km.
- Partitioning: Spatial blocks were randomly assigned to training, validation, and evaluation partitions in a 4:1:1 ratio.Block-level assignment was used to ensure dataset independence while maintaining similar class distributions.
- Sentinel 2 time series: The time series contains 274 Sentinel 2 products acquired on 98 observation dates between 3 January 2016 and 15 November 2017.The products cover two growing seasons and include images with cloud coverage below 80%.
- Ground truth: Ground truth comes from 137 k field parcels labeled by the Bavarian Ministry of Food, Agriculture and Forestry.The study aggregates approximately 200 reported field labels into 17 crop classes occurring at least 400 times in the AOI.
5. Results
The results examine internal recurrent representations, cloud-related gating, quantitative crop-classification accuracy, and qualitative predictions across two growing seasons.
- 5.1. Internal Network Activations: The cell-state tensor gained progressively detailed structures as observations were encoded, consistent with additional information being written over time.The final cell state cT=36 was used to produce class activations.
- 5.1. Internal Network Activations: A small subset of recurrent cells appeared sensitive to cloud cover, with cell 47 reducing input-gate activity for cloudy observations.The input gate approached zero across whole tiles or cloudy patches at several observation times.
- 5.2. Quantitative Classification Evaluation: Pixel-averaged precision, recall, and F-score ranged from 89.3% to 89.9%, while overall accuracies were 89.7% and 89.5% across the evaluated seasons.Overall kappa was 0.870; class-wise accuracy ranged from 41.5% for peas to 96.8% for maize.
- 5.2. Quantitative Classification Evaluation: Frequent classes were generally classified more confidently, although peas performed relatively well in 2016 and some underrepresented classes also performed well.Asparagus and hop were examples of less frequent classes with good performance.
- 5.2. Quantitative Classification Evaluation: Many confusions varied between seasons, suggesting that external factors such as weather and sun exposure influenced classification accuracy.Peas were classified well in 2016 but poorly in 2017 because of confusions with meadow and potato.
- 5.3. Qualitative Classification Evaluation: The qualitative examples were mostly accurate, but fewer available satellite images coincided with poor classification and lower-amplitude, more blurred activations in one region.The network also correctly resolved narrow wheat and maize fields without smoothing class activations.
6. Discussion
The proposed network is compared with prior multi-temporal crop-classification approaches despite the lack of a common benchmark. It achieves state-of-the-art accuracy with many classes while using uncorrected Sentinel 2 data and comparatively little preprocessing.
- Comparison with prior work: No multi-temporal benchmark dataset is available for equal-footing comparison, so the study surveys prior crop-classification methods in Table 3.The comparison categorizes approaches by methodology and overall accuracy, while recognizing heterogeneity in data sources and evaluation settings.
- Comparison with prior work: Existing multi-temporal EO methods commonly require separate preprocessing, feature extraction, classification, manual supervision, or region-specific expert parameters.These requirements impede applicability at large scales, while multiple satellite images can also increase acquisition costs.
- Comparison with prior work: Direct accuracy comparisons remain difficult because results depend on evaluated sample count, area extent, and number of classified categories.The authors therefore compare in detail with approaches whose reported accuracies are on a similar level.
- Comparison with prior work: The network achieves state-of-the-art classification accuracy with a comparatively large number of classes.The comparison uses Sentinel 2 top-of-atmosphere values without atmospheric correction, additional cloud classes, or further preprocessing.
- Comparison with prior work: A single recurrent encoding layer with convolutional recurrent cells enables processing larger tiles than previous work.One classification decision is derived from the entire observation sequence rather than separate cloud-class handling.
7. Conclusion
The work presents an automated end-to-end approach for multi-temporal crop classification using uncorrected, cloudy Sentinel 2 sequences. Its results support learned cloud masking and crop-specific phenological modeling, while some crops remain inconsistently classified across seasons.
- Conclusion: The end-to-end approach achieves state-of-the-art accuracies for crop classification with a large number of crop classes.The reported accuracies use atmospherically uncorrected data containing clouds and omit radiometric and geometric preprocessing.
- Conclusion: Internal gate activations provide evidence that some recurrent cells are sensitive to cloud coverage and may internally mask cloudy pixels.The authors frame this as cloud masking learned jointly with classification, analogous to external cloud filtering.
- Conclusion: Quantitative and qualitative evaluations support the interpretation that the network approximates a discriminative crop-specific phenological model from raw Sentinel 2 observations.The encoded model may also implicitly incorporate seasonally variable environmental conditions.
- Conclusion: Some crops were inconsistently classified in both growing seasons, possibly because environmental conditions vary seasonally.This is presented as a possible explanation rather than a definitive causal finding.
- Conclusion: The architecture is not methodologically limited to vegetation modeling and may support tasks that benefit from temporal-feature extraction.The network implementation and additional evaluation materials were made publicly available.