Source-linked AI summary
Crop mapping from image time series: deep learning with multi-scale label hierarchies
Mehmet Ozgur Turkoglu, Stefano D'Aronco, Gregor Perich, Frank Liebisch, Constantin Streit, Konrad Schindler, Jan Dirk Wegner
TL;DR
Crop mapping must handle rare classes in realistic satellite-image time series. The paper encodes an expert label hierarchy in a convolutional recurrent network, evaluates it on ZueriCrop, and reports superior performance with improved rare-class classification.
Problem
Existing crop-classification datasets often contain few, relatively balanced classes, while realistic agricultural data include rare classes that make supervised training challenging.
Method
The method integrates a domain-expert label hierarchy into an end-to-end multi-temporal convolutional recurrent network that predicts labels at multiple granularities.
Results
The hierarchical approach outperforms competing methods on ZueriCrop and greatly improves classification of many rare classes.
Takeaways & Limitations
Coarser predictions can provide useful outputs when fine-grained predictions are uncertain, while hierarchical representations improve rare-crop mapping.
Takeaways & Limitations
Exceedingly rare crops remain confused with more frequent classes, and 10-meter Sentinel-2 pixels can affect predictions at small-field boundaries.
Abstract
from arXiv · showhide
The aim of this paper is to map agricultural crops by classifying satellite image time series. Domain experts in agriculture work with crop type labels that are organised in a hierarchical tree structure, where coarse classes (like orchards) are subdivided into finer ones (like apples, pears, vines, etc.). We develop a crop classification method that exploits this expert knowledge and significantly improves the mapping of rare crop types. The three-level label hierarchy is encoded in a convolutional, recurrent neural network (convRNN), such that for each pixel the model predicts three labels at different level of granularity. This end-to-end trainable, hierarchical network architecture allows the model to learn joint feature representations of rare classes (e.g., apples, pears) at a coarser level (e.g., orchard), thereby boosting classification performance at the fine-grained level. Additionally, labelling at different granularity also makes it possible to adjust the output according to the classification scores; as coarser labels with high confidence are sometimes more useful for agricultural practice than fine-grained but very uncertain labels. We validate the proposed method on a new, large dataset that we make public. ZueriCrop covers an area of 50 km x 48 km in the Swiss cantons of Zurich and Thurgau with a total of 116'000 individual fields spanning 48 crop classes, and 28,000 (multi-temporal) image patches from Sentinel-2. We compare our proposed hierarchical convRNN model with several baselines, including methods designed for imbalanced class distributions. The hierarchical approach performs superior by at least 9.9 percentage points in F1-score.
1. Introduction
The paper addresses crop mapping under realistic, imbalanced agricultural data by combining expert-defined label hierarchies with an end-to-end multi-temporal deep-learning architecture. It introduces ZueriCrop and reports improved classification, particularly for rare crops.
- Agricultural monitoring supports sustainable management, biodiversity, ecosystem services, and policy programs addressing intensive agriculture.
- Existing multi-temporal crop-mapping datasets typically contain few, relatively balanced classes, whereas rare classes make data-driven training difficult.
- Hierarchical labels pool shared information among related rare crops, enabling coarser features to be learned from larger groups while fine-grained classifiers distinguish fewer subclasses.
- The proposed architecture is hierarchical, convolutional, and recurrent, using stacked convolutional recurrent networks to predict successively finer labels and a CNN refinement module to exploit cross-level label correlations.
- The method encodes a domain-specific hierarchy in an end-to-end trainable model and significantly improves classification accuracy, including for underrepresented crop types.
- ZueriCrop contains 28,000 Sentinel-2 image patches observed 71 times over 52 weeks, covering 48 classes and 116,000 agricultural fields across 50 km × 48 km.
2. Related Work
Related work covers deep learning for multi-temporal satellite imagery, established strategies for class imbalance, and earlier hierarchical remote-sensing methods. The paper positions its integrated convRNN and realistic, imbalanced dataset as distinct from prior approaches.
- Deep-learning methods model spectral, spatial, and temporal patterns without hand-engineered features, using RNNs, convolutional recurrent networks, CNN-RNN pipelines, temporal CNNs, and attention.
- Long-tailed datasets cause standard training to focus on dominant classes, while global metrics can obscure poor performance on rare classes.
- Class-imbalance methods reweight or resample data, but reweighting can bias models toward rare classes, undersampling can omit distributional information, and oversampling fails when minority data are scarce.
- Earlier hierarchical remote-sensing systems used handcrafted features and scene-specific decision trees, with multi-stage workflows requiring computation and manual tuning.
- The paper presents its method as the first deep-learning crop-classification approach to explicitly encode the domain-expert label hierarchy for satellite image sequences.
- Its integrated convolutional recurrent model captures spatio-temporal correlations and is evaluated with a dataset having more classes and a less balanced distribution than earlier datasets.
3. Method
The method encodes an expert-defined crop hierarchy in a multi-stage convolutional recurrent network that predicts increasingly fine labels from image time series. A refinement CNN then combines multi-level predictions to improve coherence and produce the final fine-grained output.
- Hierarchical formulation: The model predicts crop labels at multiple hierarchy levels from a sequence of multi-spectral images.The input is a temporal image sequence, and each pixel receives labels at different granularities.
- Convolutional recurrent representation: Convolutional recurrent layers replace recurrent matrix operations with convolutions to represent spatio-temporal satellite data.The deepest hidden state summarizes the input sequence and can be passed to a decoder for prediction.
- Multi-stage hierarchy: The multi-stage architecture feeds each hierarchy level’s hidden states into the next stage, yielding successively finer label scores.Each stage contains a two-layer convSTAR architecture followed by a CNN classifier.
- Hierarchical supervision: Losses from all granularity levels impose the label hierarchy as a soft prior on feature encoding.This encourages related fine classes, such as apple and pear orchards, to support their shared coarse orchard label.
- Label refinement: A label refinement CNN concatenates probability volumes from the hierarchy stages and produces a final prediction at the finest granularity.The refinement module models interactions among stage predictions; empirically, refinement was used only for the finest level.
4. Dataset
ZueriCrop is a geographically defined Swiss crop-mapping dataset with multi-temporal Sentinel-2 imagery, field-level labels, and an expert-built hierarchy spanning 48 classes. Its agricultural setting produces diverse, highly imbalanced crop distributions represented across multiple spatial and temporal data products.
- Dataset composition: ZueriCrop contains 116,000 field instances with dominant 2019 crop labels covering 48 agricultural classes.Labels come from Swiss Federal Office for Agriculture field polygons.
- Dataset composition: The dataset covers 50 km × 48 km in Zurich and Thurgau using 71 atmospherically corrected Sentinel-2 images collected during 2019.The imagery has a 10-meter ground sampling distance.
- Dataset composition: The scene is partitioned into 24 px × 24 px patches, with approximately 48% of pixels lacking reference labels.Only Red, Green, Blue, and Near-Infrared channels are used at 10-meter resolution.
- Class distribution: The Swiss agricultural setting yields a diverse, highly skewed class distribution because farms use crop rotation and include substantial grassland.Class distributions are shown on a logarithmic pixel-count scale.
- Crop class hierarchy: The 48 crop classes are organized into an expert-defined hierarchy whose levels progress from landscape categories to refined agronomic groups and crop species.The first level reflects broad visual and landscape categories, while later levels encode plant family, agronomic use, and species distinctions.
5. Experiments
The experiments use geographically disjoint five-fold cross-validation to compare the hierarchical method against matched convSTAR baselines, imbalance-aware variants, and external crop-classification methods. Baselines isolate the hierarchical labeling scheme while broader comparisons establish performance against prior approaches.
- Evaluation protocol: Five-fold leave-one-out cross-validation uses five geographically disjoint strips, with four strips for training and one for testing.Performance metrics are averaged across all five train/test splits to reduce regional bias.
- Baselines: The standard convSTAR baseline removes intermediate outputs, losses, and label refinement while keeping hyperparameters aligned with the hierarchical method.This comparison isolates the effect of hierarchical labeling using the same backbone.
- Baselines: Additional convSTAR baselines test data augmentation and class-balanced loss functions as alternative strategies for improving imbalanced-class performance.A second class-balanced variant adjusts the effective learning rate after applying class weights.
- External comparisons: External comparisons include Random Forest and state-of-the-art LSTM, temporal convolutional, and related deep-learning methods for multi-temporal crop classification.These methods provide references beyond the shared convSTAR backbone.
6. Results
The hierarchical ms-convSTAR consistently outperforms non-hierarchical baselines and state-of-the-art methods, especially on rare classes. Its multi-level outputs also support confidence-based label coarsening, while performance remains limited for exceedingly rare classes.
- 6.1. Performance Comparison: All baselines are clearly outperformed by hierarchical ms-convSTAR on every reported performance metric.Simple data augmentation is the strongest baseline, but does not match the hierarchical model.
- 6.1. Performance Comparison: Mean class precision increases by > 11 percentage points, mean class recall by > 10 percentage points, and F1-score by > 11 percentage points over the baseline convSTAR.These gains particularly improve less frequent classes, whereas oversampling raises F1-score by 2.9 percentage points but reduces overall accuracy by 2.3 percentage points.
- 6.1. Performance Comparison: ms-convSTAR improves many less frequent classes without harming frequent classes, but exceedingly rare classes with typically < 1000 pixels remain difficult.Some exceedingly rare classes occur in only a subset of cross-validation stripes and may be absent from either the training or test set.
- 6.1. Performance Comparison: More than 9.9 percentage points of F1-score improvement separates ms-convSTAR from competing state-of-the-art crop-classification methods.The comparison averages precision, recall, and F1-score over five cross-validation folds.
- 6.2. Ablation Study: The CNN-based label refinement module consistently improves all performance metrics, particularly per-class performance.This result comes from an ablation comparing the full hierarchical model with ms-convSTAR without label refinement.
- 6.2. Ablation Study: The hierarchical approach resolves deep-network gradient problems for LSTM and GRU cells and yields superior performance when combined with convSTAR.Without hierarchy, deeper six-layer convLSTM and convGRU versions perform poorly; the hierarchical versions substantially improve them.
- 6.3. Simultaneous multi-level classification: Accuracy improves at coarser hierarchy levels, enabling confidence-based switching from uncertain fine labels to more reliable coarse labels.For example, uncertainty between apple and pear orchard can be replaced by the higher-confidence orchard label for applications prioritizing coverage.
- 6.3. Simultaneous multi-level classification: Lowering the confidence threshold increases coverage but decreases accuracy, while the full three-level model improves with increasing confidence in the full-coverage setting.This supports the usefulness of the imposed hierarchy because coarser classes are easier to discriminate than their finer subclasses.
7. Discussion
The discussion identifies persistent errors for exceedingly rare crops and mixed pixels, alongside broader limitations from label scarcity and domain shift.
- Exceedingly rare crops such as beets, lupine, and grain are still often confused with more frequent classes despite hierarchical classification.Very rare crops are commonly misclassified as meadow, the dataset’s largest class.
- Mixed pixels can cause errors at field boundaries because Sentinel-2’s 10-meter resolution is large relative to Switzerland’s small fields.Boundary pixels may combine meadow and hedge signals or multiple crop types.
- Crop confusions also arise between visually or taxonomically similar classes, including apples versus pears and sugar beets versus other beets.These pairs share genus or species and differ in cultivar or variety.
- The model requires substantial training labels, while active learning could reduce label needs at the cost of a more prolonged and less predictable annotation process.Active learning alternates model training with expert annotation of maximally informative samples.
- Applying the model under domain shift may require retraining or fine-tuning when crop types or ecological conditions change.The stated limitation reflects the model’s dependence on its training data.
8. Conclusion
The conclusion presents ms-convSTAR as a hierarchical, multi-stage approach for multi-temporal crop classification and reports improved performance on the realistic, imbalanced ZueriCrop dataset.
- ms-convSTAR is a multi-stage convolutional recurrent neural network that uses an explicit hierarchical tree of labels for satellite-image crop classification.The scheme is described as generic beyond Sentinel-2.
- The method predicts at multiple interdependent hierarchy levels and uses a CNN-based label-refinement component to favor consistency across levels.Classification proceeds simultaneously across different granularities.
- ZueriCrop uses expert-defined labels based on the Swiss governmental reporting scheme and densely covers a large agricultural region in Central Europe.The dataset is described as larger, more imbalanced, and more representative of real applications than earlier science datasets.
- On ZueriCrop, ms-convSTAR improves per-class performance, outperforms competing state-of-the-art methods, and greatly improves classification of many rare classes.
Appendix A. Performance comparison: Without polygon aggregation
Without polygon aggregation, all methods show reduced performance, with decreases occurring in similar proportions, especially for overall accuracy and F1-score.
- Scores for each method decrease in a similar proportion when polygon aggregation is omitted as post-processing.The comparison concerns results without majority voting over polygons.
- The largest reductions without polygon aggregation are especially evident for overall accuracy and F1-score.
- Table A.6 compares ms-convSTAR with state-of-the-art methods using precision, recall, and F1-score averaged over five cross-validation folds.The table reports mean values over all classes.
Appendix B. Number of Input Channels
The input-channel experiment finds marginal changes for Random Forest and a small performance drop for the proposed method when 20-meter bands are added to the four 10-meter bands.
- For Random Forest, adding the 20-meter bands to the four 10-meter bands produces only marginal improvements.
- For the proposed method, using all nine bands results in a small performance drop compared with using exclusively the four 10-meter bands.
Appendix C. Class-wise Performance
Appendix C reports class-wise performance for the proposed ms-convSTAR across different levels of granularity.
- Class-wise accuracy for each granularity is reported in Table C.8.
Appendix D. More Details about Baselines
The appendices provide baseline parameters and code sources, channel-count comparisons, class-wise performance, and qualitative crop-map results. The qualitative examples also compare model failures at field edges.
- Baseline parameters are provided in Table D.9.
- Source-code links for baseline methods are provided in Table D.10.
- Table B.7 compares performance with respect to the number of input channels using 5-fold evaluation.All channels use bi-cubic upsampling with 20 meter GSD.
- Both models often fail at edge cases, while the 9-channels model makes more mistakes at field edges.The comparison is between the 4-channels and 9-channels models without polygon aggregation as post-processing.
- Figure B.15 places the Sentinel-2 false-color image, the ms-convSTAR crop map, and the ground-truth map in successive rows.The false-color composite uses NIR, Red, and Green.
- Table C.8 reports class-wise performance of ms-convSTAR at different granularity levels across five cross-validation folds.Class frequencies are given in parentheses.