PaperlayerView exact revision on arXiv

Source-linked AI summary

Cropland PAtteRNS: Parallel Dimensional Attention Networks and Attention to Dataset Disparity for Crop Segmentation in Satellite Imagery Time Series Data

Joseph Metcalfe, Sara Sharifzadeh, Fabio Caraffini

arXiv:2609.38165v1cs.CVcs.LG

TL;DR

Prior crop-segmentation studies often omit parcel-boundary quality assessment, and reviewed studies do not explicitly analyse how dataset tile size affects model performance. PAtteRNS introduces fully factorised temporal-spectral-spatial attention, and achieves the strongest results on nearly all reported metrics and dataset variants.

  • Problem

    Reviewed crop-segmentation studies often omit parcel-boundary quality assessment, and no reviewed studies explicitly analyse tile-size effects on model performance.

    Source: §2.3 · p. 4, §2.4 · p. 5

  • Method

    PAtteRNS introduces a crop-segmentation architecture using fully factorised temporal-spectral-spatial attention and a convolutional refinement head.

    Source: §1 · p. 2

  • Results

    PAtteRNS achieves the strongest results on nearly all metrics and dataset variants, while UTAE leads OA% by at most 0.6% (mean lead 0.3%).

    Source: §5 · p. 14

  • Takeaways & Limitations

    At p = 2, PAtteRNS has a greater performance margin at higher computational cost, while remaining faster and more capable than TSViT.

    Source: §6 · p. 16

    Boundary IoU's fixed pixel-width boundary makes mBIoU scores incomparable across tile-size variants.

    Source: §5.2 · p. 15

Abstract

from arXiv · show

The landscape of satellite imagery time series datasets and boundary-pushing architectures for cropland segmentation has never been richer. However, in this gold rush, important truths are being missed on both fronts, as a drive for the most novel concepts or the largest datasets pushes finer details to the side. In this paper, we present our hybrid transformer-convolutional model, Cropland Parallel Attention and Refinement Network for Segmentation (PAtteRNS), the first model to use self-attention mechanisms separately for each of the temporal, spectral, and spatial aspects of Sentinel-2 multispectral SITS data. To achieve fully-factorised attention in our proposed model, we introduce a novel parallel transformer architecture which significantly reduces the computational complexity of triple-factorised self-attention. We validate our architecture with an in-depth ablation study, and analyse the performance of our model against state-of-the-art crop segmentation models on multiple tile-size variants of the popular PASTIS and MTLCC datasets. Our findings show our model to outperform all others in the task of crop class segmentation, verified across multiple important segmentation metrics, with especially strong performance against compared models seen in the often under-reported parcel delineation quality, for which we use the Boundary IoU metric. We also find that flawed class groupings within datasets can have a significant negative impact on model performance, and report that alternate tile-size variants of crop segmentation datasets produce results incomparable to one-another, invalidating fair comparison between model performance when trained on different tile-sizes. Based on these findings, we suggest further work is required to standardise best practices when constructing SITS crop segmentation datasets, and to enable future dynamic-tile-sizing for ideal model performance.

1. Introduction

The introduction identifies parcel boundaries and dataset design as challenges for cropland segmentation, then presents PAtteRNS and studies architectural and dataset choices. It reports stronger segmentation performance, especially on parcel boundaries.

1. Introduction

  • SITS offers multispectral and multi-temporal imagery suited to crop mapping, while parcel boundaries remain challenging when adjacent fields grow the same crop or are separated by sub-resolution features.The introduction links crop monitoring to food security, agricultural sustainability, environmental change, and subsidy verification.Source: §1 · pp. 1–2
  • The model combines factorised attention across spatial, spectral, and temporal dimensions with convolutional refinement for spatial representations.Parallel temporal and spectral branches make the approach computationally viable and widen reception of raw input signals across the network.Source: §1 · p. 2
  • Tile size strongly affects results, so models trained on different tile sizes cannot be compared fairly.The study compares state-of-the-art models across tile-size variants from multiple datasets.Source: §1 · p. 2
  • Flawed crop-class selections and groupings can degrade segmentation performance more than architectural choices.The paper examines how informed crop-class selection affects achievable performance.Source: §1 · p. 2
  • PAtteRNS outperforms existing state-of-the-art models on key segmentation metrics, particularly parcel boundary quality measured by Boundary IoU.The introduction also states that the model is publicly available for use and adaptation.Source: §1 · p. 2

2. Background

The background traces the shift from convolutional and recurrent approaches to attention-based crop segmentation, then identifies gaps in spectral attention, boundary assessment, and dataset practice.

2.1. Basis of Deep Learning in Crop Segmentation

  • CNN-based segmentation progressed from FCNs and U-Net to methods combining spatial convolution with recurrent temporal processing or 3D convolutions.Single-state methods do not jointly learn spatial and spectral information within meaningful temporal features.Source: §2.1 · pp. 2–3

2.2. Introduction of Vision Transformers

  • Vision transformers use self-attention over image tokens, but standard attention has quadratic sequence-length complexity; ViViT factorises it into sequential spatial and temporal attention.A ViT encoder repeats attention and feed-forward blocks, while local-window attention or hierarchical designs reduce computational costs.Source: §2.2 · p. 3, Figure 1 · p. 3

2.3. Contemporary Crop Segmentation Models

  • UTAE combines a CNN encoder-decoder with temporal attention, while TSViT uses a transformer-only temporal-then-spatial design; spectral attention across SITS dimensions remains under-studied.Prior spectral-attention approaches described here lack a temporal axis or do not factorise attention across spatial and spectral dimensions.Source: §2.3 · p. 4
  • Crop segmentation studies often omit parcel-boundary quality assessment, and traditional metrics may not reflect consistency in boundary-pixel placement.The background notes that no single ideal boundary-accuracy metric has emerged for agricultural delineation.Source: §2.3 · p. 4

2.4. Contemporary Crop Segmentation Dataset Landscape

  • Dataset crop taxonomies vary: seasonal and broad aggregate classes can obscure meaningful differences, while omitting minority classes may make reported performance less representative of deployment.The paper also reports finding no prior studies that explicitly analyse how dataset tile size affects model performance.Source: §2.4 · p. 5

3. Methods

The study uses PASTIS and MTLCC Sentinel-2 datasets, establishes a factorised transformer baseline, and extends it with parallel temporal and spectral attention before spatial attention.

3.1. Datasets

  • PASTIS and MTLCC contain crop-class grouping issues, including taxonomic aggregates and non-homogeneous parcels that differ substantially in prevalence across the datasets.PASTIS has 60.0% non-homogeneous crop parcels; MTLCC has 37.6%.Source: §3.1 · p. 5

3.2. Baseline Transformer Crop Segmentation Model

  • The baseline transformer attends over time for each spatial patch, retains class tokens, then attends across patches and projects predictions back to pixel resolution.Its input tile is represented as temporal, spectral, height, and width dimensions; temporal padding supports fixed sequence lengths.Source: §3.2 · pp. 6–7

3.3. Proposed Parallel Temporal-Spectral Attention Structure

  • Parallel temporal and spectral branches process the same spatial patches independently before spatial attention, avoiding the sequential design’s approximately 5-fold increase in MACs.The temporal branch preserves time while flattening spectral and spatial dimensions; the spectral branch preserves spectral bands while flattening temporal and spatial dimensions.Source: Figure 3 · p. 7, §3.3 · pp. 7–8

3.4. Temporal-Spectral Cross-Attention

The model fuses temporal and spectral class-token representations before spatial attention, optionally refines its output with convolutions, and uses DSRD temporal encodings for comparisons.

3.4. Temporal-Spectral Cross-Attention

  • Cross-attention fusion lets temporal and spectral representations exchange information before spatial attention, with alternatives including one-way attention and balanced mutual exchange.The compared fusion choices include simple concatenation and cross-attention variants.Source: §3.4 · p. 8, Figure 4 · p. 8

3.5. Convolutional Refinement Head

  • The optional Convolutional Refinement Head applies convolutions to unpatched transformer outputs and adds residual corrections to the class logits.It is placed directly before model output and is designed to be portable across transformer and convolutional architectures.Source: §3.5 · pp. 8–9, Figure 5 · p. 9

3.6. Temporal Encodings

  • The experiments use Days since Reference Date (DSRD) encodings to standardise temporal positions across datasets with different date ranges.The authors note that PASTIS spans more than one year and does not start at the calendar-year boundary, and assess encoding effects in an ablation.Source: §3.6 · p. 9

3.7. Comparative Models

The study compares selected crop-segmentation models under a shared experimental framework, evaluates pixel and boundary quality, and specifies training and loss choices.

3.7. Comparative Models

  • UTAE and TSViT are compared as current methods, with UNet3D included as a convolutional baseline; the study aligns comparisons across four dataset variants.The authors state that earlier results used differing tile sizes or evaluation conditions, motivating a same-conditions comparison.Source: §3.7 · pp. 9–10

3.8. Segmentation Metrics

  • Evaluation includes OA%, mIoU, Dice/F1, and Boundary IoU, with mBIoU scores explicitly not directly comparable across tile sizes.Boundary IoU is included to assess parcel-boundary quality; its distance parameter is set to 5% of the tile diagonal.Source: §3.8 · p. 10

3.9. Training and Experimental Setup

  • Models use Adam, cross-entropy loss, batch size 4, and up to 150 training epochs with early stopping; only PASTIS Void pixels are masked.Background pixels are retained because crop-versus-non-crop discrimination is considered important.Source: §3.9 · p. 10

4. Results

The ablations identify the full temporal, spectral, and spatial transformer configuration, with convolutional refinement, as the strongest design. Across dataset tile-size variants, PAtteRNS leads on most reported segmentation metrics, while patch size and dataset conditions affect results.

4.1. Transformer Architecture Design Ablation

  • The full temporal, spectral, and spatial transformer combination performs best across both datasets, with temporal attention the most important single component.Spectral Queries Temporal cross-attention outperforms the reverse direction and performs especially well on PASTIS-128.Source: §4.1 · pp. 10–11, Table 2 · p. 11
  • Adding the convolutional refinement head to all-transformer models improves mean performance by +0.5 OA%, 1.9% in mIoU, and 2.7% in mBIoU.The spatial transformer still improves performance over the temporal-spectral pair, so the refinement head complements rather than replaces it.Source: §4.1 · p. 11
  • The spectral transformer generally adds modest gains, but on PASTIS-128 its inclusion slightly lowers mBIoU from 0.437 to 0.435.On MTLCC-48, inclusion raises OA from 92.0% to 92.1%, mIoU from 0.772 to 0.774, and mBIoU from 0.533 to 0.537.Source: §4.1 · pp. 11–12
  • Reducing patch size from p = 4 to p = 2 improves mBIoU by 5.7%, while other metrics vary little and inference time and model size increase significantly.The study therefore retains d = 128 because increasing the embedding size from 128 to 256 more than triples parameters and roughly doubles inference time.Source: §4.1 · p. 12, Table 3 · p. 12
  • Changing date encoding had little effect, and masking padded temporal values produced minimal change; the final model retains DSRD encoding and explicit masking.The authors retain masking in later experiments to avoid unforeseen variance from pad values.Source: §4.1 · p. 12, Table 4 · p. 12

4.2. Model Performance Comparisons

  • With p = 2, PAtteRNS achieves the highest mIoU, Dice, and mBIoU on both datasets, while UTAE marginally leads in OA%.The comparison covers multiple dataset tile-size variants; MTLCC results are significantly higher than PASTIS results overall.Source: §4.2 · p. 12, Table 5 · p. 13
  • For ViT-based models, reducing patch size improves mBIoU by an average of 8.1% on MTLCC and 8.2% on PASTIS-24, but PAtteRNS at p = 2 declines slightly on PASTIS-128.TSViT at p = 2 was computationally infeasible on PASTIS-128.Source: §4.2 · pp. 12–13

5. Discussion

PAtteRNS matches or exceeds UTAE on metrics that can be compared fairly, with a particularly strong mBIoU result at p = 2. The discussion also identifies tile-size and class-grouping effects, and limits on interpreting boundary scores across tile sizes.

5. Discussion

  • PAtteRNS matches or exceeds UTAE on every metric with a valid fair comparison, despite UTAE leading OA% by at most 0.6% (mean lead 0.3%).Across the strongest configurations, PAtteRNS leads UTAE by 1.4% in mIoU and 0.9% in Dice on MTLCC-24; cross-tile-size mBIoU comparisons are not valid.Source: §5 · p. 14
  • UTAE’s narrow OA% advantage is concentrated in the large Background classes, rather than reflecting broader crop-class segmentation performance.UTAE’s higher true positive rate on Background contributes +0.52 and +1.16 percentage points to its OA% lead on MTLCC-24 and PASTIS-24, respectively.Source: §5 · p. 14
  • At p = 2, PAtteRNS leads the next-best model in mBIoU by an average of 4.4%; Figure 7 compares results across all four dataset and tile-size combinations.At p = 4, PAtteRNS trails UTAE by 2.5% on both MTLCC variants but leads by 2.0% on average across both PASTIS variants.Source: §5 · pp. 14–15, Figure 7 · p. 15

5.2. Boundary IoU

  • Boundary IoU captures boundary quality under-represented by traditional metrics, but its fixed pixel-width boundary makes mBIoU scores incomparable across tile-size variants.The authors propose defining boundary width in real-world spatial units to support more consistent comparisons.Source: §5.2 · pp. 15–16

5.3. Class-wise Performance Across Datasets

  • Avoidable taxonomical aggregates in PASTIS and non-homogeneous classes in both datasets substantially reduce model performance, according to the authors’ analysis.PASTIS-128 parcels were 60% non-homogeneous in ground coverage, comprising 25.2% of dataset pixels.Source: §5.3 · p. 16

6. Conclusion

PAtteRNS achieves state-of-the-art crop segmentation, with particularly strong parcel-boundary delineation. The conclusion also identifies tile size and class grouping as factors shaping reported performance.

6. Conclusion

  • PAtteRNS achieves state-of-the-art results for SITS crop semantic segmentation, with performance advantages at patch sizes p = 4 and p = 2.At p = 4, the advantage comes at similar computational cost to existing models; at p = 2, the margin is greater but computational cost is higher.Source: §6 · p. 16
  • The model shows particularly strong parcel-boundary delineation on Boundary IoU, with the convolutional refinement head identified as key to this strength.The paper also presents Boundary IoU as a valuable criterion for quantitative evaluation of crop segmentation models.Source: §6 · p. 16
  • Results from different SITS tile sizes are not comparable for fair model comparison; smaller tile sizes help many models but can reduce capacity in models designed for larger tiles.The authors identify principled tile-size selection, potentially with dynamic adjustment during training, as future research; experiments suggest smaller tiles may favor global self-attention models.Source: §6 · p. 16
  • Taxonomical aggregate classes are a potential cause of poor performance across the studied models, while effects of seasonal aggregate classes remain unconfirmed.The conclusion recommends minimizing taxonomical aggregations in future datasets and calls for further study of non-homogeneous classes and positional encoding on true multi-year datasets.Source: §6 · p. 16

Data Availability Statement

The authors provide a repository for the associated code and models and identify the datasets and tile-size variants used in the study.

Data Availability Statement

Appendix A. Dataset Class Comparison Table

The appendix table maps crop classes between PASTIS and MTLCC and annotates classes with identified grouping issues; some class names were changed for alignment or clarity.

Appendix A. Dataset Class Comparison Table

  • Marked classes were renamed in one or both datasets to match equivalent classes or improve clarity from their original names.Source: Appendix A · p. 20
  • Table A.7 orders corresponding crop classes first by PASTIS ID, then by remaining MTLCC-exclusive class ID.Source: Table A.7 · p. 20
  • Table A.7 annotates classes for non-homogeneous, taxonomical aggregate, and seasonal aggregate issues, abbreviated NH, TA, and SA.Source: Table A.7 · p. 20

Appendix B. Class-wise Performance Analysis

Class-wise errors differ across the datasets and include confusion involving background, biologically related crops, and aggregate classes. Taxonomical aggregate classes have substantially lower recall in PASTIS-24, while overall model performance is higher on MTLCC than PASTIS.

Appendix B. Class-wise Performance Analysis

  • In MTLCC, Meadow pixels are often confused with Background, while Winter Wheat errors include false positives on Winter Spelt and Winter Triticale.Rapeseed, the seasonal aggregate class, is the highest-performing class for all three models in that dataset.Source: Appendix B · p. 21
  • In PASTIS-24, taxonomical aggregates averaged 60.6% recall for PAtteRNS p = 2, 54.6% for TSViT p = 2, and 52.5% for UTAE.Source: Appendix B · p. 21
  • In PASTIS-24, classes excluding taxonomical aggregates averaged 83.4% recall for PAtteRNS p = 2, 81.0% for TSViT p = 2, and 80.8% for UTAE.Source: Appendix B · p. 21
  • PASTIS errors include Meadow and Grapevine confusion with Background and biological overlap among crop classes, while Winter Triticale is confused with Soft Winter Wheat and Mixed Cereal.The paper reports substantially higher model performance on MTLCC than PASTIS, supporting its theory that taxonomical aggregates limit achievable performance.Source: Appendix B · p. 21

Appendix C. Additional Model Output Visualisations

Appendix C presents class-segmentation outputs and boundary error maps for models on example tiles from MTLCC-48, MTLCC-24, PASTIS-128, and PASTIS-24.

Appendix C. Additional Model Output Visualisations

Appendix D. Confusion Matrices of PAtteRNS, TSViT, & UTAE

Appendix D provides cumulative class-wise confusion matrices for PAtteRNS, TSViT, and UTAE across seeded repeats and folds on PASTIS and MTLCC tile variants.

Appendix D. Confusion Matrices of PAtteRNS, TSViT, & UTAE

Related papers

Loading 2609.38165v1…