Source-linked AI summary
Multi-Modal Temporal Attention Models for Crop Mapping from Satellite Time Series
Vivien Sainte Fare Garnot, Loic Landrieu, Nesrine Chehata
TL;DR
Crop mapping lacks a unified temporal-attention approach for exploiting complementary optical and radar time series across multiple tasks. The paper adapts these models with several fusion schemes, introduces PASTIS-R, and finds that multimodality improves performance and cloud robustness, with fusion choices trading accuracy against efficiency.
Problem
Temporal attention-based crop-mapping methods have largely focused on single modalities despite the complementary information in optical and radar time series.
Method
The paper reformulates fusion strategies for temporal attention-based SITS encoders, adds model enhancements, and augments PASTIS with aligned Sentinel-1 radar time series to create PASTIS-R.
Results
Across parcel classification, semantic segmentation, and panoptic segmentation, combining optical and radar imagery improved overall performance and robustness to cloud obstruction.
Takeaways & Limitations
Late fusion performed best for parcel-based classification, while mid-fusion offered a lower-parameter alternative with slightly worse performance.
Takeaways & Limitations
Results are most relevant to crop mapping under the meteorological, terrain, and crop conditions represented by four French metropolitan regions, and cloud robustness was tested with artificially reduced optical observations.
Abstract
from arXiv · showhide
Optical and radar satellite time series are synergetic: optical images contain rich spectral information, while C-band radar captures useful geometrical information and is immune to cloud cover. Motivated by the recent success of temporal attention-based methods across multiple crop mapping tasks, we propose to investigate how these models can be adapted to operate on several modalities. We implement and evaluate multiple fusion schemes, including a novel approach and simple adjustments to the training procedure, significantly improving performance and efficiency with little added complexity. We show that most fusion schemes have advantages and drawbacks, making them relevant for specific settings. We then evaluate the benefit of multimodality across several tasks: parcel classification, pixel-based segmentation, and panoptic parcel segmentation. We show that by leveraging both optical and radar time series, multimodal temporal attention-based models can outmatch single-modality models in terms of performance and resilience to cloud cover. To conduct these experiments, we augment the PASTIS dataset with spatially aligned radar image time series. The resulting dataset, PASTIS-R, constitutes the first large-scale, multimodal, and open-access satellite time series dataset with semantic and instance annotations.
1 Introduction
The paper targets crop mapping by combining complementary optical and radar satellite time series in temporal attention-based models. It introduces PASTIS-R and evaluates multimodal fusion across classification and segmentation tasks, reporting broader performance and cloud-robustness gains.
- Motivation: Optical and radar observations provide complementary information because their modalities differ in captured content, resolution, and resilience to atmospheric conditions.The paper specifically motivates combining multispectral optical imagery with C-band radar for crop mapping.
- Dataset and evaluation: 2433 multimodal Sentinel-1 and Sentinel-2 image time series form the PASTIS-R dataset used for evaluation.PASTIS-R supports experiments on parcel-based classification, semantic segmentation, and panoptic segmentation.
- Method: The study reformulates fusion strategies for temporal attention-based SITS encoders and evaluates them with standard model enhancements.The contribution focuses on adapting temporal attention models to multimodal sequences.
- Dataset and evaluation: PASTIS-R is presented as the first large-scale, multimodal, open-access satellite time-series dataset with panoptic annotations.The dataset augments PASTIS with aligned radar time series.
- Evaluation: Multimodal fusion is evaluated across parcel classification, semantic segmentation, and panoptic segmentation, defining a new state-of-the-art for all tasks.The experiments compare fusion schemes and enhancements across these three crop-mapping settings.
- Results: Combining optical and radar imagery improves robustness to varying cloud cover.The paper identifies this robustness as a central benefit of multimodal crop mapping.
2 Related Work
Prior multimodal SITS work spans traditional machine learning, deep learning, and several fusion strategies. The paper organizes these approaches around early, late, and decision fusion while introducing a mid-fusion strategy for temporal image sequences.
- Traditional approaches: Traditional crop-mapping methods commonly combine optical and radar features through early fusion before model processing.Feature selection and decision fusion have also been explored as alternatives.
- Deep learning: Early deep-learning approaches concatenate optical and radar channels, while later work separately encodes the modalities with dedicated networks.These designs differ in whether multimodal interaction occurs before or during learned representation construction.
- Other fusion settings: Related multimodal settings include late fusion, common feature spaces, and separate encoders for mono-temporal or heterogeneous observations.These examples extend beyond the crop-mapping time-series setting.
- Radar processing: SAR crop-mapping methods may use backscattering, interferometric, or polarimetric features derived from radar signals.Backscattering coefficients are the most commonly used radar representation in the reviewed crop-mapping literature.
- Fusion taxonomy: The three commonly used schemes differ by fusion stage: raw features, learned spatio-temporal features, or independently averaged decisions.These correspond to early, late, and decision fusion, respectively.
3 Methods
The paper defines multimodal fusion strategies for temporal satellite imagery, introducing mid-fusion and auxiliary training techniques for temporal attention models. These strategies are integrated across crop classification and segmentation tasks.
- Fusion setup: Multimodal temporal inputs are modeled as M modality-specific sequences, each represented by temporal, channel, and spatial dimensions.All modalities are assumed to be resampled to the same spatial resolution.
- Early Fusion: Early fusion concatenates modalities at the raw feature level, but differing acquisition times require interpolation that can be costly and questionable for rapidly changing processes.Cloud-obstructed optical observations further complicate temporal interpolation.
- Late Feature Fusion: Late fusion encodes each modality separately, concatenates the resulting embeddings, and uses a shared decoder to make a joint prediction.The decoder can therefore use all modalities simultaneously despite modality-specific latent features.
- Mid-Fusion: Mid-fusion temporally interweaves modality-specific spatial features before a single temporal encoder and decoder, combining simultaneous multimodal processing with one temporal encoder and minimal preprocessing.Each modality has a dedicated spatial encoder, after which features are concatenated chronologically.
- Auxiliary Supervision: Auxiliary losses supervise modality-specific predictions to address unequal discriminative information, with benefits expected mainly for late and decision fusion rather than early or mid-fusion.Early-fusion auxiliary supervision would require duplicating the network, while decision fusion already produces single-modality predictions.
- Training and evaluation: The study also evaluates temporal dropout and integrates the fusion schemes into temporal attention networks for parcel-based crop classification and related tasks.Temporal dropout randomly removes observations to reduce over-reliance on a single modality.
4 Experiments
The experiments assess multimodal temporal-attention models across parcel classification, semantic segmentation, and panoptic segmentation using the augmented PASTIS-R dataset. Fusion generally improves performance and cloud-cover robustness, while fusion strategy and training enhancements affect efficiency and resilience.
- Experimental setup: PASTIS-R augments PASTIS with Sentinel-1 radar observations, retaining semantic and instance annotations for three crop-mapping tasks.The dataset contains roughly 70 radar acquisitions for each of 2,433 patches, totaling about 339,000 added radar images.
- Parcel classification: Late fusion with auxiliary supervision and temporal dropout improves parcel-classification performance by 3.3 mIoU over an optical-only network.Without enhancements, early and mid-fusion improve over optical-only models, whereas auxiliary supervision and temporal dropout improve later fusion models.
- Training analysis: Auxiliary supervision changes gradient allocation across modalities, increasing the proportion of flow in selected radar and optical modules rather than only strengthening radar.Spatial encoders contribute as much as or more than temporal encoders despite having four times fewer parameters.
- Semantic segmentation: Late fusion improves semantic-segmentation performance by 2.7 mIoU over unimodal models, with auxiliary supervision adding another 0.5 point.Temporal dropout was necessary to fit the memory-intensive late-fusion model, whose input sequences exceeded the available GPU memory without reduction.
- Cloud-cover robustness: Multimodal models remain nearly constant in performance with up to 50% of optical acquisitions missing, while decision fusion is most resilient under more severe removal.With 90% of optical images removed, decision fusion still outperforms the radar modality by several mIoU points; auxiliary supervision and temporal dropout also improve resilience.
- Panoptic segmentation: Radar improves panoptic quality by 1.6 points with early fusion and 1.2 points with late fusion, mainly through better recognition rather than boundary segmentation.Under inference with only 30% of optical observations, late fusion achieves 37.6 PQ versus 33.0 PQ for the optical baseline.
5 Discussion
Combining optical and radar imagery improved performance across all tasks and increased robustness to cloud cover, while fusion schemes presented distinct trade-offs in efficiency, adaptability, and resilience. Late fusion with enhancements was the strongest general choice, whereas other schemes suited narrower computational or cloud conditions.
- Combining optical and radar imagery increased performance across all tasks and improved robustness to cloud cover.The study reports this pattern for parcel classification, semantic segmentation, and panoptic segmentation.
- Recommendations: Early fusion was compact and competitive but required expensive interpolation and was the least robust scheme to cloud cover.Its interpolation preprocessing also made it slower than late fusion despite using a smaller network for parcel classification and semantic segmentation.
- Qualitative results: Radar improved qualitative panoptic predictions by recovering missed parcels, correcting crop types, and handling parcels with internal subdivisions better than optical-only models.The figures also identify benefits for difficult classes such as Winter triticale.
- Recommendations: Mid fusion had the fastest runtime and lowest memory requirement without preprocessing, but its cloud robustness and task adaptability were limited.The method was recommended for parcel classification without extensive cloud cover, while its separate spatial and temporal encoders hinder straightforward pixel-based adaptation.
- Recommendations: Late fusion with enhancements delivered the best performance, highest adaptability, and strong resilience to even extreme cloud cover.The authors present late fusion as the default recommendation for multimodal temporal-attention models.
- Recommendations: Decision fusion was the most cloud-resilient approach but had the highest parameter count, lagged in performance, and was prohibitively costly for panoptic segmentation.The authors recommend it when only a few optical observations are expected at inference.
- Enhancements: Temporal dropout benefited all fusion schemes and the optical baseline while reducing the networks’ training memory footprint.Auxiliary supervision particularly improved precision for late and decision fusion, which use dedicated modality encoders.
- Limitations: The findings are geographically constrained because PASTIS covers four regions of metropolitan France and omits crop types such as rice.The authors note that unrepresented crops may benefit even more from radar because of distinctive SAR responses.
6 Conclusion
The paper develops temporal-attention fusion architectures for crop mapping from radar and optical satellite time series and evaluates them across three tasks. Multimodal inputs improve performance and cloud robustness, with late fusion strongest for parcel classification and mid fusion offering a lower-parameter alternative.
- The paper explores fusion architectures using temporal attention to predict agricultural crop type maps from radar and optical satellite time series.
- Across parcel classification, semantic segmentation, and panoptic segmentation, both modalities improved overall performance and robustness to cloud obstruction.Sentinel-2 multispectral information was more discriminative than Sentinel-1 SAR, but combining both remained beneficial.
- Late fusion outperformed other approaches on parcel-based classification, while decision fusion may be favored under highly unpredictable cloud conditions.Both approaches require distinct spatio-temporal encoders and are computationally costly.
- Mid fusion performed marginally worse than late fusion on parcel classification while using close to half the trainable parameters.The authors identify it as a valid option for applications with limited computational resources and release PASTIS-R for further multimodal research.