PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks

Vivien Sainte Fare Garnot, Loic Landrieu

arXiv:2107.07933v4cs.CV

TL;DR

Panoptic segmentation of agricultural parcels needs temporal satellite observations because parcel content and boundaries exhibit complex dynamics, yet existing methods largely address semantic segmentation or single images. The paper combines temporal-attention encoding with an end-to-end single-stage panoptic model and introduces PASTIS for evaluation. Its approach outperforms competing semantic-segmentation methods and establishes the first state of the art for SITS panoptic segmentation.

  • Problem

    Existing remote-sensing methods were limited to semantic segmentation or single-image instance segmentation, leaving SITS panoptic segmentation without a dedicated approach despite its important agricultural applications.

  • Method

    The paper combines U-TAE temporal self-attention with the PaPs single-stage panoptic framework and introduces the PASTIS panoptic SITS dataset.

  • Results

    The approach outperforms all reimplemented competing methods for semantic segmentation and establishes the first state of the art for SITS panoptic segmentation.

  • Takeaways & Limitations

    Temporal attention and dedicated panoptic modeling provide a foundation for evaluating parcel segmentation from satellite image sequences.

  • Takeaways & Limitations

    The corrected panoptic evaluation changed all Table 2 results by approximately 3 PQ because the original implementation incorrectly counted predictions matched to void instances as false positives.

Abstract

from arXiv · show

Unprecedented access to multi-temporal satellite imagery has opened new perspectives for a variety of Earth observation tasks. Among them, pixel-precise panoptic segmentation of agricultural parcels has major economic and environmental implications. While researchers have explored this problem for single images, we argue that the complex temporal patterns of crop phenology are better addressed with temporal sequences of images. In this paper, we present the first end-to-end, single-stage method for panoptic segmentation of Satellite Image Time Series (SITS). This module can be combined with our novel image sequence encoding network which relies on temporal self-attention to extract rich and adaptive multi-scale spatio-temporal features. We also introduce PASTIS, the first open-access SITS dataset with panoptic annotations. We demonstrate the superiority of our encoder for semantic segmentation against multiple competing architectures, and set up the first state-of-the-art of panoptic segmentation of SITS. Our implementation and PASTIS are publicly available.

1. Introduction

Panoptic segmentation of agricultural parcels in satellite image time series addresses economically and environmentally important monitoring needs while capturing temporal patterns that single images miss. The paper introduces a dedicated encoder, panoptic model, and dataset for this setting.

  • Motivation: Agricultural parcel monitoring supports fair subsidy allocation and compliance with crop-rotation practices, carrying major financial and environmental stakes.The passage cites 50 and 22 billion euros in annual subsidies in Europe and the US, respectively.
  • Problem setting: Panoptic segmentation assigns each pixel both a class and a unique instance label, framing monitoring of parcel content and extent as sequence segmentation.Satellite image time series differ from videos because parcels are static in a geo-referenced frame and acquisition time carries temporal information.
  • Problem setting: Agricultural parcels require learning complex temporal, spatial, and spectral patterns, including phenology, subpixel borders, and rapid interventions such as harvests.These characteristics are not commonly encountered in video processing.
  • Research gap: Existing remote-sensing methods focus on semantic segmentation or single-image instance segmentation, while panoptic networks from computer vision require substantial adaptation for SITS.The paper identifies no dedicated approach for detecting individual objects in satellite image time series.
  • Contributions: The paper introduces U-TAE, PaPs, and PASTIS: a temporal-attention encoder, an end-to-end SITS panoptic method, and an open-access panoptic dataset.PASTIS contains over 2 billion annotated pixels covering more than 4000 km^2; the approach outperforms reimplemented semantic-segmentation competitors and establishes the first SITS panoptic state of the art.

2. Related Work

Prior work provides tools for encoding satellite sequences and segmenting objects in images or videos, but no instance or panoptic segmentation method had been proposed specifically for SITS. The paper positions its approach at this intersection while accounting for SITS-specific differences from video.

  • Research gap: No instance or panoptic segmentation method had been proposed for satellite image time series, despite substantial related work in sequence encoding and image or video panoptic segmentation.This gap motivates adapting methods across these research areas to the SITS setting.
  • Satellite sequence encoding: SITS temporal encoding evolved from handcrafted descriptors and probabilistic models toward recurrent, convolutional, differential, attention-based, and hybrid architectures.Attention-based approaches improved pixel-wise and parcel-wise classification, while U-Net and recurrent hybrids suited semantic segmentation.
  • Satellite image instance segmentation: Remote-sensing instance segmentation mostly analyzes single acquisitions, targeting objects such as trees, buildings, and fields.Some methods delineate borders before postprocessing, while others cluster features after segmentation preprocessing.
  • Video panoptic segmentation: CenterMask is a lighter, more efficient single-stage alternative to Mask-RCNN, and video methods extend image segmentation across time.The paper uses CenterMask as a starting point but notes that SITS differs from natural video in ways requiring architectural adaptation.

3. Method

The method combines multi-scale spatial convolution, temporal self-attention, and a single-stage parcel-centered panoptic decoder for satellite image sequences. U-TAE collapses temporally attended features at every resolution, while PaPs detects parcel centerpoints and predicts their instance properties.

  • Spatio-Temporal Encoding: U-TAE independently embeds each image with a shared multi-level spatial convolutional encoder before temporal aggregation.The encoder stacks feature maps across timestamps and uses Group Normalization because batch samples from different acquisition times are not identically distributed.
  • Spatio-Temporal Encoding: A convolutional U-Net decoder upsamples the temporally collapsed maps and concatenates encoder features to produce spatio-temporal features at all resolutions.Decoder blocks use strided transposed convolutions, convolutions, ReLU activations, and BatchNorm.
  • Spatio-Temporal Encoding: Temporal self-attention generates pixel-wise masks at the lowest resolution, interpolates them across scales, and uses them to collapse each feature sequence into one map per level.L-TAE provides the attention masks, while channel groups are temporally averaged with the corresponding masks before concatenation and projection.
  • Panoptic Segmentation: For each tentative parcel, concatenated multi-scale features predict box size, class, and shape, which are combined with a global saliency map to form pixel-precise masks.The resulting instance predictions are combined into a panoptic map using centerness as the quality measure.
  • Panoptic Segmentation: PaPs adapts a single-stage CenterMask design, avoiding complex region proposal networks because agricultural parcel borders are relatively simple.The panoptic module uses multi-scale feature maps learned by the spatio-temporal encoder.
  • Panoptic Segmentation: A single centerness heatmap detects parcel centerpoints, postponing class prediction to a centerpoint-wise module because parcel shape and border characteristics are mostly crop-independent.Predicted centerpoints are local maxima exceeding their eight neighbors, and the highest-centerness point associated with each parcel is used during training.

4. Experiments

Experiments evaluate PASTIS, semantic segmentation against six SITS encoders, and panoptic segmentation with U-TAE and PaPs. Results show gains from multi-resolution temporal attention and temporal sequences, while panoptic segmentation remains challenging and computationally costly.

  • The PASTIS Dataset: PASTIS contains 2,433 multispectral sequences with semantic and panoptic annotations, including 124,422 individualized agricultural parcels.Sequences contain 38–61 observations and 10 channels; annotations include parcel bounding boxes, pixel-precise masks, and crop types.
  • Semantic Segmentation: U-TAE semantic segmentation outperforms six reimplemented SITS encoders in precision, while recurrent baselines show weaker or similar performance.The comparison reports Overall Accuracy, mIoU, model size, and inference time across approximately 490 sequences per fold.
  • Semantic Segmentation: 4.8 mIoU points are lost when skip connections use temporal means instead of interpolated attention masks, and 3.1 points when channel grouping is removed.The ablations support controlling temporal collapse at all encoder resolutions and using grouped attention masks.
  • Semantic Segmentation: 27.1 performance points are lost with batch normalization, while single-date training drops performance by 24.8 and 42.5 points for August and May.Cloud-covered images received 58% less attention on average than cloud-free images, indicating that the attention module can filter corrupted acquisitions.
  • Panoptic Segmentation: Replacing U-TAE’s temporal encoder with U-BiConvLSTM reduces panoptic quality by 8.2 PQ, and single-image input produces low panoptic quality.The paper attributes the difficulty to ambiguous borders and hard-to-classify parcel content; inference on 490 sequences takes 129 seconds.

5. Conclusion

The paper presents U-TAE, a spatio-temporal encoder using spatial convolution and temporal attention, alongside PaPs and PASTIS. Together, they support panoptic segmentation of satellite image sequences and establish a first state-of-the-art result for this task.

  • U-TAE combines spatial convolution with temporal attention for spatio-temporal encoding.
  • PaPs is the first panoptic segmentation framework operating on satellite image time series.
  • PASTIS is presented as the first large-scale panoptic-ready dataset for satellite image time series.
  • On PASTIS, the approach significantly outperformed other approaches for semantic segmentation and established the first state-of-the-art for panoptic segmentation of satellite image sequences.
  • The authors hope the open-access dataset and results will encourage remote sensing and computer vision research on panoptic SITS segmentation.

Metric Correction

This version corrects a Recognition Quality implementation bug that affected the reported panoptic segmentation metrics in the ICCV 2021 version.

  • The corrected implementation ignores void target instances when computing Recognition Quality.The original implementation counted predictions matched to void targets as false positives, artificially reducing RQ.
  • Across methods, reevaluation increased Panoptic Quality by approximately 3 points, driven by a similar increase in Recognition Quality.
  • Because panoptic metrics were not used in the training loss, the bug did not affect the overall training procedure.

Supplementary Material

The appendix provides additional information about the PASTIS dataset, exact model configuration, and complementary qualitative experiments.

  • The supplementary material documents the PASTIS dataset and the exact model configuration.
  • It also provides complementary qualitative experimental results.

A.1. PASTIS Dataset

PASTIS consists of multi-temporal Sentinel-2 patches with parcel-level semantic and instance annotations, selected across French regions and organized for five-fold evaluation. The dataset preserves realistic cloud-related and irregular temporal sampling conditions while defining explicit background and void-label handling.

  • Overview: Each patch stacks available acquisitions into a T × C × H × W multispectral sequence, with parcel crop-type, background, and instance labels.Parcel pixels receive semantic labels and unique instance labels for each parcel.
  • Dataset Extent: PASTIS uses Sentinel-2 imagery collected from four French tiles and subdivided into 128×128-pixel patches at 10m resolution.The selected tiles cover different French regions with varied climates and crop distributions.
  • Nomenclature: The dataset adopts 18 crop classes selected from FLPIS classes having at least 400 parcels and representation in at least two Sentinel-2 tiles.
  • Patch Boundaries: Parcels with more than 50% of their surface outside a patch are assigned the void label after parcel extents and bounding boxes are cropped to patch boundaries.
  • Void and Background Labels: Background denotes pixels outside declared parcels, while void parcels are excluded from semantic and panoptic metrics and losses.Predictions overlapping void parcels by more than 0.5 IoU are ignored by the metric.
  • Cross-Validation: The 2,433 selected patches are randomly divided into five cross-validation splits for benchmarking.The official train, validation, and test allocation is specified for each fold.
  • Temporal Sampling: PASTIS contains irregularly sampled sequences with 33 to 61 acquisitions because of Sentinel-2 orbits and cloud-related data processing.Reducing inference-time acquisitions by 32, 24, 16, and 8 dates decreased mIoU by −0.7, −2.0, −5.5, and −14.6 points, respectively.
  • Clouds Cover: The dataset retains partially cloud-obstructed acquisitions without further cloud detection or preprocessing.This design expects algorithms to learn to handle cloud-covered observations.

A.2. Implementation Details

The implementation combines U-TAE’s multi-scale convolutional encoder-decoder with temporal attention, while specifying competing architectures, PaPs components, training, and variable-length sequence handling.

  • PaPs and training: Training uses Adam with batch size 4; semantic segmentation runs at learning rate 0.001, whereas panoptic training uses 0.01 before decreasing to 0.001.Both schedules span 100 epochs, split into two 50-epoch stages for panoptic segmentation.
  • U-TAE: U-TAE uses a shared convolutional block in its encoder and decoder, with Group Normalisation in encoding and Batch Normalisation in decoding.The block contains an initial 3 × 3 convolution followed by a residual 3 × 3 convolution.
  • U-TAE: Temporal encoding uses an L-TAE with 16 heads and key-query dimension d_k = 4, with Group Normalisation at its input and output.The normalisation causes each head’s inputs to be layer-normalized.
  • Competing architectures: Competing recurrent models replace L-TAE with ConvLSTM or BiConvLSTM, while standalone ConvLSTM and ConvGRU use hidden sizes of 160 and 188.The bidirectional ConvLSTM uses hidden size 32 in each direction, and ConvLSTM uses 64.
  • Competing architectures: The 3D-Unet uses five 3D-convolution blocks, spatial down-sampling after blocks two and four, and 128 channels in its innermost feature maps.Its blocks double channel counts and use Leaky ReLU with 3D Batch Normalisation.
  • PaPs and training: PaPs predicts saliency and heatmaps from the 32-channel high-resolution feature map d1 using separate convolutional blocks with sigmoid outputs.Each block has convolutional widths 32 and 1, with Batch Normalisation and ReLU after the first convolution.
  • Sequence handling: Variable-length sequences are batch-padded with all-zero images, while a padding mask excludes padded values from spatial and temporal encoding.The same mask also excludes padded values from subsequent processing described in the implementation.

A.3. Additional Results

Additional results show U-TAE’s stronger class-wise semantic segmentation and qualitative sequence-based performance, while revealing confusions among similar classes and failures on thin or fragmented parcels.

  • Semantic segmentation: U-TAE improves class-wise semantic segmentation over the other methods across all crop types, with confusions mainly among semantically close classes.Examples include different cereal types and Sunflower versus Fruits, Vegetable, Flower.
  • Failure cases: Qualitative panoptic and semantic results include failures to recover thin or visually fragmented parcels correctly.These cases are shown in Figure 12 against the ground truth.
  • Semantic segmentation: U-TAE’s multi-scale temporal attention masks produce pixel-precise and consistent predictions for large parcels.Figure 13 compares U-TAE with 3D-Unet, U-BiConvLSTM, and convGRU.
  • Temporal inference: Using the full satellite-image sequence detects a parcel border that is essentially invisible in a single image, whereas mono-temporal classification is poor.The comparison is presented as an example of single-image versus full-sequence inference.
Loading 2107.07933v4…