PaperlayerView exact revision on arXiv

Source-linked AI summary

MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining

Di Wang, Jing Zhang, Minqiang Xu, Lin Liu, Dongsheng Wang, Erzhong Gao, Chengxi Han, Haonan Guo, Bo Du, Dacheng Tao, Liangpei Zhang

arXiv:2403.13430v2cs.CV

TL;DR

Remote sensing foundation models face domain and task discrepancies because pretraining often uses natural-image classification or object-discrimination objectives. The paper introduces shared-encoder multi-task pretraining on SAMRS and finds competitive performance across 14 downstream datasets, including against larger models.

  • Problem

    Pretraining based on natural images or image-level discrimination can mismatch remote sensing domains and downstream task granularity.

  • Method

    MTP jointly pretrains CNN and vision-transformer backbones with task-specific decoders for semantic segmentation, instance segmentation, and rotated object detection on SAMRS.

  • Results

    MTP improves pretrained models across scene classification, detection, segmentation, and change detection, with competitive performance against existing methods, including larger models.

  • Takeaways & Limitations

    Multi-task supervised pretraining provides transferable remote sensing representations across diverse downstream tasks and performs especially well in low-data finetuning scenarios.

Abstract

from arXiv · show

Foundation models have reshaped the landscape of Remote Sensing (RS) by enhancing various image interpretation tasks. Pretraining is an active research topic, encompassing supervised and self-supervised learning methods to initialize model weights effectively. However, transferring the pretrained models to downstream tasks may encounter task discrepancy due to their formulation of pretraining as image classification or object discrimination tasks. In this study, we explore the Multi-Task Pretraining (MTP) paradigm for RS foundation models to address this issue. Using a shared encoder and task-specific decoder architecture, we conduct multi-task supervised pretraining on the SAMRS dataset, encompassing semantic segmentation, instance segmentation, and rotated object detection. MTP supports both convolutional neural networks and vision transformer foundation models with over 300 million parameters. The pretrained models are finetuned on various RS downstream tasks, such as scene classification, horizontal and rotated object detection, semantic segmentation, and change detection. Extensive experiments across 14 datasets demonstrate the superiority of our models over existing ones of similar size and their competitive performance compared to larger state-of-the-art models, thus validating the effectiveness of MTP.

I. INTRODUCTION

Remote sensing foundation-model pretraining faces domain gaps, limited annotated data, and a mismatch between pretraining and downstream task granularity. The paper introduces multi-task pretraining to align representations with diverse downstream tasks.

  • Natural-image pretraining creates domain gaps because remote sensing images differ in viewpoint, color, and spatial resolution.
  • Large annotated remote sensing datasets remain scarce, although MillionAID expanded the scale of supervised scene pretraining.
  • Unlabeled remote sensing imagery is abundant, motivating self-supervised approaches for learning from large image collections.
  • Existing approaches often rely on a single in-domain data source, potentially limiting the specialized knowledge learned by remote sensing models.
  • Task-granularity mismatch can make models pretrained for scene classification less effective on pixel-level segmentation.
  • MTP addresses this discrepancy by combining multi-task supervised pretraining with semantic segmentation, instance segmentation, and rotated object detection labels.

B. Multi-Stage Pretraining for RS Foundation Model

The paper situates MTP among sequential and multi-task pretraining strategies and describes the RS-specific backbone components used to support adaptive spatial modeling. RVSA introduces learnable window transformations suited to varied object orientations.

  • Multi-Stage Pretraining for RS Foundation Model: Sequential pretraining transfers models from natural images to remote sensing data through multiple pretraining stages.
  • Multi-Task Pretraining for RS Foundation Model: Prior multi-task frameworks were constrained by limited model scale, single-task finetuning, or the absence of jointly annotated datasets.
  • Multi-Task Pretraining for RS Foundation Model: SAMRS supplies annotations for semantic segmentation, instance segmentation, and rotated object detection to support unified multi-task pretraining.
  • RVSA: RVSA adapts attention windows through learnable scaling, translation, and rotation factors for remote sensing objects with varied orientations.
  • RVSA: RVSA replaces most full attention with varied-size window attention while retaining full attention in one-quarter of the network depth.

2) InternImage:

InternImage is presented as a CNN foundation-model backbone that combines large-kernel and transformer-inspired mechanisms, while MTP uses feature pyramids and task-specific decoders.

  • 2) InternImage:: InternImage integrates dynamic sparse kernels with long-range context capture, adaptive spatial aggregation, and efficient computation.
  • 2) InternImage:: MTP feeds backbone feature pyramids into multiple decoders for rotated detection, instance segmentation, and semantic segmentation.
  • 2) InternImage:: The RVSA model configurations are documented across different model variants.

C. Multi-Task Pretraining

MTP jointly optimizes a shared foundation model with task-specific decoders using SAMRS annotations for three dense prediction tasks. The implementation unifies components from single-task vision frameworks.

  • C. Multi-Task Pretraining: The authors concurrently pretrain on semantic segmentation, instance segmentation, and rotated object detection using SAMRS annotations.
  • C. Multi-Task Pretraining: SAMRS-derived masks, boxes, and semantic labels produce the task losses L_rod, L_ins, and L_sem.
  • C. Multi-Task Pretraining: The total objective combines rotated detection, instance segmentation, and semantic segmentation losses: L = L_rod + L_ins b + L_ins m + L_sem.
  • C. Multi-Task Pretraining: Because the underlying packages support single tasks, their dataloaders, model structures, loss functions, and metric calculators are integrated into one unified pipeline.
  • C. Multi-Task Pretraining: The training-cost configurations for implementing MTP differ across the selected models.

D. Implementation Details

The evaluation examines pretrained models across scene classification, using EuroSAT and RESISC-45, with results reported as overall accuracy.

  • Scene classification is evaluated because it requires no extra decoder and reflects the pretrained model’s overall representation capability.
  • EuroSAT contains 27,000 Sentinel-2 images across 10 land-use and land-cover classes, while RESISC-45 contains 31,500 images across 45 categories.
  • The section reports overall accuracy for different model pretraining strategies and finetuned pretrained models on EuroSAT and RESISC-45.

2) Implementation Details:

The implementation uses standard finetuning setups for scene classification and horizontal object detection, with dataset-specific training configurations and AP50 evaluation for detection.

  • Scene classification: Scene classification uses EuroSAT and RESISC-45, with 100 and 200 training epochs, respectively, and batch size 64.
  • Horizontal object detection: Horizontal detection is evaluated on Xview and DIOR using RetinaNet for Xview and Faster R-CNN for DIOR.
  • Horizontal object detection: Xview contains 846 available high-resolution training images, from which 700 are used for training and 146 for testing.
  • Horizontal object detection: Table V reports AP50 for finetuned pretrained models on Xview and DIOR, including supervised ImageNet-1K and random-initialization baselines.

3) Finetuning Results and Analyses:

Finetuning results show strong MTP gains for horizontal and rotated detection, though advantages vary across rotated-detection datasets and remain below STD on DOTA-V1.0.

  • Horizontal object detection: MTP substantially improves ViT-L + RVSA on Xview and enables it to outperform CtxMIM, achieving the best reported performance there.
  • Horizontal object detection: MTP enables ViT-B + RVSA to outperform existing methods, including SkySense, on DIOR horizontal detection, while also improving ViT-L + RVSA.
  • Horizontal object detection: MTP models demonstrate strong horizontal-box detection despite RVSA being originally designed around the diverse orientations of remote-sensing objects.
  • Rotated object detection: Rotated detection is evaluated on DIOR-R, FAIR1M-2.0, DOTA-V1.0, and DOTA-V2.0, reflecting a task shaped by overhead-view object orientations.
  • Rotated object detection: Except on DIOR-R, MTP models do not always show obvious advantages on rotated detection, possibly because longer finetuning on larger datasets diminishes the benefit.
  • Rotated object detection: RVSA-L outperforms a ViT-G foundation model with over 1 billion parameters on DOTA-V2.0, while the models remain behind STD on DOTA-V1.0.

D. Semantic Segmentation

MTP improves finetuned semantic segmentation performance across SpaceNetv1 and LoveDA, though gains vary by task complexity and model capacity.

  • Experimental setup: Semantic segmentation evaluates both building-footprint extraction and multi-class land-cover perception using SpaceNetv1 and LoveDA.SpaceNetv1 provides single-class building extraction, while LoveDA represents classical land-cover classification.
  • Results: MTP enhances pretrained-model performance on semantic segmentation, with larger improvements on LoveDA than on SpaceNetv1.The reported comparison identifies stronger gains on the land-cover dataset than on the building-extraction dataset.
  • Results: On SpaceNetv1, MTP-pretrained models achieve state-of-the-art accuracy, while on LoveDA they surpass previous methods except BillionFM.BillionFM uses a model with over 1 billion parameters.
  • Limitations: Increasing model capacity does not improve performance on simple extraction tasks, and MTP decreases ViT-L + RVSA performance on SpaceNetv1.The passage attributes the SpaceNetv1 behavior probably to overfitting and identifies it as requiring further exploration.

1) Dataset:

The change-detection evaluation spans datasets of different scales and shows strong MTP results on larger datasets, with weaker performance on small, out-of-domain OSCD data.

  • Dataset: Change detection is evaluated on OSCD, WHU, LEVIR, and SVCD/CDD datasets spanning different scales and image characteristics.The evaluation uses bi-temporal change detection, including urban, building, and seasonal-change settings.
  • Results: MTP improves pretrained-model performance across the change-detection datasets, with especially competitive results on WHU, LEVIR, and SVCD/CDD.Even a base RVSA model with a simple UNet achieves competitive performance, while larger models provide further gains.
  • Results: The models achieve the best accuracy on WHU and LEVIR, outperforming almost all existing methods, including SkySense, ChangeCLIP, and BAN.The comparison includes methods using larger networks, language-derived knowledge, or existing foundation-model adapters.
  • Limitations: On small-scale OSCD, MTP remains useful but leaves relatively large gaps compared with current methods.The authors attribute this to data discrepancy and image size.
  • Limitations: OSCD performance is constrained by its lower-resolution multispectral imagery and small 96 × 96 training crops, which may hinder non-hierarchical vision transformers.The pretrained models use high-resolution imagery, while several comparison methods use multispectral pretraining, pyramid features, or larger crops.
  • Limitations: Further exploration is needed to improve finetuning on out-of-domain datasets with small volumes and input sizes.This scope boundary follows the reported OSCD limitations and the contrast with the other change-detection datasets.

1) Influence Factors of Multi-Task Pretraining:

MTP generally improves finetuning, but its benefit depends on training duration, dataset characteristics, and available samples. The evidence suggests stronger gains in limited-data settings and diminishing advantages during prolonged finetuning.

  • Overall influence: MTP improves foundation-model finetuning in most cases across five downstream task types and fourteen datasets, though gains are not universal.The authors investigate training-related factors to explain datasets where all three pretrained models do not improve.
  • Training duration: Increasing training iterations tends to reduce MTP’s finetuning benefit, although this pattern varies across pretrained models and segmentation tasks.The authors use Average Iteration per Class (AIC) to account for differences in dataset class counts.
  • Training duration: A relatively large AIC corresponds to a negative MTP impact, suggesting that MTP advantages diminish during extended finetuning.The critical AIC threshold remains uncertain because experimentation is limited.
  • Dataset capacity: Average Pixels per Class (APC) shows trends similar to AIC, but its relationship with MTP performance is less discernible, possibly because RS images contain redundant pixels.APC approximately reflects the data volume encountered during finetuning.
  • Fewer-sample finetuning: As training samples decrease, accuracies fall for all models while performance gaps widen, indicating increasingly significant MTP benefits in limited-data finetuning.These findings support MTP’s usefulness when downstream training data are scarce.

3) Decoder Parameter Reusing:

The study examines MTP-based models across detection, segmentation, and change detection, including visual predictions from multiple remote-sensing datasets. These results support strong representation capability across diverse downstream tasks.

  • Decoder parameter reusing: MTP uses task-specific segmentation and detection decoders, motivating experiments that reuse their pretrained weights during finetuning.Decoder reuse is evaluated with a ViT-B + RVSA backbone, although only semantic-segmentation and rotated-detection decoders are eligible.
  • Downstream visualizations: MAE + MTP pretrained ViT-L + RVSA predictions are presented for horizontal and rotated detection, semantic segmentation, and bi-temporal change detection.The visualizations span Xview, DIOR, DIOR-R, FAIR1M-2.0, DOTA-V1.0, DOTA-V2.0, SpaceNetv1, LoveDA, OSCD, WHU, LEVIR, and SVCD/CDD.
  • Overall outcome: MTP enables an RS foundation model with over 300 parameters and superior representation capability across various downstream tasks.The conclusion also reports competitive performance against existing methods, including larger models.

APPENDIX

The appendix provides detailed finetuning accuracies for three pretrained models across six remote-sensing datasets. The listed tables cover horizontal detection, rotated detection, and semantic segmentation benchmarks.

  • Appendix scope: Detailed finetuning accuracies are reported for ViT-B + RVSA, ViT-L + RVSA, and InternImage-XL across DIOR, DIOR-R, FAIR1M-2.0, DOTA-V1.0, DOTA-V2.0, and LoveDA.The tables distinguish the datasets and models used for the appendix comparisons.
  • Horizontal detection: Table XI reports detailed accuracies on the DIOR dataset.The table is part of the appendix’s model-comparison results.
  • Rotated detection: Table XII reports detailed accuracies on the DIOR-R dataset.The table covers a rotated-detection benchmark.
  • Rotated detection: Table XIII reports detailed accuracies on the FAIR1M-2.0 dataset.The table covers a rotated-detection benchmark.
  • Rotated detection: Tables XIV and XV report detailed accuracies on DOTA-V1 and DOTA-V2, respectively.Both tables are included among the appendix’s rotated-detection results.
  • Semantic segmentation: Table XVI reports detailed accuracies on the LoveDA dataset.The table covers a semantic-segmentation benchmark.
Loading 2403.13430v2…