PaperlayerView exact revision on arXiv

Source-linked AI summary

RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model

Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, Zhenwei Shi

arXiv:2306.16269v2cs.CV

TL;DR

SAM’s category-agnostic design requires manual prompts, and its effectiveness for remote sensing instance segmentation is insufficiently established. RSPrompter learns semantic prompt generation for SAM, with experiments on WHU, NWPU, and SSDD supporting its effectiveness.

Abstract

from arXiv · show

Leveraging the extensive training data from SA-1B, the Segment Anything Model (SAM) demonstrates remarkable generalization and zero-shot capabilities. However, as a category-agnostic instance segmentation method, SAM heavily relies on prior manual guidance, including points, boxes, and coarse-grained masks. Furthermore, its performance in remote sensing image segmentation tasks remains largely unexplored and unproven. In this paper, we aim to develop an automated instance segmentation approach for remote sensing images, based on the foundational SAM model and incorporating semantic category information. Drawing inspiration from prompt learning, we propose a method to learn the generation of appropriate prompts for SAM. This enables SAM to produce semantically discernible segmentation results for remote sensing images, a concept we have termed RSPrompter. We also propose several ongoing derivatives for instance segmentation tasks, drawing on recent advancements within the SAM community, and compare their performance with RSPrompter. Extensive experimental results, derived from the WHU building, NWPU VHR-10, and SSDD datasets, validate the effectiveness of our proposed method. The code for our method is publicly available at kychen.me/RSPrompter.

I. INTRODUCTION

Remote sensing instance segmentation supports semantic understanding of object location, category, and shape, but complex backgrounds and SAM’s dependence on manual prompts limit automated application. RSPrompter addresses this by learning semantic prompts for SAM and evaluating the approach across remote sensing datasets.

  • Remote sensing instance segmentation provides object location, category, and shape information for applications including defense, surveying, disaster monitoring, and traffic planning.Source: §I · p. 1
  • Complex remote sensing backgrounds challenge the generalization and adaptability of existing instance segmentation algorithms.Source: §I · p. 1
  • SAM offers strong zero-shot generalization but requires manually supplied points, boxes, or masks and performs category-agnostic segmentation.Source: §I · pp. 1–2
  • SAM’s remote sensing performance depends heavily on prompt type, location, and quantity, while background interference and unclear object edges further hinder segmentation.Source: Figure 1 · p. 1, §I · p. 2
  • RSPrompter learns category-related prompts for SAM so its decoder can produce multiple instance-level masks with category labels.Source: §I · p. 2
  • The method uses a lightweight feature enhancer and flexible prompt embeddings, and the study evaluates SAM extensions and RSPrompter on three diverse datasets.Source: §I · p. 2

A. Deep Learning based Instance Segmentation

Prompt learning reframes SAM adaptation for remote sensing instance segmentation by generating category-relevant prompts while preserving SAM’s encoder–prompt encoder–decoder framework. RSPrompter uses enhanced intermediate features and alternative prompting structures to produce instance masks automatically.

  • Prompt learning redesigns downstream inputs to align with a model’s pre-training task, reducing semantic discrepancies and limiting head overfitting.Source: C. Prompt Learning · p. 3

A. A Revisit of SAM

  • SAM encodes images and prompts separately, then uses mask-decoder cross-attention to generate masks conditioned on foreground/background points, boxes, or masks.Source: A. A Revisit of SAM · p. 3

C. Extensions on SAM for Instance Segmentation

2) Multi-scale Feature Enhancer:

3) Anchor-based Prompter:

  • The anchor-based prompter uses semantic, localization, and prompt heads, with positional encoding added to preserve image-level spatial information during prompt generation.Source: 3) Anchor-based Prompter · p. 5

A. Experimental Dataset and Settings

The study evaluates RSPrompter on three remote sensing instance segmentation datasets differing in size, categories, and modalities, using dataset-specific imagery and train-test splits.

WHU

NWPU VHR-10

SSDD

B. Evaluation Protocol and Metrics

Performance is evaluated with COCO mean average precision for box and mask predictions across IoU thresholds, with stricter thresholds indicating more demanding mask accuracy.

C. Implementation Details

RSPrompter uses SAM with a frozen ViT-Huge image encoder, learned prompt embeddings, dataset-specific prompt-group counts, and standard augmentation and optimization settings.

1) Quantitative Results on the WHU Dataset:

On WHU building segmentation, RSPrompter-query achieves the strongest reported box and mask performance, outperforming the listed SAM-based and conventional comparisons.

2) Quantitative Results on the NWPU Dataset:

On the diverse, ten-category NWPU dataset, RSPrompter achieves the strongest reported box or mask results among the compared methods, while SAM-based approaches remain competitive.

3) Quantitative Results on the SSDD Dataset:

On SSDD, a single-category SAR ship dataset with a modality and training-domain shift from SAM, SAM-backbone variants provide only marginal gains, whereas RSPrompter improves performance further.

E. Ablation Study

The ablation studies examine encoder scale, feature aggregation, prompt counts and embeddings, sine regularization, decoder training, and multi-scale supervision for RSPrompter-query on NWPU.

Ablation Study

  • RSPrompter-query is used for ablations because its design is straightforward and it is treated as the principal method for this study.Source: E. Ablation Study · p. 10

Encoder and features

Feature aggregation

Prompt design

Decoder and supervision

10) Analysis of the Low Performance on SAM-cls:

SAM-cls has sub-optimal instance-segmentation performance despite accurate classification, indicating that its weakness lies in SAM’s presegmentation rather than the classifier.

F. Discussions

RSPrompter introduces a SAM-based prompt-learning methodology with anchor- and query-based implementations, while identifying computational and convergence issues for future improvement.

  • The method is designed to extend beyond SAM to other foundation models.Source: F. Discussions · p. 14, §V · p. 15
  • The mask decoder remains computationally demanding because it processes more than 64 × 64 input tokens and requires a forward calculation for each prompt group.For 100 instance targets in one image, this calculation is executed 100 times.Source: F. Discussions · p. 15
  • RSPrompter-query converges more slowly than RSPrompter-anchor, despite having a simpler, lighter network structure and superior performance.The slower convergence is attributed to the absence of relatively explicit supervisory information.Source: F. Discussions · p. 15
  • RSPrompter learns prompt inputs that enable SAM to autonomously produce semantic instance-level masks for remote sensing images.This replaces the manually crafted prompts required by the original SAM.Source: §V · p. 15
  • RSPrompter-anchor uses preset anchors, whereas RSPrompter-query uses queries with optimal transport matching.The two schemes provide distinct implementation choices within the proposed prompt-learning design.Source: §V · p. 15

Related papers

Loading 2306.16269v2…