PaperlayerView exact revision on arXiv

Source-linked AI summary

RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation based on Visual Foundation Model

Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, Zhenwei Shi

arXiv:2306.16269v2cs.CV

TL;DR

SAM’s category-agnostic design requires manual prompts, and its effectiveness for remote sensing instance segmentation is insufficiently established. RSPrompter learns semantic prompt generation for SAM, with experiments on WHU, NWPU, and SSDD supporting its effectiveness.

  • Problem

    SAM requires manually crafted points, boxes, or masks and lacks established performance evidence for remote sensing instance segmentation.

  • Method

    RSPrompter learns category-related prompt embeddings or other prompts for SAM to autonomously generate semantic instance-level masks, using anchor- and query-based schemes.

  • Results

    RSPrompter improves remote sensing instance segmentation across WHU, NWPU, and SSDD evaluations, reaching 72.5 APmask on WHU, 67.5 APmask on NWPU, and further improving performance on SSDD.

  • Takeaways & Limitations

    Learning prompts adapts SAM to optical and SAR remote sensing imagery, while the frozen SAM backbone can still provide instance segmentation guidance.

Abstract

from arXiv · show

Leveraging the extensive training data from SA-1B, the Segment Anything Model (SAM) demonstrates remarkable generalization and zero-shot capabilities. However, as a category-agnostic instance segmentation method, SAM heavily relies on prior manual guidance, including points, boxes, and coarse-grained masks. Furthermore, its performance in remote sensing image segmentation tasks remains largely unexplored and unproven. In this paper, we aim to develop an automated instance segmentation approach for remote sensing images, based on the foundational SAM model and incorporating semantic category information. Drawing inspiration from prompt learning, we propose a method to learn the generation of appropriate prompts for SAM. This enables SAM to produce semantically discernible segmentation results for remote sensing images, a concept we have termed RSPrompter. We also propose several ongoing derivatives for instance segmentation tasks, drawing on recent advancements within the SAM community, and compare their performance with RSPrompter. Extensive experimental results, derived from the WHU building, NWPU VHR-10, and SSDD datasets, validate the effectiveness of our proposed method. The code for our method is publicly available at kychen.me/RSPrompter.

I. INTRODUCTION

Remote sensing instance segmentation supports semantic understanding of object location, category, and shape, but complex backgrounds and SAM’s dependence on manual prompts limit automated application. RSPrompter addresses this by learning semantic prompts for SAM and evaluating the approach across remote sensing datasets.

  • Remote sensing instance segmentation provides object location, category, and shape information for applications including defense, surveying, disaster monitoring, and traffic planning.
  • Complex remote sensing backgrounds challenge the generalization and adaptability of existing instance segmentation algorithms.
  • SAM offers strong zero-shot generalization but requires manually supplied points, boxes, or masks and performs category-agnostic segmentation.
  • SAM’s remote sensing performance depends heavily on prompt type, location, and quantity, while background interference and unclear object edges further hinder segmentation.
  • RSPrompter learns category-related prompts for SAM so its decoder can produce multiple instance-level masks with category labels.
  • The method uses a lightweight feature enhancer and flexible prompt embeddings, and the study evaluates SAM extensions and RSPrompter on three diverse datasets.

A. Deep Learning based Instance Segmentation

Prompt learning reframes SAM adaptation for remote sensing instance segmentation by generating category-relevant prompts while preserving SAM’s encoder–prompt encoder–decoder framework. RSPrompter uses enhanced intermediate features and alternative prompting structures to produce instance masks automatically.

  • Prompt learning redesigns downstream inputs to align with a model’s pre-training task, reducing semantic discrepancies and limiting head overfitting.
  • A. A Revisit of SAM: SAM encodes images and prompts separately, then uses mask-decoder cross-attention to generate masks conditioned on foreground/background points, boxes, or masks.
  • C. Extensions on SAM for Instance Segmentation: SAM-based remote sensing extensions include an added instance-segmentation head, classification of masks from “everything” mode, and detector-generated boxes used as prompts.
  • C. Extensions on SAM for Instance Segmentation: RSPrompter generates category-relevant prompt embeddings for instance segmentation while keeping designated model parameters frozen and training the prompting components.
  • 2) Multi-scale Feature Enhancer:: RSPrompter processes SAM intermediate features with a lightweight multi-scale enhancer before generating multiple prompt groups, each associated with semantic categories and instance masks.
  • 3) Anchor-based Prompter:: The anchor-based prompter uses semantic, localization, and prompt heads, with positional encoding added to preserve image-level spatial information during prompt generation.

A. Experimental Dataset and Settings

The study evaluates RSPrompter on three remote sensing instance segmentation datasets differing in size, categories, and modalities, using dataset-specific imagery and train-test splits.

  • The experiments use WHU building, NWPU VHR-10, and SSDD, covering datasets with different sizes, categories, and modalities.
  • WHU: WHU provides 8188 non-overlapping 512×512 RGB aerial images for building instance segmentation.
  • NWPU VHR-10: NWPU uses 80% of its data for training and 20% for testing, with instance annotations used for both training and evaluation.
  • SSDD: SSDD contains 1160 SAR images spanning 1–15 meters in resolution and 2540 ship instances, split randomly into 80% training and 20% testing.

B. Evaluation Protocol and Metrics

Performance is evaluated with COCO mean average precision for box and mask predictions across IoU thresholds, with stricter thresholds indicating more demanding mask accuracy.

  • The evaluation uses COCO mean average precision (mAP) to assess object detection and instance segmentation performance.
  • A prediction is a true positive when its predicted box or mask exceeds the IoU threshold with ground truth and its category matches.
  • AP is averaged across 10 IoU thresholds from 0.50 to 0.95 and across all categories, with larger values indicating more accurate instance masks.
  • AP50 uses a 0.50 IoU threshold, whereas AP75 uses 0.75 and is stricter for evaluating mask accuracy.

C. Implementation Details

RSPrompter uses SAM with a frozen ViT-Huge image encoder, learned prompt embeddings, dataset-specific prompt-group counts, and standard augmentation and optimization settings.

  • Unless otherwise specified, experiments use SAM’s ViT-Huge backbone.
  • Each learned prompt produces one final instance mask by selecting SAM’s first output mask, with five prompt embeddings per group.
  • The number of learnable prompt tokens is set to 100, 70, and 30 for WHU, NWPU, and SSDD, respectively.
  • Training uses 1024 × 1024 images, horizontal flipping, large-scale jittering, a frozen image encoder, and testing of up to 100 masks per image.
  • RSPrompter is compared with multi-stage and single-stage instance segmentation methods, including Mask R-CNN, HTC, SOLOv2, CondInst, BoxInst, and Mask2Former.

1) Quantitative Results on the WHU Dataset:

On WHU building segmentation, RSPrompter-query achieves the strongest reported box and mask performance, outperforming the listed SAM-based and conventional comparisons.

  • RSPrompter-query achieves APbox/APmask of 72.5/72.5 on the WHU building dataset, the strongest reported performance in the comparison.
  • SAM-seg (Mask2Former) improves over original Mask2Former from 69.3/69.2 to 70.7/71.1 on APbox/APmask.
  • SAM-seg (Mask R-CNN) improves over original Mask R-CNN from 66.4/65.6 to 70.3/70.1 on APbox/APmask.
  • The results suggest that learning to prompt adapts SAM to optical remote sensing instance segmentation, including with a completely frozen SAM backbone.

2) Quantitative Results on the NWPU Dataset:

On the diverse, ten-category NWPU dataset, RSPrompter achieves the strongest reported box or mask results among the compared methods, while SAM-based approaches remain competitive.

  • NWPU contains 10 remote-sensing object classes despite being smaller than WHU, testing generalization across more diverse categories.
  • 70.3 APbox is achieved by RSPrompter-anchor, surpassing the other methods in box prediction.
  • 67.5 APmask is achieved by RSPrompter-query, the strongest reported mask-prediction result on NWPU.

3) Quantitative Results on the SSDD Dataset:

On SSDD, a single-category SAR ship dataset with a modality and training-domain shift from SAM, SAM-backbone variants provide only marginal gains, whereas RSPrompter improves performance further.

  • SSDD is a single-category SAR ship instance-segmentation dataset with a modality that differs from the previously discussed datasets and training data from SAM.
  • RSPrompter-anchor and RSPrompter-query further improve performance beyond the SAM-backbone baselines on SSDD.The passage attributes this improvement to liberating the constrained prompt space.
  • 68.7/66.1 for SAM-seg (Mask R-CNN) marginally improves on 67.7/64.3 for Mask R-CNN.
  • 63.0/66.5 for SAM-seg (Mask2Former) marginally improves on 62.7/64.4 for Mask2Former.
  • Qualitatively, RSPrompter produces sharper edges, more defined contours, improved completeness, and closer ground-truth agreement across the evaluated datasets.

E. Ablation Study

The ablation studies examine encoder scale, feature aggregation, prompt counts and embeddings, sine regularization, decoder training, and multi-scale supervision for RSPrompter-query on NWPU.

  • Ablation Study: RSPrompter-query is used for ablations because its design is straightforward and it is treated as the principal method for this study.
  • Encoder and features: The ViT encoder version affects both inference speed and instance-segmentation performance.
  • Encoder and features: Selecting hierarchical feature maps for the aggregator changes prompt-generation performance and efficiency.
  • Feature aggregation: Reducing aggregator feature channels to 32 does not cause a significant performance decrease despite lowering the original ViT-B, ViT-L, and ViT-H dimensions.
  • Feature aggregation: Residual connections materially affect segmentation efficacy, indicating that a wholly serial aggregator is unsuitable for disparate ViT-layer features.
  • Prompt design: The best prompt configuration uses Np = 70 prompt sets and Kp = 5 prompts per set, with each set representing an instance mask.
  • Prompt design: Sine mapping reconciles the prompter’s smooth features with the SAM prompt encoder’s high-frequency embeddings.
  • Decoder and supervision: Excluding SAM’s mask decoder from training significantly reduces segmentation performance, supporting downstream decoder fine-tuning.

10) Analysis of the Low Performance on SAM-cls:

SAM-cls has sub-optimal instance-segmentation performance despite accurate classification, indicating that its weakness lies in SAM’s presegmentation rather than the classifier.

  • SAM-cls remains sub-optimal for remote-sensing instance segmentation despite this classifier accuracy.
  • 98.5% Top-1 classification accuracy shows that the classifier accurately categorizes the predicted masks’ classes.
  • The analysis attributes SAM-cls’s low performance to SAM’s presegmentation under domain shift, not to classifier categorization.
  • SAM-det also does not consistently produce robust segmentation even with accurate categories and precise annotated boxes.

F. Discussions

RSPrompter introduces a SAM-based prompt-learning methodology with anchor- and query-based implementations, while identifying computational and convergence issues for future improvement.

  • The method is designed to extend beyond SAM to other foundation models.
  • The mask decoder remains computationally demanding because it processes more than 64 × 64 input tokens and requires a forward calculation for each prompt group.For 100 instance targets in one image, this calculation is executed 100 times.
  • RSPrompter-query converges more slowly than RSPrompter-anchor, despite having a simpler, lighter network structure and superior performance.The slower convergence is attributed to the absence of relatively explicit supervisory information.
  • RSPrompter learns prompt inputs that enable SAM to autonomously produce semantic instance-level masks for remote sensing images.This replaces the manually crafted prompts required by the original SAM.
  • RSPrompter-anchor uses preset anchors, whereas RSPrompter-query uses queries with optimal transport matching.The two schemes provide distinct implementation choices within the proposed prompt-learning design.
Loading 2306.16269v2…