PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARK-SAM: Self-Prompt Adaptation with Response Knowledge for SAM in Infrared Small Target Segmentation

Aji Mao, Zhenming Peng, Bailin Mu, Tian Pu

arXiv:2608.20754v1cs.CV

TL;DR

Direct transfer of promptable segmentation models to IRSTD suffers a prompt–response mismatch even with target-covering boxes. SPARK-SAM learns target-domain response knowledge and an image-conditioned joint self-prompt state, achieving strong automatic segmentation results across three benchmarks. Its staged analysis separates response acquisition from prompt grounding, while qualitative failures remain on cluttered or diffuse cases.

  • Problem

    The best official SAM2.1 direct-transfer results reach only 4.69%, 1.64%, and 2.28% IoU on three IRSTD datasets despite target-covering loose-box prompts.

  • Method

    SPARK-SAM jointly adapts target-domain prompt-to-mask response behavior and generates an image-conditioned joint self-prompt state, trained with benchmark masks and reliability-aware response guidance.

  • Results

    SPARK-SAM reaches 75.78%, 86.49%, and 68.34% IoU across three IRSTD datasets with 0.726M additional parameters, ranking first on two benchmarks among 14 retrained SAM variants and adaptations.

  • Takeaways & Limitations

    Response adaptation supplies most mask accuracy before reliable point grounding, while prompt supervision aligns candidates and refinement improves localization across the evaluated datasets.

  • Takeaways & Limitations

    Qualitative failures remain on hot clutter, target-like distractors, and diffuse boundaries.

Abstract

from arXiv · show

Promptable segmentation models provide a reusable interface, but direct transfer to automatic infrared small-target segmentation (IRSTD) exposes a mismatch between spatial prompts and target-domain mask responses. In a diagnostic using target-covering loose-box prompts deterministically derived from test reference masks, the best official SAM2.1 results are only 4.69%, 1.64%, and 2.28% IoU on NUAA-SIRST, NUDT-SIRST, and IRSTD-1K. We introduce SPARK-SAM (Self-Prompt Adaptation with Response Knowledge for SAM), which learns target-domain response knowledge and conditions the decoder through an image-conditioned joint self-prompt state. Training combines benchmark-mask supervision with reliability-aware response guidance. SPARK-SAM achieves 75.78%, 86.49%, and 68.34% IoU with 0.726M additional parameters, ranking first on two benchmarks among 14 retrained SAM variants and adaptations evaluated as automatic image-to-mask methods. The staged IRSTD-1K diagnostic shows that response adaptation reaches most of the final IoU before the predicted points acquire reliable target grounding. Prompt supervision aligns the predicted prompt candidates with target locations, and frozen-weight interventions measure output sensitivity to the joint self-prompt state. Matched ablations show consistent accuracy gains from response guidance and high-resolution prompt refinement across all three datasets. Code is available at https://github.com/Sakauma/SPARK-SAM.

Introduction

SPARK-SAM addresses the prompt–response gap in direct SAM2.1 transfer to infrared small-target segmentation by learning target-domain responses and generating image-conditioned self-prompts. It reaches 75.78%, 86.49%, and 68.34% IoU across three datasets with 0.726M additional parameters.

  • Motivation: 4.69%, 1.64%, and 2.28% IoU are the best direct-transfer results on NUAA-SIRST, NUDT-SIRST, and IRSTD-1K despite target-covering loose-box prompts.The diagnostic finds high recall alongside low precision and widespread false-positive activation.
  • Approach: SPARK-SAM learns target-domain prompt-to-mask response knowledge and an image-conditioned joint self-prompt state for automatic inference.The framework retains SAM2’s encoder–prompt encoder–decoder structure while adapting both the conditional response and invoking prompt state.
  • Results: 75.78%, 86.49%, and 68.34% IoU are achieved across three IRSTD datasets with 0.726M additional parameters.The evaluation also tracks mask accuracy, prompt grounding, and decoder dependence on the joint self-prompt state.
  • Analysis: Stagewise diagnostics show that response adaptation reaches most final IoU before reliable point grounding emerges.The contributions also use spatial measurements and frozen-weight interventions to examine self-prompt alignment and decoder use.

Related Work

Prior IRSTD systems use task-specific context, shape, decomposition, kernel, and background-suppression designs, while promptable-model transfer remains underexplored. Related work also investigates automatic prompting, infrared segmentation adaptations, and auxiliary knowledge or distillation.

  • Infrared small-target segmentation: Task-specific IRSTD architectures combine contextual, shape-aware, sparse-background, target-sensitive, and noise-suppression designs.These approaches address weak targets and clutter through specialized modeling choices.
  • Infrared small-target segmentation: Effective transfer of general promptable models to IRSTD remains underexplored despite strong accuracy from task-specific architectures.This gap motivates studying how promptable foundation models can be adapted to infrared targets.
  • Promptable segmentation beyond natural images: SAM and SAM2 separate visual encoding, prompt encoding, and mask decoding, while downstream variants add adapters, prediction heads, and domain specialization.IRSAM and SAMamba are cited as SAM-family adaptations for IRSTD.
  • Automatic prompting and auxiliary knowledge: Automatic-prompting studies infer spatial prompts from image features for nuclei, medical, shadow, and infrared segmentation.SAM-SPL addresses single-frame infrared imagery, while related work also studies sequential infrared segmentation.
  • Automatic prompting and auxiliary knowledge: Auxiliary knowledge methods transfer soft responses or preserve spatial relations, and EdgeSAM incorporates prompt information into distillation.SPARK-SAM anchors its optimization in benchmark masks.

Method

SPARK-SAM adapts both SAM’s target-domain mask response and the image-conditioned prompt state that invokes it. Its training combines benchmark-mask supervision, reliability-filtered response guidance, candidate grounding, and high-resolution prompt refinement.

  • Response and self-prompt adaptation: SPARK-SAM forms an image-conditioned joint self-prompt state from coordinates, local evidence tokens, residual context, and a compact box.The state is decoded from image features and supplies the spatial evidence used by the mask decoder.
  • Response guidance: Benchmark masks define the desired response, while reliability-filtered auxiliary responses provide structural, ranking, boundary, and calibration targets.The guidance pipeline supplies soft masks, boundary distributions, quality estimates, and reliability-weighted supervision.
  • Response and self-prompt adaptation: The self-prompt module predicts objectness, offsets, candidate scores, quality estimates, and a compact box, then samples local image evidence at selected coordinates.Candidate selection uses top-ranked objectness locations, while the joint state combines coordinate and box encodings with local tokens.
  • Response guidance: Training response guidance derives candidate references from infrared cues and frozen-model feedback, including objectness, contrast, compactness, and point–mask agreement.The reference procedure ranks candidates, decodes selected positive points, and uses mask-quality feedback to construct guidance.
  • High-resolution prompt refinement: High-resolution refinement adds fine objectness hypotheses to retained coarse candidates, preserving low-resolution coverage while improving localization for few-pixel targets.A residual head refines the low-resolution objectness map from an early high-resolution encoder map, with objectness, ranking, and coverage losses.

Experiments

Experiments evaluate SPARK-SAM across three IRSTD benchmarks, direct-transfer diagnostics, staged prompting analyses, ablations, efficiency, and qualitative failure cases. The results show strong adaptation gains, measurable dependence on joint self-prompts, and remaining failures under clutter and diffuse boundaries.

  • Benchmark comparison: 75.78%, 86.49%, and 68.34% IoU are achieved by SPARK-SAM on NUAA-SIRST, NUDT-SIRST, and IRSTD-1K, respectively.Among fourteen retrained SAM variants and adaptations, it ranks second on NUAA-SIRST and first on the other two datasets.
  • Direct-transfer diagnostic: All four official SAM2.1 scales remain below 5% IoU despite receiving target-covering mask-derived boxes.The diagnostic uses privileged external boxes computed from test reference masks, while retrained methods use automatic image-to-mask inference.
  • Staged prompting analysis: 68.01% IoU appears at Hit@K = 0.005 before prompt supervision, while refinement reaches 68.34% IoU and Hit@K = 0.955.Prompt supervision raises Hit@K to 0.540, adds 2.89 precision points, and reduces FA by 10.11 px/MP with a 0.35-point IoU decrease.
  • Frozen-weight interventions: Complete joint self-prompt removal lowers mean nIoU by 8.84–28.27 points, while every spatial intervention degrades nIoU across datasets.Box shuffling lowers nIoU by 0.11–0.20 points, and mask-derived box replacement raises IoU by 1.57, 2.48, and 1.77 points across the three datasets.
  • Ablations: Removing response guidance lowers IoU by 12.35–23.87 points, whereas removing refinement lowers IoU by 0.50–1.52 points.Refinement also reduces Hit@K from 0.955 to 0.540; the matched ablations attribute the larger gain to guidance.
  • Efficiency and failure cases: 0.726M additional parameters raise mean latency by only 0.051 ms, with SPARK-SAM reaching 31.8 frames per second.The model contains 39.688M parameters and adds 6.80 MiB peak memory and 6.134 profiled GFLOPs to SAM2.1-Tiny.

Conclusion

SPARK-SAM addresses the prompt–response gap in frozen SAM2.1 transfer by combining target-domain response knowledge with an image-conditioned joint self-prompt state for automatic inference. Across three benchmarks, it outperforms most retrained SAM variants and adaptations, while remaining subject to several post-hoc failure cases.

  • SPARK-SAM combines target-domain response knowledge with an image-conditioned joint self-prompt state for automatic image-to-mask inference.
  • SPARK-SAM’s automatic masks outperform most retrained SAM variants and adaptations across three benchmarks.
  • The staged IRSTD-1K evaluation shows that response adaptation reaches most final IoU before reliable point grounding, while interventions confirm decoder use of the joint self-prompt state.
  • Post-hoc failures involve dim targets, hot clutter, target-like distractors, and diffuse responses.
  • Future work will test SPARK-SAM in cross-dataset and temporal settings.
Loading 2608.20754v1…