Source-linked AI summary
IRSAM: Advancing Segment Anything Model for Infrared Small Target Detection
Mingjin Zhang, Yuchun Wang, Jie Guo, Yunsong Li, Xinbo Gao, Jing Zhang
TL;DR
Directly applying pretrained SAM to IRSTD is limited by the domain gap between natural and infrared images and the weak boundary cues of small targets. IRSAM modifies SAM with WPMD in the encoder and GAD in the decoder, and experiments on three public datasets report superiority over state-of-the-art methods.
Problem
Pretrained SAM performs unsatisfactorily for IRSTD because natural and infrared images have a notable domain gap, while small targets provide limited boundary and structural cues.
Method
IRSAM adapts SAM’s encoder-decoder architecture with WPMD for edge preservation and noise suppression, plus GAD for multi-granularity feature fusion.
Results
IRSAM demonstrates superiority over state-of-the-art methods on NUAA-SIRST, IRSTD-1K, and NUDT-SIRST in objective metrics and subjective evaluation.
Takeaways & Limitations
The results support refining a natural-image segmentation foundation model for IRSTD through task-specific encoder and decoder designs.
Abstract
from arXiv · showhide
The recent Segment Anything Model (SAM) is a significant advancement in natural image segmentation, exhibiting potent zero-shot performance suitable for various downstream image segmentation tasks. However, directly utilizing the pretrained SAM for Infrared Small Target Detection (IRSTD) task falls short in achieving satisfying performance due to a notable domain gap between natural and infrared images. Unlike a visible light camera, a thermal imager reveals an object's temperature distribution by capturing infrared radiation. Small targets often show a subtle temperature transition at the object's boundaries. To address this issue, we propose the IRSAM model for IRSTD, which improves SAM's encoder-decoder architecture to learn better feature representation of infrared small objects. Specifically, we design a Perona-Malik diffusion (PMD)-based block and incorporate it into multiple levels of SAM's encoder to help it capture essential structural features while suppressing noise. Additionally, we devise a Granularity-Aware Decoder (GAD) to fuse the multi-granularity feature from the encoder to capture structural information that may be lost in long-distance modeling. Extensive experiments on the public datasets, including NUAA-SIRST, NUDT-SIRST, and IRSTD-1K, validate the design choice of IRSAM and its significant superiority over representative state-of-the-art methods. The source code are available at: github.com/IPIC-Lab/IRSAM.
1 Introduction
IRSTD benefits from infrared imaging in difficult visible-light conditions, but applying SAM directly is hindered by the natural–infrared domain gap and small targets’ limited structural cues. IRSAM adapts SAM with WPMD and GAD to improve infrared feature representation, suppress noise, and preserve target structure.
- Infrared imaging captures thermal radiation and can provide richer target information than visible light under fog or rain.
- Limited training data and specialized architectures motivate testing whether a large-scale natural-image model can initialize IRSTD effectively.
- SAM’s zero-shot segmentation capability does not directly transfer well to IRSTD because infrared and natural images differ substantially.
- IRSAM redesigns SAM’s encoder-decoder architecture for infrared small-object representation, using lightweight Mobile-SAM as its base.
- IRSAM is presented as the first application of SAM to IRSTD and is reported to outperform vanilla SAM and SOTA methods on challenging benchmarks.
- WPMD enhances edge-feature preservation while suppressing infrared noise, and GAD fuses multi-granularity features to represent varied target shapes and sizes.
2 Related Work
Prior IRSTD research includes handcrafted, deep-learning, transformer-based, and SAM-derived methods, each addressing detection, fusion, shape, enhancement, or efficiency. IRSAM targets remaining limitations from scenario-specific generalization, prompt dependence, and the natural–infrared domain gap.
- Handcrafted IRSTD methods are tailored to specific scenarios, limiting generalization to challenging situations.
- Deep learning methods address IRSTD through strategies including adversarial learning, asymmetric context modulation, nested U-Nets, random-connection attention, edge modeling, and image enhancement.
- SAM applies prompt-driven segmentation using a transformer-based architecture, while Mobile-SAM and Fast SAM reduce its computational complexity.
- Existing SAM-based approaches may remain prompt-dependent, restricting applicability and efficiency for IRSTD.
- Perona-Malik diffusion adapts diffusion to local image features, supporting noise removal while preserving edge and texture information.
3 Methodology
IRSAM adapts SAM for infrared small-target detection through WPMD-enhanced encoding and granularity-aware decoding. WPMD preserves structural features while suppressing noise, while GAD combines encoder features across granularities to produce refined masks.
- Architecture: IRSAM uses a pre-trained ViT-Tiny encoder with WPMD blocks and a decoder that fuses features from different granularities.The decoder also uses output tokens to interact with image features and generate the target mask.
- Wavelet transform-based PMD Block: WPMD replaces the PMD gradient term with wavelet high-frequency components and is inserted at multiple encoder levels to preserve structure and suppress noise.A convolutional layer maps the resulting structural features to the encoder feature dimension.
- Wavelet transform-based PMD Block: Perona-Malik diffusion smooths homogeneous regions strongly while applying weak diffusion near edges, thereby removing noise and retaining edge information.The diffusion coefficient is g(|∇u|) = 1/(1 + |∇u|^2/k^2), with k controlling diffusion strength.
- Granularity-Aware Decoder: GAD performs bidirectional cross-attention between encoder image embeddings and mask and edge tokens, integrating global context into the updated tokens.The decoder introduces a learnable edge token alongside the mask tokens.
- Granularity-Aware Decoder: GAD fuses shallow and deep encoder features, applies dynamic convolution kernels to fine and coarse features, and produces edge and mask predictions.The predicted edge is used to improve mask shape and size, yielding high-quality infrared small-target masks.
- Loss Functions: IRSAM is trained with a weighted sum of Dice loss for mask differences and BCE loss for edge prediction.The balancing hyper-parameter λ is empirically set to 10.
4 Experiments
Experiments across three infrared small-target datasets show that IRSAM outperforms representative methods in objective metrics and visual evaluations. Ablations and visualizations associate its gains with WPMD’s structural-feature preservation and noise suppression and GAD’s multi-granularity fusion.
- Experimental Setup: 427 real, 1,000 real, and 1,327 synthetic infrared images are evaluated across NUAA-SIRST, IRSTD-1k, and NUDT-SIRST, respectively.All images are resized to 512×512 and split into 50% training, 30% validation, and 20% test sets.
- Quantitative Results: IRSAM outperforms state-of-the-art methods on every reported evaluation metric across NUAA-SIRST, IRSTD-1k, and NUDT-SIRST.The reported metrics include IoU, nIoU, Pd, and Fa.
- Quantitative Results: IRSAM’s AUC is notably larger than those of traditional and CNN-based methods on NUAA-SIRST.The comparison is presented through ROC curves in Fig. 6.
- Quantitative Results: IRSAM outperforms other fine-tuned SAM models across IoU, nIoU, Pd, and Fa while using a lightweight ViT-Tiny backbone.The comparison also reports Flops(G) and Params(M), and the competing SAM encoders remain frozen during adaptation.
- Visual Results: Visual results show more accurate and complete masks for complex aircraft gaps, adjacent objects, and long narrow objects.The 3D views also show effective segmentation of multiple nearby objects.
- Ablation Studies: Removing WPMD reduces IoU and nIoU and substantially increases Fa, while four WPMD blocks produce the best ablation performance.Feature-map and edge-map comparisons report stronger target features, preserved internal details, and reduced noise with WPMD.
5 Conclusion
IRSAM adapts SAM for infrared small-target detection by adding WPMD to improve encoder edge-feature extraction and GAD to integrate multi-granularity decoder features. Experiments on three public datasets demonstrate superiority over state-of-the-art methods in objective metrics and subjective evaluation.
- 5 Conclusion: IRSAM leverages SAM’s generic segmentation capability and adds WPMD and GAD for infrared small-target detection.WPMD improves encoder edge-feature extraction, while GAD integrates multi-granularity features for shape representation.