Source-linked AI summary
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
Junde Wu, Wei Ji, Yuanpei Liu, Huazhu Fu, Min Xu, Yanwu Xu, Yueming Jin
TL;DR
SAM underperforms on medical images because it lacks medical-specific knowledge, motivating adaptation without full fine-tuning. Med-SA uses SD-Trans for 2D-to-3D adaptation and HyP-Adpt for prompt-conditioned adaptation, achieving strong results across 17 tasks while updating only 2% of SAM parameters.
Problem
SAM underperforms in medical image segmentation because it lacks medical-specific knowledge, including handling of 3D modalities and medical visual characteristics.
Method
Med-SA uses parameter-efficient adaptation with SD-Trans for 2D-to-3D processing and HyP-Adpt for prompt-conditioned adaptation.
Results
Across 17 medical image segmentation tasks and multiple modalities, Med-SA outperforms SAM, fully fine-tuned MedSAM, and several SOTA methods while updating only 2% of SAM parameters.
Takeaways & Limitations
Med-SA provides a parameter-efficient extension of SAM for medical segmentation across diverse modalities and supports high-dimensional 3D data.
Abstract
from arXiv · showhide
The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation due to its impressive capabilities in various segmentation tasks and its prompt-based interface. However, recent studies and individual experiments have shown that SAM underperforms in medical image segmentation, since the lack of the medical specific knowledge. This raises the question of how to enhance SAM's segmentation capability for medical images. In this paper, instead of fine-tuning the SAM model, we propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model using a light yet effective adaptation technique. In Med-SA, we propose Space-Depth Transpose (SD-Trans) to adapt 2D SAM to 3D medical images and Hyper-Prompting Adapter (HyP-Adpt) to achieve prompt-conditioned adaptation. We conduct comprehensive evaluation experiments on 17 medical image segmentation tasks across various image modalities. Med-SA outperforms several state-of-the-art (SOTA) medical image segmentation methods, while updating only 2\% of the parameters. Our code is released at https://github.com/KidsWithTokens/Medical-SAM-Adapter.
Introduction
SAM performs strongly on natural images but underperforms in medical segmentation because it lacks medical-specific knowledge. Med-SA addresses this gap with parameter-efficient adaptation, supporting 2D-to-3D processing and prompt-conditioned segmentation while updating only 2% of SAM parameters.
- SAM underperforms on medical images because it lacks medical-specific knowledge and faces low contrast, ambiguous boundaries, and tiny lesions.
- Full fine-tuning of SAM on medical data is costly in computation and memory, motivating parameter-efficient adaptation.
- Med-SA inserts lightweight Adapter modules into SAM and updates only a small subset of additional parameters.
- SD-Trans adapts 2D SAM to 3D medical images, while HyP-Adpt incorporates visual prompts into adaptation.
- 2% of SAM parameters are updated while Med-SA is evaluated across 17 medical image segmentation tasks and outperforms SAM, MedSAM, and several SOTA methods.
Related Work
Related work establishes interactive segmentation as a clinically relevant area and identifies parameter-efficient adaptation as a way to specialize large pretrained models with fewer updated parameters.
- Interactive segmentation has received limited attention in medical imaging despite its critical role in clinical practice.
- Medical images may require interactive segmentation of multiple targets, including vessels, optic disc, optic cup, and macula, depending on clinical use.
- PEFT freezes most pretrained-model parameters and often updates less than 5% of the total, enabling efficient learning with faster updates.
- Prior studies report that PEFT can avoid catastrophic forgetting and improve out-of-domain generalization, especially in low-data regimes.
- Adaptation has been adopted for downstream computer-vision tasks, supporting its use for transferring SAM to the medical domain.
Method
Med-SA adapts SAM for medical segmentation by keeping the pretrained architecture largely fixed while adding lightweight adapters for 2D/3D inputs and prompt-conditioned decoding. SD-Trans captures depth correlations in 3D data, while HyP-Adpt generates adapter weights from visual prompts.
- SAM architecture: SAM uses an image encoder, sparse prompt encoder, and mask decoder with two-way cross-attention between prompt and image embeddings.The decoder upsamples image embeddings and predicts target masks through a dynamic linear classifier.
- Adapter-based adaptation: Med-SA freezes pretrained SAM parameters and inserts bottleneck Adapters that compress and expand embeddings through down-projection, ReLU, and up-projection.The approach updates only additional adapter parameters rather than fully fine-tuning SAM.
- Integrated design: The architecture uses standard Adapters for 2D encoder processing, SD-Trans-enhanced encoders for 3D processing, and HyP-Adpt in the decoder for prompt incorporation.Training uses click and bounding-box prompts, with random initialization followed by iterative clicks placed in erroneous prediction regions.
- HyP-Adpt: HyP-Adpt incorporates prompts into decoder adaptation by generating weight maps from prompt embeddings and applying them to reduced adapter embeddings.Prompt information includes click location, click attribution, and bounding-box location; three hyper-prompting layers use separate MLP projections.
- SD-Trans: SD-Trans adapts SAM to 3D images by splitting attention into space and depth branches, transposing embeddings so shared attention learns volumetric correlations.The depth-branch output is transposed back and added to the space-branch result.
Experiments
Experiments evaluate Med-SA across multiple datasets, modalities, prompts, baselines, and ablations. Med-SA achieves strong cross-task performance while using substantially fewer trainable parameters, with SD-Trans and HyP-Adpt contributing to improvements.
- Datasets and setup: The experiments cover abdominal multi-organ, fundus, brain MRI, ultrasound, and dermoscopic segmentation across five datasets and multiple modalities.BTCV evaluates twelve abdominal anatomies; other tasks include optic disc/cup, brain tumor, thyroid nodule, and melanoma or nevus segmentation.
- Abdominal multi-organ segmentation: 89.8% Dice with BBox 0.75 surpasses Swin-UNetr by 2.9% on BTCV, while Med-SA updates 13M rather than 138M parameters.One-point Med-SA achieves SOTA performance for all 12 organs, and finer-grained prompts further improve results.
- Abdominal multi-organ segmentation: 13M versus 636M trainable parameters lets Med-SA outperform fully fine-tuned MedSAM across all prompt variations.This corresponds to updating only 2% of MedSAM’s total trainable parameters.
- Prompt comparison: Three-point prompts slightly outperform one-point prompts, while BBox 0.75 is often comparable or better and BBox 0.5 performs subpar.The results show similar prompt-response behavior across SAM, MedSAM, and Med-SA.
- Evaluation: Table 2 compares Med-SA with SAM, MedSAM, general medical segmentation methods, and task-specific methods using Dice, IoU, and HD95.Performance is omitted when an algorithm fails on over 70% of samples.
- Multi-modality comparison: Med-SA achieves SOTA performance across the evaluated segmentation tasks and modalities, surpassing Swin-UNetr on BraTs by 2.1% Dice and 1.86 HD95 with less than 10% of its trainable parameters.The comparison attributes this result to Med-SA’s adaptability to 3D images.
- Ablation study: Ablations show that SD-Trans improves performance over processing 3D images as independent 2D slices, while HyP-Adpt provides further enhancement.The ablation compares against a SAM-plus-Adapter baseline on BTCV and BrainTumor benchmarks.
Conclusion
The paper extends SAM for medical image segmentation with Med-SA, combining parameter-efficient adaptation with SD-Trans and HyP-Adpt. It reports SOTA performance across 17 tasks and five image modalities.
- Conclusion: Med-SA extends SAM for medical image segmentation using parameter-efficient adaptation with SD-Trans and HyP-Adpt.SD-Trans addresses high-dimensional 3D data, while HyP-Adpt supports prompt-conditioned adaptation.
- Conclusion: Med-SA achieves SOTA performance across 17 medical image segmentation tasks spanning five image modalities.