Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation

• SAM underperforms on medical images because it lacks medical-specific knowledge and faces low contrast, ambiguous boundaries, and tiny lesions.Source: Introduction · p. 1
• Full fine-tuning of SAM on medical data is costly in computation and memory, motivating parameter-efficient adaptation.Source: Introduction · p. 1
• Med-SA inserts lightweight Adapter modules into SAM and updates only a small subset of additional parameters.Source: Introduction · p. 1
• SD-Trans adapts 2D SAM to 3D medical images, while HyP-Adpt incorporates visual prompts into adaptation.Source: Introduction · pp. 1–2
• 2% of SAM parameters are updated while Med-SA is evaluated across 17 medical image segmentation tasks and outperforms SAM, MedSAM, and several SOTA methods.Source: Introduction · p. 2

• Interactive segmentation has received limited attention in medical imaging despite its critical role in clinical practice.Source: Interactive Segmentation · p. 2
• Medical images may require interactive segmentation of multiple targets, including vessels, optic disc, optic cup, and macula, depending on clinical use.Source: Interactive Segmentation · p. 2
• PEFT freezes most pretrained-model parameters and often updates less than 5% of the total, enabling efficient learning with faster updates.Source: Parameter-Efficient Fine-Turning · p. 2
• Prior studies report that PEFT can avoid catastrophic forgetting and improve out-of-domain generalization, especially in low-data regimes.Source: Parameter-Efficient Fine-Turning · p. 2
• Adaptation has been adopted for downstream computer-vision tasks, supporting its use for transferring SAM to the medical domain.Source: Parameter-Efficient Fine-Turning · p. 2

SAM architecture

• SAM uses an image encoder, sparse prompt encoder, and mask decoder with two-way cross-attention between prompt and image embeddings.The decoder upsamples image embeddings and predicts target masks through a dynamic linear classifier.Source: Preliminary: SAM architecture · p. 2

Adapter-based adaptation

• Med-SA freezes pretrained SAM parameters and inserts bottleneck Adapters that compress and expand embeddings through down-projection, ReLU, and up-projection.The approach updates only additional adapter parameters rather than fully fine-tuning SAM.Source: Med-SA architecture · pp. 2–3

Integrated design

• The architecture uses standard Adapters for 2D encoder processing, SD-Trans-enhanced encoders for 3D processing, and HyP-Adpt in the decoder for prompt incorporation.Training uses click and bounding-box prompts, with random initialization followed by iterative clicks placed in erroneous prediction regions.Source: Figure 1 · p. 3, Training Strategy · pp. 4–5

HyP-Adpt

• HyP-Adpt incorporates prompts into decoder adaptation by generating weight maps from prompt embeddings and applying them to reduced adapter embeddings.Prompt information includes click location, click attribution, and bounding-box location; three hyper-prompting layers use separate MLP projections.Source: Figure 2 · p. 3, HyP-Adpt · p. 4

SD-Trans

• SD-Trans adapts SAM to 3D images by splitting attention into space and depth branches, transposing embeddings so shared attention learns volumetric correlations.The depth-branch output is transposed back and added to the space-branch result.Source: SD-Trans · p. 3

Datasets and setup

• The experiments cover abdominal multi-organ, fundus, brain MRI, ultrasound, and dermoscopic segmentation across five datasets and multiple modalities.BTCV evaluates twelve abdominal anatomies; other tasks include optic disc/cup, brain tumor, thyroid nodule, and melanoma or nevus segmentation.Source: Dataset · pp. 5–6

Abdominal multi-organ segmentation

• 89.8% Dice with BBox 0.75 surpasses Swin-UNetr by 2.9% on BTCV, while Med-SA updates 13M rather than 138M parameters.One-point Med-SA achieves SOTA performance for all 12 organs, and finer-grained prompts further improve results.Source: Comparing with SOTA on Abdominal… · p. 6
• 13M versus 636M trainable parameters lets Med-SA outperform fully fine-tuned MedSAM across all prompt variations.This corresponds to updating only 2% of MedSAM’s total trainable parameters.Source: Comparing with SOTA on Abdominal… · p. 7

Prompt comparison

• Three-point prompts slightly outperform one-point prompts, while BBox 0.75 is often comparable or better and BBox 0.5 performs subpar.The results show similar prompt-response behavior across SAM, MedSAM, and Med-SA.Source: Comparing with SOTA on Abdominal… · p. 7

Evaluation

• Table 2 compares Med-SA with SAM, MedSAM, general medical segmentation methods, and task-specific methods using Dice, IoU, and HD95.Performance is omitted when an algorithm fails on over 70% of samples.Source: Table 2 · p. 6, Comparing with SOTA on Multi-modality… · p. 7

Multi-modality comparison

• Med-SA achieves SOTA performance across the evaluated segmentation tasks and modalities, surpassing Swin-UNetr on BraTs by 2.1% Dice and 1.86 HD95 with less than 10% of its trainable parameters.The comparison attributes this result to Med-SA’s adaptability to 3D images.Source: Comparing with SOTA on Multi-modality… · p. 7

Ablation study

• Ablations show that SD-Trans improves performance over processing 3D images as independent 2D slices, while HyP-Adpt provides further enhancement.The ablation compares against a SAM-plus-Adapter baseline on BTCV and BrainTumor benchmarks.Source: Ablation Study · pp. 7–8

Conclusion

• Med-SA extends SAM for medical image segmentation using parameter-efficient adaptation with SD-Trans and HyP-Adpt.SD-Trans addresses high-dimensional 3D data, while HyP-Adpt supports prompt-conditioned adaptation.Source: Conclusion · p. 8
• Med-SA achieves SOTA performance across 17 medical image segmentation tasks spanning five image modalities.Source: Conclusion · p. 8