DeSAM: Decoupled Segment Anything Model for Generalizable Medical Image Segmentation

1 Introduction

• Single-source domain generalization uses one training domain to generalize to unseen data, but augmentation methods can require expert-designed functions or complex adversarial training.Unsupervised domain adaptation and multi-source generalization also rely on target-domain or multiple-source training data.Source: §1 · p. 2
• Earlier large-dataset pretrained models showed promise for medical segmentation, but their small capacity did not effectively improve cross-domain generalizability.SAM and other vision foundation models motivate adapting pretrained models for this setting.Source: §1 · p. 2
• SAM's automatic masks remain sensitive to poor prompts because cross-attention couples prompt tokens with image embeddings, making output masks prompt-dependent.Wrong prompts include points outside the mask or boxes substantially larger than it.Source: §1 · p. 2
• DeSAM splits automatic SAM decoding into prompt-relevant IoU regression and prompt-decoupled mask learning to reduce degradation from wrong prompts.PRIM predicts IoU scores and produces mask embeddings; PDMM fuses them with image embeddings to generate masks.Source: §1 · pp. 2–3

2 Related Work

• Single-source generalization methods include bias-field, random-convolution, style-mixing, worst-case, and causality-inspired augmentations; some require expert-designed transforms or complex adversarial training.DeSAM is described as avoiding a complex data-augmentation pipeline.Source: §2 · p. 3
• Medical SAM adaptations have fine-tuned encoders or mask decoders for specific tasks, while DeSAM targets automatic-segmentation degradation with PRIM and PDMM.The related-work discussion also highlights multi-scale feature fusion for context capture.Source: Segment Anything Model · p. 3

Segment Anything Model

• The overview assigns PRIM the image-and-prompt embedding inputs and mask-embedding/IoU outputs, while PDMM fuses mask and image embeddings to produce the mask.The overview also states that the prompt encoder is frozen during training.Source: Figure 1 · p. 4

3.1 Architecture

• DeSAM's architecture retains SAM's encoders and adds PRIM and PDMM as its two main components.The image and prompt encoders are inherited from SAM.Source: §3.1 · p. 4

Prompt-Relevant IoU Module (PRIM).

• PRIM keeps a cross-attention transformer and IoU prediction head, using the transformer to extract mask embeddings instead of directly predicting the mask.This removes the mask prediction head from the SAM-like decoder structure.Source: §3.1 · p. 4

Prompt-Decoupled Mask Module (PDMM).

• PDMM extracts intermediate image features at layers 8, 16, and 24, refines and upsamples them, and fuses scales through skip connections.It merges PRIM's mask embeddings with PDMM bottleneck embeddings, leveraging pretrained SAM mask-decoder weights and supporting gradient flow between modules.Source: Prompt-Decoupled Mask Module (PDMM) · p. 5

3.2 Training strategies

• Training loads SAM ViT-H weights, freezes both encoders, and fine-tunes PRIM and PDMM; grid-point mode supervises masks with Dice and cross-entropy losses and IoU with MSE.Whole-box mode supervises mask generation only because the ground-truth mask must lie inside the box.Source: §3.2 · p. 5
• The decoder ablation compares PDMM alone, PDMM plus PRIM without an IoU head, and without mask-embedding fusion against the full DeSAM design.The quantitative ablation table identifies the IoU prediction head as IPH and mask embedding fusion as MEF.Source: Figure 2 · p. 6, Table 1 · p. 6

4.1 Dataset and implementation details

• Evaluation covers cross-modality abdominal and cross-site prostate segmentation, using one domain for training and in-domain validation and other domains for testing.The training/validation split is 9:1, and Dice score measures mask quality.Source: §4.1 · p. 6

4.2 Ablation studies

• Adding the IoU prediction head and mask-embedding fusion progressively improves performance over the PDMM-only variant.The ablation compares PDMM-only, PDMM and PRIM without IoU prediction, and PDMM and PRIM without mask-embedding fusion.Source: §4.2 · p. 8
• Performance does not degrade as grid-point counts increase, supporting DeSAM's robustness to poor prompts in grid-point mode.Fig. 3 reports quantitative results for different numbers of points.Source: Figure 3 · p. 7, §4.2 · p. 8

4.3 Comparison with state-of-the-art methods

• 79.02% overall Dice, with DeSAM-P exceeding CSDG by 8.96% on overall Dice across the abdominal and prostate datasets.On abdominal CT and MRI, DeSAM-P scores 86.68% and 80.05% Dice, respectively.Source: Table 2 · p. 7, §4.3 · p. 8
• In prostate visual comparisons, DeSAM has no background false positives and produces boundaries close to ground truth.Fig. 4 compares methods for cross-site prostate and cross-modality abdominal segmentation.Source: §4.3 · p. 8, Figure 4 · p. 8

• DeSAM decouples mask generation from prompts and leverages SAM's pre-trained weights for single-source medical image segmentation generalization.The architecture aims to learn prompt-invariant features from robust image embeddings.Source: §5 · p. 9
• Fusing image embeddings at multiple scales is part of DeSAM's approach to handling unseen distribution changes.Source: §5 · p. 9
• On two public datasets, DeSAM outperforms other state-of-the-art methods.Source: §5 · p. 9