SAM-Med2D

Junlong Cheng; Jin Ye; Zhongying Deng; Jianpin Chen; Tianbin Li; Haoyu Wang; Yanzhou Su; Ziyan Huang; Jilong Chen; Lei Jiang; Hui Sun; Junjun He; Shaoting Zhang; Min Zhu; Yu Qiao

Source-linked AI summary

SAM-Med2D

Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, Hui Sun, Junjun He, Shaoting Zhang, Min Zhu, Yu Qiao

arXiv:2308.16184v1Aug 30, 2023cs.CV

TL;DR

SAM performs unreliably on diverse medical images because of the domain gap between natural and medical imagery. SAM-Med2D bridges this gap with a large medical segmentation dataset, comprehensive prompt and model fine-tuning, and broad evaluation, achieving superior performance and generalization compared with SAM.

Problem
The natural-image and medical-image domain gap causes SAM to generalize unreliably across multimodal, multi-object medical datasets.
Method
The paper builds a 4.6M-image, 19.7M-mask dataset and fine-tunes SAM using point, bounding-box, and mask prompts with adapter-based encoder adaptation.
Results
SAM-Med2D demonstrates superior performance and generalization compared with SAM across challenging medical structures, lesions, boundaries, prompt modes, and evaluation datasets.
Takeaways & Limitations
SAM-Med2D provides broad medical segmentation capabilities across different prompt modes, modalities, organs, and anatomical structures.

Abstract

from arXiv · show

The Segment Anything Model (SAM) represents a state-of-the-art research advancement in natural image segmentation, achieving impressive results with input prompts such as points and bounding boxes. However, our evaluation and recent research indicate that directly applying the pretrained SAM to medical image segmentation does not yield satisfactory performance. This limitation primarily arises from significant domain gap between natural images and medical images. To bridge this gap, we introduce SAM-Med2D, the most comprehensive studies on applying SAM to medical 2D images. Specifically, we first collect and curate approximately 4.6M images and 19.7M masks from public and private datasets, constructing a large-scale medical image segmentation dataset encompassing various modalities and objects. Then, we comprehensively fine-tune SAM on this dataset and turn it into SAM-Med2D. Unlike previous methods that only adopt bounding box or point prompts as interactive segmentation approach, we adapt SAM to medical image segmentation through more comprehensive prompts involving bounding boxes, points, and masks. We additionally fine-tune the encoder and decoder of the original SAM to obtain a well-performed SAM-Med2D, leading to the most comprehensive fine-tuning strategies to date. Finally, we conducted a comprehensive evaluation and analysis to investigate the performance of SAM-Med2D in medical image segmentation across various modalities, anatomical structures, and organs. Concurrently, we validated the generalization capability of SAM-Med2D on 9 datasets from MICCAI 2023 challenge. Overall, our approach demonstrated significantly superior performance and generalization capability compared to SAM.

1 Introduction

The paper motivates adapting SAM to medical images because domain differences and limited, heterogeneous annotations undermine generalization. SAM-Med2D addresses this through large-scale medical data, broader prompt support, adapter-based tuning, and comprehensive evaluation.

Medical images span diverse modalities, structures, and annotation qualities, limiting existing methods’ generalizability across clinical scenarios.
4.6M images and 19.7M masks form a large medical segmentation dataset covering multiple modalities and objects.
Bounding boxes, points, and masks serve different medical segmentation scenarios, motivating fine-tuning across all three prompt modes.
Adapter-based encoder tuning incorporates medical-domain information without retraining the entire image encoder.
Evaluation covers imaging diversity, alternative fine-tuning strategies, and generalization on 9 MICCAI2023 datasets.
SAM-Med2D outperforms SAM on complex structures, lesions, unclear boundaries, and multiple prompt modes.

2 Related Work

Related work centers on large-scale vision models and medical adaptations of SAM. Prior medical fine-tuning is often dataset-specific, while this work emphasizes broader prompt coverage and analysis across modalities and anatomy.

Large-scale Vision Models: SAM is a general-purpose segmentation model with zero-shot interactive or automatic segmentation capabilities on natural images.
Fine-tuned SAM in Medical Image Analysis: Medical applications report performance declines because natural-image and medical-image domains differ substantially.
Fine-tuned SAM in Medical Image Analysis: Existing medical SAM studies primarily fine-tune on specific datasets, limiting coverage of diverse segmentation tasks.
Fine-tuned SAM in Medical Image Analysis: This work supports bounding-box, point, and mask prompts and analyzes performance across imaging modalities, anatomical structures, and organs.

3 Methods

The method transfers SAM to medical 2D segmentation using a large, diverse dataset and targeted architectural adaptation. It preserves the SAM framework while supporting medical prompts and broad evaluation data.

Dataset: Training data comprise approximately 4.6M images and 19.7M masks collected from public and private datasets.
Dataset: The dataset covers 10 imaging modalities, five anatomical categories, lesions, and 31 consolidated organs.
Preprocessing: 3D datasets are normalized, sliced along three axes, and filtered to avoid extremely blurry resized targets.
Evaluation data: The data split assigns 80% to training and 20% to testing, while 9 MICCAI2023 datasets are reserved for generalization validation.
Architecture: The pipeline uses a frozen image encoder with learnable adapters, fine-tuned prompt encoding for points, boxes, and masks, and an updated mask decoder.
Architecture: SAM-Med2D retains image, prompt, and mask-decoder components while extending SAM through medical-domain fine-tuning.

A. Adapting Image Encoder

The image encoder is adapted efficiently by freezing pretrained parameters and adding adapters to each Transformer block. These adapters learn medical-domain information while reducing the cost of globally updating the encoder.

The original image encoder remains frozen during fine-tuning, with one adapter deployed in each Transformer block.
Adapters adapt image features along both channel and spatial dimensions to incorporate medical-domain knowledge.

B. Prompt Encoder and Mask Decoder

SAM-Med2D retains point, bounding-box, and mask prompting while fine-tuning the mask decoder to produce ambiguity-aware predictions.

SAM’s prompt encoder supports point, bounding-box, mask, and text prompts, but fine-tuning excludes text prompts because large-scale medical image-text pretraining is unavailable.
The medical adaptation retains the three non-text prompt modes rather than fine-tuning only a single prompt type.
The mask decoder structure remains unchanged, while its parameters are updated during training.
Each prompt predicts three masks by default, and backpropagation uses the prediction with the highest IoU against the ground truth.

C. Fine-tuning Strategy

SAM-Med2D is fine-tuned through simulated interactive segmentation with successive prompts derived from segmentation errors, then evaluated across interaction modes and medical-image modalities.

C. Fine-tuning Strategy: SAM-Med2D is trained for 9 iterations per batch using simulated interactive segmentation.
C. Fine-tuning Strategy: The first iteration randomly selects a foreground point or perturbed maximum-enclosing bounding box as the sparse prompt with equal probability.
C. Fine-tuning Strategy: From the second iteration, 1, 3, 5, or 9 points are sampled from prediction-error regions to support single-point and multi-point interaction.
C. Fine-tuning Strategy: After the first iteration, only the mask decoder is updated; the first iteration updates the Adapter layer, prompt encoder, and mask decoder simultaneously.
C. Fine-tuning Strategy: The evaluation is intended to characterize performance, applicability, and modality-specific strengths and challenges for future medical image segmentation research.
C. Fine-tuning Strategy: Evaluation compares SAM-Med2D with SAM using bounding-box and point interactions across 10 medical-image modalities.

4 Experiments and Evaluation

Experiments compare SAM, FT-SAM, and SAM-Med2D across prompting modes, resolutions, anatomical structures, modalities, organs, and qualitative cases. SAM-Med2D generally improves segmentation and generalization, while performance remains relatively weaker for head-and-neck regions.

Overall performance: 79.30% Dice was achieved by SAM-Med2D with bounding-box prompts, a 17.67% improvement over SAM.FT-SAM improved Dice by 11.93% over SAM in the same prompting mode.
Anatomical structures: SAM-Med2D outperformed other methods across all evaluated anatomical structures, with fine-tuning especially improving point-prompt segmentation.The authors attribute this to learning positional relationships of points within target regions from large-scale medical data.
Anatomical structures: Head-and-neck segmentation remained relatively subpar across models and resolutions, indicating a need for further improvement.The authors associate this region with smaller targets and less distinct boundaries.
Point prompts: With one point interaction, SAM-Med2D outperformed other methods using five points, indicating more efficient point-based interaction for medical segmentation.Segmentation performance generally improved with additional iterations, but SAM’s multi-point performance declined in histopathology and microscopy at 1024×1024.
Major organs: SAM-Med2D achieved higher results than FT-SAM in 24 organs, with a maximum Dice difference of 6.95%.The gap between five-point and bounding-box prompts was relatively small, and point prompts were more effective for some skeletal regions.
Generalization: Across nine challenge datasets, SAM-Med2D generalized better under both point and bounding-box prompts, whereas SAM generalized well only with bounding boxes.Without adapter parameters during inference, SAM-Med2D with one point reached 83.41% versus SAM’s 85.35% with bounding boxes.
Qualitative evaluation: Qualitative comparisons showed clearer, more ground-truth-aligned boundaries and fewer interactive operations for SAM-Med2D than SAM.SAM-Med2D was especially better at describing target regions with equal numbers of point prompts and at segmenting difficult regions.

5 Discussion and Conclusion

SAM-Med2D improves medical image segmentation across organs, modalities, and prompt modes through large-scale fine-tuning, while retaining limitations on difficult objects and broad “everything” segmentation.

Discussion: 17.67% improvement was achieved by fully fine-tuning SAM-Med2D, compared with 11.93% from fine-tuning only the mask decoder in Bbox prompt mode.At equal resolution, the fully fine-tuned model also substantially outperformed SAM in the 1 pt prompt mode.
Discussion: SAM-Med2D outperformed other methods overall across anatomical regions and achieved better results on 24 of more than 30 major organs.Its maximum improvement over FT-SAM was 6.95%, while performance varied across modalities and regions.
Discussion: Adding more point prompts improved segmentation beyond the Bbox prompt, and SAM-Med2D could generate desired masks quickly in point-prompt mode.The paper reports that point-prompt performance could surpass other methods using Bbox prompts.
Limitations: Further optimization is needed for complex shapes, unclear boundaries, small or low-contrast objects, and comprehensive medical-domain “everything” segmentation.The authors also identify missing datasets for natural-language interaction and a remaining scale and resolution gap relative to SAM’s training data.
Conclusion: The study concludes that large-scale medical-image fine-tuning adapts SAM to the medical domain and improves performance and generalization.The authors state that code and pretrained models will be made available to researchers.