Source-linked AI summary
SAM on Medical Images: A Comprehensive Study on Three Prompt Modes
Dongjie Cheng, Ziyuan Qin, Zekun Jiang, Shaoting Zhang, Qicheng Lao, Kang Li
TL;DR
Medical image segmentation often requires specialized models and substantial annotation because of domain and modality differences. This study evaluates SAM across more than 12 public medical datasets and prompt modes, finding that proper prompts improve performance while results vary across datasets and remain below supervised models overall.
Problem
The paper investigates whether SAM can generalize to medical image segmentation across domains and modalities where specialized models and substantial annotation are commonly required.
Method
The study evaluates SAM directly on test sets from more than 12 public medical datasets spanning multiple modalities, compares prompt modes, and benchmarks against supervised SOTA results.
Results
Across datasets, proper prompts—especially bounding boxes—substantially improve SAM performance, while box-size perturbations affect accuracy and predicted mask quality varies widely.
Takeaways & Limitations
Bounding-box prompting is identified as a particularly suitable way to leverage SAM for medical image segmentation, although performance is not uniformly competitive with supervised methods.
Abstract
from arXiv · showhide
The Segment Anything Model (SAM) made an eye-catching debut recently and inspired many researchers to explore its potential and limitation in terms of zero-shot generalization capability. As the first promptable foundation model for segmentation tasks, it was trained on a large dataset with an unprecedented number of images and annotations. This large-scale dataset and its promptable nature endow the model with strong zero-shot generalization. Although the SAM has shown competitive performance on several datasets, we still want to investigate its zero-shot generalization on medical images. As we know, the acquisition of medical image annotation usually requires a lot of effort from professional practitioners. Therefore, if there exists a foundation model that can give high-quality mask prediction simply based on a few point prompts, this model will undoubtedly become the game changer for medical image analysis. To evaluate whether SAM has the potential to become the foundation model for medical image segmentation tasks, we collected more than 12 public medical image datasets that cover various organs and modalities. We also explore what kind of prompt can lead to the best zero-shot performance with different modalities. Furthermore, we find that a pattern shows that the perturbation of the box size will significantly change the prediction accuracy. Finally, Extensive experiments show that the predicted mask quality varied a lot among different datasets. And providing proper prompts, such as bounding boxes, to the SAM will significantly increase its performance.
I. INTRODUCTION
This study investigates whether SAM can generalize to medical image segmentation, where domain and modality differences make specialized models and substantial training data common. It evaluates prompting strategies and finds that proper prompts improve zero-shot performance, although results remain below supervised models overall.
- The study asks whether SAM can serve as a foundation model for medical image segmentation despite domain and modality differences.
- Experiments compare auto-prompt, box-prompt, and point-prompt modes across medical image segmentation tasks.
- Box prompts achieve higher Dice accuracy than the other evaluated prompt modes, while increasing point prompts approaches box-prompt performance.
- Adding jitter to bounding boxes significantly affects prediction accuracy, indicating sensitivity to prompt-box perturbations.
- SAM can show competitive zero-shot generalization with proper prompts, but its overall performance remains below supervised trained models.
II. RELATED WORKS
SAM-related research rapidly expanded after its publication, including work on medical imaging and prompt-based segmentation. Figure 1 presents segmentation results under different prompt modes.
- SAM quickly attracted research attention across AI, computer vision, image processing, and cross-disciplinary applications.
- Figure 1 presents segmentation results obtained with different prompt modes.
A. About Segment Anything
Foundation-model research expanded from language and cross-modal tasks toward computer vision, motivating segmentation models such as SAM. SAM is presented as a foundation model for zero-shot processing and downstream-task development.
- The success of large language models motivated researchers to investigate whether comparable foundation models could emerge in computer vision.
- SAM is introduced as a major computer-vision effort centered on segmenting anything.
- SAM opened foundation-model approaches for zero-shot processing and secondary development across downstream tasks.
B. SAM in Medical Imaging
SAM has enabled zero-shot processing and secondary development across downstream tasks, including medical imaging. Related medical-imaging efforts have addressed segmentation in several specific organs and modalities.
- SAM supports zero-shot processing and secondary development for downstream tasks, including medical image processing.
- Seven cited medical-imaging works focused on segmentation in digital pathology, liver tumors, abdominal CT, brain MRI, brain tumors, polyps, or ophthalmology.
C. Other SAM Works
SAM has been applied beyond medical segmentation to broader natural-image evaluation and diverse downstream tasks.
- Researchers evaluated SAM on broader natural-image datasets and also adapted it for tasks including inpainting, 3D reconstruction, style transformation, counting, and 3D segmentation.
A. Dataset
The study evaluates SAM across 12 public datasets spanning multiple medical modalities and uses test-set segmentation and Dice computation for comparison with supervised-learning results.
- The study collected 12 public datasets covering CT, X-ray, MRI, endoscopy, ultrasound, and OCT.
- The datasets include breast ultrasound, polyp detection, foot-ulcer segmentation, chest X-ray, MRI, thyroid ultrasound, and liver or liver-tumor segmentation tasks.
- SAM was applied directly to dataset test sets, with segmentation results and Dice scores compared against supervised-learning state-of-the-art results from related studies.
B. Applying SAM with different prompt modes
The study tests auto-prompt, point-prompt, and box-prompt modes for zero-shot medical image segmentation. Point and box prompts are constructed from curated object information, while box prompts also receive scale jitter.
- SAM is evaluated in three modes: auto-prompt, point-prompt, and box-prompt.
- Auto-prompt uses a regular grid of points to predict a set of masks for each point prompt.
- Box prompts use ground-truth-derived object boxes with different jitter scales, while single-point prompts use the center of each curated box.
- Across prompt settings, SAM's zero-shot prediction accuracy remains below commonly used fully supervised segmentation models.
- Real-world use still requires methods for generating high-quality prompts because ground-truth-derived masks are unavailable.
C. How to properly prompt the SAM with boxes and points
Prediction accuracy decreases as box jitter increases, suggesting that SAM is sensitive to box size and reliability. Producing reliable boxes of suitable size remains an open application issue.
- Prediction accuracy decreases as the bounding-box jitter scale increases across most evaluated datasets.
- The authors conjecture that larger perturbations include more background or unrelated areas within the bounding box.
- The observed pattern indicates that SAM's prediction quality is sensitive to box size and reliability.
- Producing reliable and suitably sized box prompts is identified as a possible research topic for applying SAM across domains.
IV. RESULTS
SAM’s medical-image performance varies substantially across datasets and prompt modes. Box prompts with zero jitter achieve the strongest overall results, while jittering box coordinates significantly changes prediction accuracy.
- Box prompts with zero jitter achieve the highest average Dice across the evaluated medical-image datasets and occasionally approach SOTA performance.
- Increasing box jitter significantly changes prediction-mask accuracy across datasets.
- SAM’s zero-shot results are generally below SOTA, although several datasets exceed SOTA performance.