Source-linked AI summary
Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation
Peilun Shi, Jianing Qiu, Sai Mu Dalike Abaxi, Hao Wei, Frank P. -W. Lo, Wu Yuan
TL;DR
The paper asks how well the generalist SAM transfers to medical image segmentation, evaluating it quantitatively and qualitatively across nine diverse benchmarks. Using standardized centroid prompts and zero-shot evaluation, it finds inconsistent performance across unseen medical domains, with substantial gaps versus specialized models but marked improvement after limited fine-tuning.
Problem
SAM’s zero-shot segmentation performance on diverse, unseen medical imaging domains was insufficiently characterized by prior studies.
Method
The study evaluates SAM using standardized centroid prompts across nine medical segmentation benchmarks spanning diverse modalities and applications.
Results
SAM’s zero-shot performance varied across medical modalities, was generally below specialized models, and improved substantially after fine-tuning on a small retinal vessel dataset.
Takeaways & Limitations
The findings identify where SAM works and indicate that fine-tuning can improve its performance on domain-specific medical segmentation tasks.
Takeaways & Limitations
The evaluation used only one official prompt type, and fine-tuning experiments covered only retinal vessel datasets.
Abstract
from arXiv · showhide
In this paper, we examine the recent Segment Anything Model (SAM) on medical images, and report both quantitative and qualitative zero-shot segmentation results on nine medical image segmentation benchmarks, covering various imaging modalities, such as optical coherence tomography (OCT), magnetic resonance imaging (MRI), and computed tomography (CT), as well as different applications including dermatology, ophthalmology, and radiology. Those benchmarks are representative and commonly used in model development. Our experimental results indicate that while SAM presents remarkable segmentation performance on images from the general domain, its zero-shot segmentation ability remains restricted for out-of-distribution images, e.g., medical images. In addition, SAM exhibits inconsistent zero-shot segmentation performance across different unseen medical domains. For certain structured targets, e.g., blood vessels, the zero-shot segmentation of SAM completely failed. In contrast, a simple fine-tuning of it with a small amount of data could lead to remarkable improvement of the segmentation quality, showing the great potential and feasibility of using fine-tuned SAM to achieve accurate medical image segmentation for a precision diagnostics. Our study indicates the versatility of generalist vision foundation models on medical imaging, and their great potential to achieve desired performance through fine-turning and eventually address the challenges associated with accessing large and diverse medical datasets in support of clinical diagnostics.
1. Introduction
The paper evaluates SAM’s zero-shot medical segmentation across diverse benchmarks and finds modality-dependent performance, substantial gaps versus specialized models, and strong gains from limited fine-tuning.
- SAM performed better on endoscopic and dermoscopic images than on other medical modalities.The authors conjecture this reflects SAM’s training on large volumes of RGB images, which resemble these modalities.
- SAM failed to segment images containing continuous branching structures such as blood vessels, and also failed on tree-branch images.
- SAM’s medical-image Dice coefficients were generally 0.1–0.4 lower than previous SOTA models, with a worst-case gap of 0.65.
- Fine-tuning SAM on a small amount of retinal vessel data produced impressive segmentation-quality improvements.This preliminary result indicates potential for adapting SAM to medical image segmentation.
2. Segment Anything Model
SAM accepts diverse prompts for image segmentation, and this study derives centroid prompts from ground-truth instances before selecting the highest-scored predicted mask.
- SAM is a generalist vision foundation model that supports diverse input prompts to enhance segmentation quality.
- For evaluation, the study uses each ground-truth instance’s centroid as the prompt and selects the highest-scored result among three candidate masks.
3. Experiments
The experiments apply a standardized zero-shot SAM procedure to medical images: detect contours, derive midpoint prompts, generate candidate masks, and retain the highest-scoring mask.
- The evaluation uses the official ViT-H SAM checkpoint to test zero-shot medical image segmentation across multiple modalities.The observed segmentation outcomes were not consistently satisfactory among modalities.
- For each image, the procedure derives contours from class labels and initializes an image mask.
- A midpoint detector generates prompts for each contour, after which SAM returns masks and scores.
- The algorithm selects the mask with the highest score and appends it to the output segmentation set.
3.1. Medical Image Segmentation Datasets
The study evaluates SAM on nine medical segmentation datasets spanning multiple imaging modalities and organs.
- Nine datasets were used to evaluate SAM on medical image segmentation.The benchmarks include Skin Lesion, Drishiti-GS, RIM-ONE-r3, REFUGE, AMOS, robotic instruments, Chest X-ray, Rat Colon, and AROI.
- The datasets cover OCT, MRI, and CT imaging modalities.
- The evaluated organs include the eyes, colon, spleen, kidney, gallbladder, esophagus, liver, and stomach.
3.2. Evaluation Metrics
The study evaluates segmentation accuracy using Dice and IoU, which quantify overlap between predicted masks and ground-truth masks across samples.
- Dice and IoU quantify overlap between the ground-truth mask and SAM’s predicted mask.Both metrics range from 0 to 1, representing no overlap to perfect overlap.
- The metrics are averaged across multiple samples to obtain overall segmentation accuracy.
- IoU is used alongside Dice because it is more suitable for evaluating worst-case scenarios and supports direct comparison with prior methods.
3.3. Results
The results span successful and failed SAM segmentations across dermoscopic and fundus images, illustrating strong examples alongside domain- and feature-dependent limitations.
- 3.3.1 Dermoscopic Images: SAM’s dermoscopic results were not competitive with existing methods, with failures occurring when lesions resemble adjacent tissue or lack clear boundaries.
- 3.3.2 Fundus Images: Fundus evaluation examined optic cup and optic disc segmentation using amalgamated datasets and cross-domain testing.
- 3.3.2 Fundus Images: Successful examples in Figure 1 cover eight modalities and compare SAM segmentations with corresponding ground truth images.
- 3.3.2 Fundus Images: Zero-shot SAM achieved only about half the accuracy of current SOTAs for fundus segmentation, partly because similar prompts make optic cup and disc separation difficult.
- 3.3.2 Fundus Images: SAM’s fundus performance varied across domains and was proportional to optic cup/disc contrast against the background.
3.3.3 Endoscopic OCT
On the OCT Rat Colon dataset, SAM’s zero-shot segmentation was far inferior to medical-imaging-specific models despite relatively simple image appearance. This contrasts with stronger performance on some other medical structures and highlights modality-dependent reliability.
- 3.3.3 Endoscopic OCT: SAM’s zero-shot segmentation on endoscopic OCT was far inferior to models specifically designed for medical images.The dataset contains three colonic-wall layers: colonic mucosa, submucosa, and muscularis externa.
- 3.3.3 Endoscopic OCT: Successful endoscopic OCT examples were not perfect, while failure cases were unsatisfactory.
3.3.6 MRI
For abdominal MRI and CT in AMOS, SAM could accurately segment some two-dimensional organs with precise localization and clear demarcation, but performance varied across individual classes and modalities. The study also reports inconsistent zero-shot results across other medical imaging settings.
- 3.3.6 MRI: SAM accurately segmented some two-dimensional organs in CT and MRI images, producing precise localization and clear demarcation.
- 3.3.6 MRI: Average Dice scores for CT and MRI were close because both modalities came from the same dataset and underwent the same preprocessing, although individual class results varied.
3.3.9 Retinal Vessel in Fundus Images
SAM failed to accurately segment retinal blood vessels in fundus images under zero-shot prompting, motivating a small supervised fine-tuning experiment. Fine-tuning substantially improved predictions, which often nearly matched the ground truth.
- 3.3.9 Retinal Vessel in Fundus Images: Zero-shot SAM could not accurately segment retinal blood vessels even with additional prompts focused on visible vessel regions.
- 3.3.9 Retinal Vessel in Fundus Images: The study attributed the difficulty to continuously branching structures, a challenge also observed for tree branches in natural images.
- 3.3.9 Retinal Vessel in Fundus Images: Fine-tuning SAM’s mask decoder used 20 image-mask pairs from DRIVE with ground-truth supervision and no prompt.
- 3.3.9 Retinal Vessel in Fundus Images: After fine-tuning, some vessel predictions almost perfectly matched the ground truth, though tiny terminal vessel parts were sometimes missing.
4. Discussion
SAM’s zero-shot medical segmentation performance varies substantially across modalities and generally does not surpass specialized medical-imaging models. The study also finds that limited fine-tuning can improve quality, while broader prompt and dataset evaluations remain necessary.
- Performance across domains: SAM’s zero-shot performance varies significantly across medical modalities and is often unsatisfactory for targets with unclear boundaries or concealed patterns.Endoscopy and dermoscopy perform relatively better, whereas some skin lesions and internal retinal structures remain challenging.
- Domain shift: SAM was trained primarily on natural images, while tested medical domains include unseen modalities and structures such as OCT layered tissue regions.This domain difference limits zero-shot discrimination of medical features that are unlike natural-image objects.
- Evaluation setup: A single center-point prompt was used for every sample, which does not fully exploit SAM’s available prompting modes.The authors state that this setup suffices to expose limitations but recommend evaluating bounding-box and automatic segmentation modes.
- Adaptation potential: Fine-tuning on a small amount of retinal vessel data produced impressive segmentation improvements, indicating potential for adapting SAM to medical imaging.The broader study identifies fine-tuning as a route toward better domain-specific performance.
- Study limitations: Fine-tuned SAM was evaluated only on retinal vessel datasets, so broader datasets are needed to assess its strengths and weaknesses across medical modalities.The authors specifically call for experiments spanning a wider range of datasets.
- Study contribution: The study evaluates SAM across a wide range of medical segmentation tasks and compares it quantitatively and qualitatively with baselines.These findings are intended to identify where SAM works and how it can be fine-tuned for medical applications.
5. Conclusions
This benchmark study evaluates SAM for zero-shot medical image segmentation across diverse modalities using standardized prompts without prior medical knowledge. Its findings expose current challenges and support developing medical foundation models tailored to this complex field.
- Benchmark scope: The study benchmarks SAM on diverse zero-shot medical segmentation tasks spanning dermoscope, fundus, CT, MRI, endoscope, X-Ray, and OCT images.The evaluation uses a standardized prompt setup without prior medical knowledge.
- Conclusion: The findings highlight the need for medical foundation models that explicitly account for challenges posed by complex and rapidly evolving medical imaging.The conclusion frames this as the overall implication of the benchmark results.