Source-linked AI summary
Segment Anything Model for Medical Image Analysis: an Experimental Study
Maciej A. Mazurowski, Haoyu Dong, Hanxue Gu, Jichen Yang, Nicholas Konz, Yixin Zhang
TL;DR
Medical image segmentation remains challenging because obtaining annotations is expensive and time-consuming. The paper extensively evaluates SAM across 19 medical imaging datasets and finds that its zero-shot performance varies substantially, with prompting strategy and task characteristics strongly shaping outcomes. SAM is promising for selected medical-imaging uses, but requires suitable prompting and careful application.
Problem
Developing segmentation models for new medical imaging tasks is challenging because collecting and curating images with careful radiologist mask annotations is expensive and time-consuming.
Method
The study evaluates SAM's zero-shot medical-image segmentation across 19 datasets spanning different imaging modalities and anatomies, including single and iterative prompts.
Results
SAM's performance is moderate on average and varies substantially across datasets and images, while box prompts generally outperform point prompts and single-point SAM usually outperforms RITM, SimpleClick, and FocalClick.
Takeaways & Limitations
SAM shows promise for medical images when suitable prompting strategies are selected for the dataset and task.
Takeaways & Limitations
With iteratively refined multiple point prompts, SAM usually gains little beyond cases involving objects with multiple parts, while other methods can surpass its point-based performance.
Abstract
from arXiv · showhide
Training segmentation models for medical images continues to be challenging due to the limited availability of data annotations. Segment Anything Model (SAM) is a foundation model that is intended to segment user-defined objects of interest in an interactive manner. While the performance on natural images is impressive, medical image domains pose their own set of challenges. Here, we perform an extensive evaluation of SAM's ability to segment medical images on a collection of 19 medical imaging datasets from various modalities and anatomies. We report the following findings: (1) SAM's performance based on single prompts highly varies depending on the dataset and the task, from IoU=0.1135 for spine MRI to IoU=0.8650 for hip X-ray. (2) Segmentation performance appears to be better for well-circumscribed objects with prompts with less ambiguity and poorer in various other scenarios such as the segmentation of brain tumors. (3) SAM performs notably better with box prompts than with point prompts. (4) SAM outperforms similar methods RITM, SimpleClick, and FocalClick in almost all single-point prompt settings. (5) When multiple-point prompts are provided iteratively, SAM's performance generally improves only slightly while other methods' performance improves to the level that surpasses SAM's point-based performance. We also provide several illustrations for SAM's performance on all tested datasets, iterative segmentation, and SAM's behavior given prompt ambiguity. We conclude that SAM shows impressive zero-shot segmentation performance for certain medical imaging datasets, but moderate to poor performance for others. SAM has the potential to make a significant impact in automated medical image segmentation in medical imaging, but appropriate care needs to be applied when using it.
1. Introduction
Medical image segmentation is important but difficult to develop for new tasks because high-quality mask annotation is expensive and time-consuming. SAM offers a prompt-based foundation-model approach that can support zero-shot segmentation and several interactive or assisted medical-imaging workflows.
- 1. Introduction: Medical image segmentation covers organs, abnormalities, bones, and other structures, but new datasets and tasks require costly, time-consuming radiologist mask annotations.The annotation burden arises because trained radiologists typically must provide careful image masks.
- 1. Introduction: Foundation models and zero-shot learning could mitigate the difficulty of developing segmentation models for new medical imaging data and tasks.SAM is presented as a foundation model with promising zero-shot segmentation performance on natural-image datasets.
- 1. Introduction: SAM segments user-defined objects by producing a mask from prompts such as points, boxes, masks, or text, including when prompts are ambiguous.The model is designed to use the prompt to identify the object of interest and return a valid segmentation mask.
- 1.2. How to segment medical images with SAM?: In medical imaging, SAM requires prompts and therefore differs from conventional segmentation models that directly map an image to one or more desired masks.The paper does not expect prompt-free operation to be useful when many other objects are present in medical images.
- 1.2. How to segment medical images with SAM?: The paper describes SAM-assisted annotation, mask generation or training of additional models, and training or fine-tuning SAM-like models for medical images.These approaches include using SAM alongside detection or classification models and using generated masks as refined training examples.
2. Methodology
The study evaluates SAM for medical image segmentation across diverse datasets and realistic prompting scenarios, including non-iterative, iterative, ambiguous, and comparative settings. Its methodology uses curated medical imaging tasks, multiple prompt-generation modes, and comparisons with interactive segmentation methods.
- Datasets: The evaluation covers 19 publicly available medical imaging datasets and treats each object type as a separate segmentation task.The datasets include radiological X-rays, MRIs, CTs, ultrasound, and PET images.
- Evaluation design: SAM is evaluated with both non-iterative prompts generated before inference and iterative prompts generated after inspecting model predictions.The study also examines segment-everything behavior and outputs produced under prompt ambiguity.
- Prompting strategies: Five prompting modes use points or tightly enclosing boxes to represent single contiguous regions, multiple disconnected regions, or an entire object mask.Point modes place prompts at object-region centers, while box modes enclose the largest region, each region, or the full mask.
- Prompting strategies: Iterative point prompts are generated by predicting a mask, computing its error region against the ground truth, and selecting subsequent points farthest from that region’s boundary.The procedure initializes a point farthest from the background and repeatedly updates the error region after each prediction.
- Prompting strategies: The study analyzes prompt ambiguity using SAM’s multiple predictions and reports oracle performance by selecting the prediction with highest IoU to the true mask.Oracle selection uses ground-truth knowledge and is therefore biased for settings without further user interaction.
- Comparative evaluation: SAM is compared with RITM, SimpleClick, and FocalClick under one-point and interactive prompting settings.The comparison evaluates masks against dataset ground-truth annotations using IoU, while Figure 3 visualizes point- and box-prompt results across IoU percentiles.
3. Results
SAM’s medical-image segmentation performance varies substantially across datasets and prompting conditions. Box-based prompts generally outperform point prompts, while iterative points provide limited gains for SAM compared with competing methods.
- SAM’s IoU ranges from 0.9118 to 0.1136 across datasets and prompting modes.
- Mode 4, using a separate box for each object part, achieves the highest average performance at 0.6542 IoU.Box prompts outperform point prompts, and separately indicating object parts is especially beneficial for multi-part objects.
- Oracle mode moderately improves over default mode, with the improvement magnitude varying substantially by dataset.
- SAM outperforms other methods on 24 of 28 non-iterative tasks, increasing to 26 of 28 tasks in oracle mode.
- Average point-prompt performance is 0.4595 IoU for SAM, compared with 0.2240 for FocalClick, 0.1910 for SimpleClick, and 0.1322 for RITM.SAM’s Mode 3 single-box setting averages 0.5891 IoU and outperforms all other methods on all but one task.
- With iterative prompts, SAM gains little after two or three points, allowing SimpleClick and RITM to surpass it at five or more points.
- Additional prompts can help SAM capture multiple non-contiguous regions, but further points may provide negligible or detrimental effects afterward.
- SAM’s highest-confidence masks often resemble region-growing outputs, while lower-confidence predictions can expand into more varied intensity or texture regions.
4. Conclusions and discussion
The study finds that SAM’s medical-image segmentation performance varies substantially with prompting and task conditions. Box prompts generally work best, while iterative point prompting benefits competing methods more than SAM.
- SAM’s zero-shot medical-image segmentation accuracy is moderate on average and varies significantly across datasets and images within a dataset.
- SAM performs best with box prompts, particularly when one box is provided for each separate object part.
- SAM outperforms RITM, SimpleClick, and FocalClick in most evaluated settings using a single non-iterative prompt point.
- With multiple iteratively refined point prompts, SAM gains little except for multi-part objects, while competing algorithms improve enough to surpass SAM’s performance.
- There is a small but non-statistically significant correlation between average object size in a dataset and SAM performance.
- Future work could investigate prompt engineering and medical-imaging-specific or three-dimensional adaptations of SAM.
A. Graphical Abstract
The graphical abstract presents a visual overview of the study’s work.
- Figure 10 provides a graphical abstract of the work.
B. Examples of segmented results on all datasets
The paper visualizes SAM’s segmentation results across all evaluated tasks using representative examples at different IoU percentiles.
- Figures 11–16 visualize segmented results across all 28 tasks.
- For each task, the figures show three examples corresponding to the 25th, 50th, and 75th IoU percentiles.
- For the PET/CT dataset, PET is represented in the PNG red channel, while CT is represented in the green and blue channels.
C. Performance of other competing methods
The paper includes visual material documenting SAM and competing methods, including detailed performance under interactive prompting.
- Figure 10 presents a graphical abstract of the work.
- Figures 11–16 provide visualization examples of SAM’s prediction results across multiple parts.
- Figure 17 presents detailed performance for RITM, SimpleClick, and FocalClick across each task under interactive prompting.