Source-linked AI summary
SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model
Saikat Roy, Tassilo Wald, Gregor Koehler, Maximilian R. Rokuss, Nico Disch, Julius Holzschuh, David Zimmerer, Klaus H. Maier-Hein
TL;DR
This paper examines how SAM performs on zero-shot medical CT organ segmentation, an early evaluation of its promptable segmentation capabilities. It tests point and box prompts on abdominal CT slices and finds box prompting, particularly at moderate jitter, highly competitive with the baselines. The authors see potential for expert-guided semi-automatic segmentation, while noting that SAM lacked a direct text-prompt interface.
Problem
Method
The study evaluates SAM on axial AMOS22 abdominal CT slices using random positive-point prompts and bounding boxes with simulated jitter, provided in an oracle fashion.
Source: §2 · p. 2Results
Box prompting, especially at moderate jitter, was highly competitive with the baselines, and single positive boxes performed considerably better than 10 positive point prompts.
Source: §3.2 · p. 3, §3.1 · p. 3Takeaways & Limitations
The authors see potential for experts to use SAM for rapid semi-automatic segmentation of most relevant structures in interactive medical segmentation pipelines.
Source: §3.2 · p. 3, §4 · p. 3To the authors’ knowledge, SAM did not yet provide a direct text-prompt interface.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
Foundation models have taken over natural language processing and image generation domains due to the flexibility of prompting. With the recent introduction of the Segment Anything Model (SAM), this prompt-driven paradigm has entered image segmentation with a hitherto unexplored abundance of capabilities. The purpose of this paper is to conduct an initial evaluation of the out-of-the-box zero-shot capabilities of SAM for medical image segmentation, by evaluating its performance on an abdominal CT organ segmentation task, via point or bounding box based prompting. We show that SAM generalizes well to CT data, making it a potential catalyst for the advancement of semi-automatic segmentation tools for clinicians. We believe that this foundation model, while not reaching state-of-the-art segmentation performance in our investigations, can serve as a highly potent starting point for further adaptations of such models to the intricacies of the medical domain. Keywords: medical image segmentation, SAM, foundation models, zero-shot learning
1. Introduction
Foundation models use large-scale training to generalize across tasks, and prompting has enabled applications in language and image generation. This paper offers an early evaluation of promptable SAM on multi-organ CT segmentation.
- The paper presents an early-stage evaluation of SAM with different visual prompts on a multi-organ CT dataset.Source: §1 · p. 1
- Foundation models are trained on very large datasets to generalize across tasks and domains.Source: §1 · p. 1
- Figure 1 depicts random point and jittered-box prompts in green and the resulting segmentation masks in blue.Source: Figure 1 · p. 2
2. Methods
The evaluation uses axial abdominal CT slices from AMOS22 and tests SAM with oracle-style positive point and box prompts, including simulated box jitter.
Slice Extraction
- The evaluation uses axial 2D slices from AMOS22, centered around each labeled class’s center of mass.Each patient and class contributes 197–240 slices, each with foreground and a corresponding binary mask.Source: §2 · p. 2
Visual Prompt Engineering
3. Results and Discussion
On abdominal CT organ segmentation, SAM’s box prompting—especially at moderate jitter—was highly competitive with the nnU-Net baselines, while single boxes outperformed ten-point prompts. The discussion identifies interactive, expert-guided segmentation as a potential use while noting that SAM lacked a direct text-prompt interface.
3.1. Results
- The authors note that, to their knowledge, SAM did not yet provide a direct text-prompt interface.Source: §3.1 · p. 2
- Box prompting at 0.1 jitter was highly competitive against the baselines, and single positive boxes performed considerably better than 10 positive points.The comparison reports Dice Similarity Coefficient (DSC); the table covers point and box prompting against 2D and 3D nnU-Net, with AVG* reporting raw-CT values.Source: §3.1 · pp. 2–3, §3.2 · p. 3, Table 1 · p. 3
3.2. Discussion
- SAM’s performance did not degrade on raw CT values, indicating robustness of box prompting to high intensity ranges.Source: §3.2 · p. 3, Table 1 · p. 3
- The authors identify rapid, semi-automatic segmentation of most structures as a potential use, with experts supplying a few prompts and manually segmenting areas near class boundaries.The paper says SAM may have decreased accuracy near class boundaries.Source: §3.2 · p. 3
4. Conclusion
The conclusion presents SAM as a zero-shot approach for medical image segmentation using a few clicks or bounding-box prompts, with potential value in interactive clinical pipelines.
- The study evaluates SAM’s zero-shot effectiveness on novel medical image segmentation tasks using a few click and bounding-box prompts.Source: §4 · p. 3
- The authors conclude that expert users can rapidly semi-automatically segment most relevant structures with SAM.Source: §4 · p. 3
- The authors describe SAM as highly valuable for interactive medical segmentation pipelines.Source: §4 · p. 3