Segment Anything Model for Medical Images?

1. Introduction

• Earlier foundation models were not explicitly optimized for image segmentation, while SAM introduced promptable zero-shot segmentation for unseen datasets and tasks.SAM uses points and boxes to indicate object semantics and location.Source: §1 · p. 2

1.1. SAM in Non-medical Image Applications

• Non-medical evaluations found mixed performance across applications, with human prompts refining results compared with automatic Everything mode.SAM performed well in some scenarios, including salient-object segmentation and agricultural analysis, but poorly in others.Source: §1.1 · p. 2

1.2. SAM in Medical Image Analysis

• Prior medical studies covered limited modalities, objects, testing strategies, and assessments, leaving broad MIS performance and object-factor relationships insufficiently examined.This report addresses that scope with COSMOS 1050K and analyses of SAM strategies and object characteristics.Source: §1.2 · pp. 4–5

2.1. Dataset Collection

• The dataset standardizes 53 public datasets to assess SAM across medical modalities, with 125 object-modality paired targets.Objects with extreme smallness, separated targets in extracted 3D slices, or dispersed structures were excluded because prompting them was difficult or confusing.Source: §2 · p. 5, §2.1 · p. 5

2.2. Dataset Preprocessing Specification

• For 3D data, preprocessing extracts labeled slices, normalizes image intensities to 0–255, resets mask labels, and saves images and labels as PNG.4D data are split into sets of 3D volumes before following the 3D processing flow.Source: §2.2 · pp. 5–7
• For 2D data, preprocessing retains images with sufficient label pixels, resets labels to object or location values, and converts image and label formats to PNG.CellSeg images and labels are reconstructed into sub-figures to handle original label values ranging from 1 to 1600.Source: §2.2 · p. 7
• 1,050,311 2D images or slices and 6,033,198 masks make up COSMOS 1050K.1,003,809 slices come from 8,653 3D volumes, and 46,502 are standalone 2D images.Source: §2.2 · p. 7

3.1. Brief Introduction to SAM

• SAM combines image, prompt, and mask-decoder components to produce masks from image embeddings and encoded prompts.Its prompt encoder accepts dense mask prompts and sparse points, boxes, and text; testing can use automatic Everything mode or manual prompts.Source: §3.1 · p. 8

3.2. Automatic Everything Mode

• Everything mode samples a grid of points across the image, generates candidate masks, then filters duplicates and low-quality masks.Filtering uses confidence scores, stability under threshold jitter, and non-maximal suppression.Source: §3.2 · p. 8

3.3. Manual Prompt Mode

• Five manual strategies test one or five positive points, positive and negative points, a box, or a box with a positive point.Points are sampled from the ground-truth mask or nearby non-target region, and the box is the tight ground-truth bounding box; the rules support repeatability.Source: §3.3 · pp. 8–9

3.4. Inference Efficiency

• Precomputing and reusing image embeddings across strategies improves testing efficiency by nearly n times.This avoids repeating the time-consuming image-encoding operation for each strategy.Source: §3.4 · p. 9

3.5. Mask-matching Mechanism for Segmentation Evaluation

• For each object, evaluation selects the predicted mask with the highest Dice score against the ground truth.The selection is made among the multiple binary masks SAM generates for an image.Source: §3.5 · p. 9

• Reusing one image embedding per test image and precomputing prompts avoided repeated extraction and calculation, improving testing efficiency approximately n×.The saved embeddings and prompt information could be loaded directly for different testing strategies.Source: §4.1 · p. 9
• Tests ran on GTX 2080Ti, GTX 3090, and A40 GPUs using Python 3.8.0, PyTorch 2.0.0, and torchvision 0.15.1.The implementation also used torch.compile, mixed precision, and inference mode.Source: §4.1 · p. 9
• The evaluation compared automatic Everything mode with five manual-prompt settings: one or five positive points, positive and negative points, a box, and a box plus a point.Source: §4.1 · p. 9

4.2. Evaluation Metrics

• The evaluation used DICE, JAC, and HD, with main-text analyses focused on DICE and HD and JAC reported in supplementary materials.Source: §4.2 · p. 10

4.3. Segmentation Performance under Different Models

• Under Everything mode, ViT-H exceeded ViT-B by 7.47% at DICE and was 10.61 pixels lower on HD; its advantage grew with more point prompts.With box prompts, the models were close, with DICE differences of 0.37% and 0.06% for the two box strategies.Source: §4.3 · p. 10

4.4. Segmentation Performance under Different Testing Modes

• Everything mode was worst, while box prompts were best; adding points improved DICE from 56.02% to 63.81% for ViT-B and 56.78% to 71.61% for ViT-H.Adding one point to a box produced no obvious change, with DICE decreases of 0.49% for ViT-B and 0.06% for ViT-H.Source: §4.4 · p. 10

4.5. Performance of Medical Images on Different Modalities

• For box prompts, X-ray had the highest average DICE for both models; histopathology, colonoscopy, and DW MRI exceeded 90% with relatively low standard deviations.Other modalities generally had lower and less stable performance; the same object could also vary across modalities.Source: §4.5 · p. 10

4.6. Inference Time Analysis of SAM

• Manual prompt encoding and mask decoding take less than 0.01s, while ViT-H requires nearly four times ViT-B’s embedding time under the controlled timing setup.Tests used one 256×256 image target on an NVIDIA GTX 3090 GPU with 24G memory.Source: §4.6 · p. 11

4.7. Analysis on the Number of Points in Everything Mode

• As Everything-mode points increased from 82 to 2562, DICE rose across four multi-object datasets, but excessive points could split objects and increase test time.This creates a trade-off between segmentation performance and test efficiency.Source: §4.7 · p. 12, Table 5 · p. 15

4.8. Analysis of Factors Correlating to Segmentation Results

• The analysis measured size, aspect ratio, foreground-background intensity difference, modality, and boundary complexity across 191,779 anatomical structures.Boundary complexity was represented using Elliptical Fourier Descriptors; the Fourier order estimates the accumulations needed for decoded-contour overlap with the original contour.Source: §4.8 · pp. 12, 14, 15
• DICE had moderate correlations with Fourier order and intensity difference, weak correlation with size, and no correlation with modality or aspect ratio; complex boundaries and low contrast were associated with poorer performance.The authors report these patterns across most testing strategies.Source: Table 6 · p. 15, §4.8 · p. 16

• SAM assistance improved annotation quality (HCE: 0.27↓) and speed by approximately 25%, saving about 1.31 minutes per image and 0.2 minutes per object.The evaluation used 100 images, 55 objects, and 620 masks across 9 modalities; three doctors with 10 years of experience annotated from scratch and adjusted SAM box-prompt predictions.Source: §4.9 · p. 16

4.10. Impact of Di

• Random shifts of 0–10, 10–20, and 20–30 pixels were used to simulate real-life variation in point and box placement.The authors note that exact center clicks and tight boxes are impractical for evaluating SAM in real use.Source: §4.10 · pp. 16–17
• With 20–30-pixel shifts, DICE fell 14.62% for a single point, 24.11% for a box, and 29.93% for a box plus one point.More point prompts alleviated the decline; the random experiments were repeated three times and averaged.Source: §4.10 · p. 17, Table 8 · p. 17
• The figures present DICE trends under different attributes for ViT-B and ViT-H with box prompts, and DICE improvement after finetuning for both encoders.The provided captions do not state the plotted values.Source: Figure 17 · p. 18, Figure 18 · p. 18

4.11. Comparison between SAM and Interactive Methods

• With one point, SAM outperformed FocalClick and SimpleClick, but using 10 points it performed worse than both interactive methods.Across iterations, SAM’s performance increased slowly or declined, while the interactive methods improved steadily; the comparison covered 10 typical organs and tumors across varied modalities and image characteristics.Source: §4.11 · p. 19

4.12. Task-specific Refinement for SAM

• The authors attribute SAM’s weak medical-image perception mainly to absent medical training data and differences in image appearance, boundaries, foreground-background contrast, and scale.Source: §4.12 · p. 19
• The authors fine-tuned 45 common objects using box prompts, freezing the image and prompt encoders and adjusting only the mask decoder.Training lasted 20 epochs, with a 1e-4 learning rate and batch size 2.Source: §4.12 · p. 19
• Finetuning improved segmentation overall for both backbones: ViT-B improved on 32 of 45 objects and ViT-H on 37 of 45.The results also showed a shift toward higher DICE values across correlating factors.Source: §4.12 · p. 19
• Performance decreased for objects with small numbers and RGB color coding, suggesting that task-specific finetuning may need more careful design.Source: §4.12 · p. 19

5. Conclusion

• SAM was strong on some objects but unstable or failed elsewhere; ViT-H outperformed ViT-B overall.Source: §5 · p. 19
• Manual prompts, especially boxes, outperformed Everything mode, while randomness in center points and tight boxes could sharply reduce performance.Source: §5 · p. 19
• SAM supported faster, high-quality annotation and beat interactive methods with one or a few points, but lost ground as point counts rose.Source: §5 · p. 19
• Performance correlated with factors including boundary complexity; task-specific finetuning raised average DICE by 4.39% for ViT-B and 6.68% for ViT-H.Source: §5 · p. 19

Semantic-aware SAM

• SAM can propose regions but cannot identify their categories; CLIP classification and Grounding DINO boxes paired with SAM are natural-image approaches, while medical semantic awareness faces varied shapes, types, and similar subclasses.Source: §6 · p. 19

SAM vs. traditional segmentation methods?

• Prior studies report that fine-tuned 2D SAM can outperform specialist U-Net models in most cases, while 3D MA-SAM with adapters can outperform nnU-Net without prompts.The discussion also flags prompt-noise robustness and multi-round interaction as unresolved problems.Source: §6 · p. 20

2D or 3D SAM?

• A consistent 2D model is described as more practical across medical data types, while limited 3D data may constrain 3D foundation models, especially those trained from scratch.The authors plan to explore synthesizing high-fidelity 3D data.Source: §6 · p. 20

SAM boosts large-scale medical annotation?

• SAM preliminarily shortened annotation time and improved quality; its efficiency advantage grows with more anatomical structures, and one network could support multimodal, multiclass labeling.The discussion links this potential to lightweight deployment in labeling software.Source: §6 · p. 20