Source-linked AI summary
Segment Anything in Medical Images
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, Bo Wang
TL;DR
Medical image segmentation lacks generality because many models are tailored to specific tasks, modalities, or disease types. MedSAM uses promptable segmentation trained on a large, diverse medical dataset, and evaluations show performance exceeding SAM while rivaling or surpassing specialist models. Its broad segmentation capability may support quantitative clinical analyses, although modality imbalance and ambiguous prompts constrain performance in some settings.
Problem
Many medical image segmentation models are task-specific and can degrade on new tasks or different imaging data, limiting generality across clinical applications.
Method
MedSAM is a promptable 2D segmentation foundation model trained on 1,570,263 image-mask pairs spanning 10 imaging modalities and over 30 cancer types.
Results
Across 86 internal and 60 external validation tasks, MedSAM exceeded SAM and rivaled or surpassed modality-wise specialist models.
Takeaways & Limitations
MedSAM provides a versatile approach for segmenting diverse anatomical structures and lesions across medical imaging modalities.
Takeaways & Limitations
CT, MRI, and endoscopy dominate training data, potentially affecting performance on less-represented modalities such as mammography; bounding-box prompts also struggle with vessel-like branching structures.
Abstract
from arXiv · showhide
Medical image segmentation is a critical component in clinical practice, facilitating accurate diagnosis, treatment planning, and disease monitoring. However, existing methods, often tailored to specific modalities or disease types, lack generalizability across the diverse spectrum of medical image segmentation tasks. Here we present MedSAM, a foundation model designed for bridging this gap by enabling universal medical image segmentation. The model is developed on a large-scale medical image dataset with 1,570,263 image-mask pairs, covering 10 imaging modalities and over 30 cancer types. We conduct a comprehensive evaluation on 86 internal validation tasks and 60 external validation tasks, demonstrating better accuracy and robustness than modality-wise specialist models. By delivering accurate and efficient segmentation across a wide spectrum of tasks, MedSAM holds significant potential to expedite the evolution of diagnostic tools and the personalization of treatment plans.
Introduction
Medical image segmentation is clinically important but current models are often task-specific and generalize poorly across new tasks or imaging data. MedSAM addresses this gap by adapting promptable foundation-model segmentation to diverse medical images.
- Accurate segmentation supports diagnosis, treatment planning, and monitoring, but manual delineation is time-consuming, labor-intensive, and expertise-dependent.
- Many deep learning segmentation models are designed for specific tasks, with performance degrading on new tasks or different imaging data.
- Universal medical segmentation models could be trained once and applied across tasks, improving versatility and potentially consistency.
- SAM performs better on targets with distinct boundaries but has substantial limitations for weak-boundary or low-contrast medical targets.
- MedSAM fine-tunes SAM on more than one million medical image-mask pairs to enhance segmentation performance on medical images.
- The training dataset spans diverse anatomical structures, pathological conditions, and medical imaging modalities.
Results
MedSAM uses a large, diverse medical image dataset and promptable architecture to support segmentation across modalities and task types. Across internal and external evaluations, it generally outperformed SAM and matched or exceeded specialist models, while also reducing annotation time.
- Dataset and model design: 1,570,263 image-mask pairs spanning 10 imaging modalities and over 30 cancer types were curated to train MedSAM.The dataset was designed to capture variation in imaging conditions, anatomies, and pathological conditions.
- Dataset and model design: MedSAM uses a promptable 2D segmentation approach that supports task-specific prompts and processes 3D images as sequences of 2D slices.Bounding boxes and points are described as user prompts, with the model following SAM’s image encoder, prompt encoder, and mask decoder architecture.
- Internal validation: 86 internal validation tasks showed MedSAM ranked first on most tasks and generally exceeded specialist U-Net and DeepLabV3+ models, whereas SAM ranked last in almost all tasks.The evaluation compared MedSAM and SAM with 10 modality-specific specialist models for each of U-Net and DeepLabV3+.
- External validation: 60 external validation tasks showed MedSAM consistently delivered superior performance, including a median DSC of 87.8% for nasopharynx cancer segmentation.This score represented improvements of 52.3%, 15.5%, and 22.7% over SAM, specialist U-Net, and DeepLabV3+, respectively.
- External validation: MedSAM generalized to unseen targets and modalities, outperforming specialist models on challenging cases with indistinguishable boundaries.For right-kidney segmentation in MR T1-weighted images, SAM, U-Net, and DeepLabV3+ achieved 90.1%, 85.3%, and 86.4%, respectively, while MedSAM also improved performance by up to 10% on some unseen modalities.
- Annotation efficiency: MedSAM assistance reduced 3D adrenal-tumor annotation time by 82.37% and 82.95% for two human experts.The assisted workflow used sparse linear annotations followed by MedSAM segmentation and manual revision.
Discussion
MedSAM is presented as a versatile, promptable foundation model for segmenting diverse anatomical structures and lesions across medical imaging tasks. Its broad generalization supports clinical quantification, while modality imbalance and ambiguous prompts constrain some applications.
- MedSAM is a promptable foundation model for segmenting anatomical structures and lesions across diverse medical imaging modalities.
- MedSAM’s internal and external evaluations demonstrate robust generalization across diverse targets, new data, and new tasks.
- MedSAM can support quantitative biomarkers by delineating structures and lesions, including tumor volumes for disease-progression and treatment-response assessment.
- Training data are imbalanced toward CT, MRI, and endoscopy, potentially affecting performance on less-represented modalities such as mammography.
- Bounding-box prompts can be ambiguous for vessel-like branching structures, although fine-tuning can adapt MedSAM to underrepresented modalities and intricate structures.
- The study highlights the feasibility of one foundation model managing many segmentation tasks instead of separate task-specific models.
Methods
The methods assemble heterogeneous public medical segmentation datasets into standardized image formats and use a transformer-based, promptable architecture. MedSAM combines image-feature extraction, bounding-box prompting, and mask decoding for segmentation.
- The dataset was curated from public medical segmentation resources with expert annotations used for model development and validation.
- 3D CT and MR images were converted to NifTI, while grayscale and RGB images were converted to PNG for compatibility and uniformity.
- CT intensities were normalized with modality-specific Hounsfield-unit windows and rescaled to [0, 255], while several other modalities were percentile-clipped.
- The network uses transformer architecture with a ViT-based image encoder, prompt encoder, and mask decoder.
- The prompt encoder integrates bounding-box interactions, and the mask decoder produces segmentation results and confidence scores from image and prompt embeddings.
- The base ViT encoder was selected because larger ViT models provided marginal accuracy gains while substantially increasing computational demands.
Training protocol and experimental setting
MedSAM was trained and evaluated using large-scale, independently split medical data and compared with foundation, modality-specialist, and task-specific segmentation models. External validation showed a clear generalization advantage over task-specific U-Net models.
- 1,570,263 medical image-mask pairs supported model development and validation, with internal data split into 80% training, 10% tuning, and 10% validation.
- Training and validation were independent, with continuity-aware splitting performed at the 3D-scan or video level where appropriate.
- MedSAM was initialized from pre-trained SAM ViT-Base, with the image encoder and mask decoder trained using perturbed bounding-box prompts derived from expert annotations.
- Comparisons included SAM, 20 modality-specific U-Net and DeepLabV3+ specialist models, and task-specific U-Net models on four representative tasks.
- Task-specific U-Net models often performed well internally but declined significantly on external datasets, whereas MedSAM maintained consistent internal and external performance without fine-tuning.
Loss function
The final training objective combines binary cross-entropy and Dice losses without weighting, using segmentation predictions and ground-truth labels to optimize the model.
- The loss function is the unweighted sum of binary cross-entropy loss and Dice loss.
- Binary cross-entropy uses predicted and ground-truth voxel values across the image’s voxels.
- The combined loss was selected because it has been shown to be robust across different medical image segmentation tasks.
Human annotation study
The human annotation study compared slice-by-slice manual tumor annotation with a workflow using sparse markers, MedSAM inference, and manual refinement. It evaluated annotation time on 10 adrenal tumor CT cases comprising 733 tumor slices.
- Study design: 10 cases and 733 tumor slices were sampled from an adrenocortical carcinoma CT dataset whose adrenal tumor target was absent from MedSAM’s training and existing validation sets.Two experienced radiologists participated independently.
- Study design: The manual group independently annotated the 3D adrenal tumor slice by slice, with time recorded for each case.The experts did not discuss or collaborate during annotation.
- MedSAM-assisted workflow: The MedSAM group independently drew long and short tumor axes every 3–10 slices as initial markers from the tumor’s top to bottom slice.These sparse linear annotations were used as inputs for MedSAM-based segmentation.
- MedSAM-assisted workflow: The workflow automatically generated slice masks, interpolated masks for unlabeled slices, converted them to bounding boxes, and fed them with images into MedSAM.The three-step process transformed sparse linear labels into segmentation inputs.
- Time accounting: The assisted annotation time comprised initial markers, MedSAM inference, and manual refinement until the experts were satisfied.All manual annotation was performed in ITK-SNAP, an open-source medical image visualization and annotation tool.
Evaluation metrics
Segmentation quality was evaluated with complementary region-overlap and boundary-consensus metrics. NSD used a specified tolerance of 2.
- Metrics: Dice Similarity Coefficient evaluated region overlap between expert annotation masks and segmentation results.It was used alongside NSD following Metrics Reloaded recommendations.
- Metrics: Normalized Surface Distance evaluated boundary consensus between expert annotation masks and segmentation results at a given tolerance.This complements the region-based DSC with a boundary-based assessment.
- Metrics: The NSD tolerance τ was set to 2 in this study.The tolerance defines the distance used for the boundary-consensus calculation.
Statistical analysis
The study used the Wilcoxon signed-rank test to compare four segmentation methods on paired samples, with results marked in DSC and NSD score tables.
- Statistical comparison: Four methods—MedSAM, SAM, U-Net, and DeepLabV3+ specialist models—were compared statistically.The comparison targeted segmentation performance across the evaluated methods.
- Statistical comparison: The Wilcoxon signed-rank test was selected as a non-parametric test suited to paired samples and data that may not satisfy normality assumptions.The analysis assessed whether any method demonstrated statistically superior segmentation performance.
- Reported results: Wilcoxon test results were marked on the DSC and NSD score tables in Supplementary Tables 6–11.These markings identify the statistical analysis results associated with the reported scores.
- Implementation: The implementation used Python 3.10 with PyTorch 2.0 and packages for data analysis and visualization.The listed tools included SimpleITK, nibabel, torchvision, NumPy, scikit-image, SciPy, pandas, matplotlib, OpenCV, ChallengeR, and Plotly.
- Data access: Training and validation datasets were publicly accessible and permitted for research purposes, with source data provided alongside the paper.Download links were provided through Supplementary Tables 16–17.
Code availability
The authors publicly released the training and inference scripts and the trained MedSAM model. The work acknowledges funding, data contributors, and computing support, and reports no competing interests.
- Code release: The training script, inference script, and trained model are publicly available in the MedSAM GitHub repository.A permanent version was also released on Zenodo.
- Acknowledgments: The authors acknowledged NSERC and CIFAR AI Chair program support for the work.The funding included specified NSERC grant and award identifiers.
- Acknowledgments: The acknowledgments thanked medical-image data owners and Meta AI for releasing Segment Anything source code.The paper also credited community-provided data and software resources.
- Competing interests: The authors declared no competing interests.