PaperlayerView exact revision on arXiv

Source-linked AI summary

Segment Anything in Medical Images

Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, Bo Wang

arXiv:2304.12306v3eess.IVcs.CV

TL;DR

Medical image segmentation models are often task-specific, limiting generalizability across tasks and imaging types. MedSAM fine-tunes SAM on 1,570,263 medical image-mask pairs spanning 10 imaging modalities and over 30 cancer types. Across internal and external validation tasks, it outperforms SAM and matches or surpasses modality-specific specialist models.

  • Problem

    Task-specific medical image segmentation models have limited generalizability across tasks and imaging types, restricting their wider clinical application.

    Source: Introduction · p. 2

  • Method

    MedSAM fine-tunes SAM on 1,570,263 medical image-mask pairs spanning 10 imaging modalities and over 30 cancer types.

    Source: Introduction · p. 2, MedSAM: a foundation model for… · p. 4

  • Results

    Across internal and external validation tasks, MedSAM outperforms SAM and matches or surpasses modality-specific specialist models.

    Source: Introduction · p. 3, Quantitative and qualitative analysis · p. 5

  • Takeaways & Limitations

    MedSAM’s delineations of anatomical structures and pathological regions support calculation of quantitative measures that serve as biomarkers.

    Source: Discussion · p. 9

    CT, MRI, and endoscopy dominate the training set, potentially affecting MedSAM’s performance on less-represented modalities such as mammography.

    Source: Discussion · p. 9

Abstract

from arXiv · show

Medical image segmentation is a critical component in clinical practice, facilitating accurate diagnosis, treatment planning, and disease monitoring. However, existing methods, often tailored to specific modalities or disease types, lack generalizability across the diverse spectrum of medical image segmentation tasks. Here we present MedSAM, a foundation model designed for bridging this gap by enabling universal medical image segmentation. The model is developed on a large-scale medical image dataset with 1,570,263 image-mask pairs, covering 10 imaging modalities and over 30 cancer types. We conduct a comprehensive evaluation on 86 internal validation tasks and 60 external validation tasks, demonstrating better accuracy and robustness than modality-wise specialist models. By delivering accurate and efficient segmentation across a wide spectrum of tasks, MedSAM holds significant potential to expedite the evolution of diagnostic tools and the personalization of treatment plans.

Introduction

MedSAM adapts SAM to medical segmentation by fine-tuning on more than one million medical image-mask pairs. Across internal and external validation tasks, it outperforms SAM and matches or surpasses same-modality specialist models.

Introduction

  • Manual segmentation is time-consuming, labor-intensive, and expertise-dependent; semi- or fully automatic methods can reduce workload, improve consistency, and enable large-scale analysis.Accurate segmentation supports disease diagnosis, treatment planning, and monitoring disease progression.Source: Introduction · p. 2
  • Task-specific medical segmentation models can lose substantial accuracy on new tasks or imaging data, limiting broader clinical application.Natural-image foundation models, including SAM and Segment Everything Everywhere with Multi-modal prompts all at once, demonstrate versatility across segmentation tasks.Source: Introduction · p. 2
  • Differences between natural and medical images limit SAM, motivating universal medical segmentation models trained once for application across diverse tasks.SAM uses point or bounding-box prompts to specify targets. It resembles conventional interactive segmentation methods but generalizes better than existing deep-learning interactive methods focused on limited tasks and modalities.Source: Introduction · p. 2
  • More than one million medical image-mask pairs underpin MedSAM’s fine-tuning of SAM, significantly improving medical image segmentation.Concurrent assessments found that unmodified SAM performed satisfactorily primarily on distinct-boundary targets but struggled with weak boundaries or low contrast.Source: Introduction · p. 2
  • 86 internal validation tasks and 60 external validation tasks demonstrate MedSAM outperforming SAM and matching or surpassing same-modality specialist models.Evaluation spans varied anatomical structures, pathological conditions, and medical imaging modalities.Source: Introduction · p. 3

Results

MedSAM combines diverse medical-image training with bounding-box prompting and ranked first on most internal and external validation tasks. Larger training sets improved performance, while MedSAM assistance substantially reduced expert annotation time.

MedSAM: a foundation model for promptable medical image segmentation

  • 1,570,263 medical image-mask pairs spanning 10 imaging modalities, over 30 cancer types, and varied imaging protocols formed MedSAM’s training dataset.CT, MRI, and endoscopy dominated the dataset, which also included ultrasound, pathology, fundus, dermoscopy, mammography, and OCT.Source: MedSAM: a foundation model for… · p. 4
  • MedSAM uses bounding-box-prompted 2D segmentation to accommodate variable clinical targets and processes 3D images slice by slice.Bounding boxes provide less ambiguous spatial context than points, addressing task variability that complicates fully automatic segmentation. Following SAM, image and prompt encoders produce image embeddings and positionally encoded box features, which the mask decoder fuses through cross-attention.Source: MedSAM: a foundation model for… · pp. 4–5

Quantitative and qualitative analysis

  • 86 internal and 60 external validation tasks showed MedSAM ranking first most often against SAM, U-Net, and DeepLabV3+ [24].MedSAM also had a narrower distribution of internal task-level DSC scores than the modality-wise specialist models; SAM ranked last on almost all internal tasks.Source: Quantitative and qualitative analysis · pp. 5–6, Figure 3 · p. 6, Figure 4 · p. 7
  • 87.8% median DSC on nasopharynx cancer segmentation accompanied reported improvements of 52.3%, 15.5%, and 22.7% over SAM, specialist U-Net, and DeepLabV3+, respectively.The DSC interquartile range was 85.0–91.4%. MedSAM also surpassed SAM and specialist models on unseen abdomen T1 Inphase and Outphase modalities, with improvements up to 10%.Source: Quantitative and qualitative analysis · p. 7
  • MedSAM better segmented challenging targets with indistinct boundaries and outperformed SAM on an unseen multiple-myeloma plasma-cell modality and task.External qualitative examples included cervical cancer in MR images, where MedSAM handled indistinguishable boundaries better than the comparison methods.Source: Quantitative and qualitative analysis · pp. 7–8

The effect of training dataset size

  • Increasing training dataset size significantly improved MedSAM performance on both internal and external validation sets.The study compared the default model with models trained on 10,000 and 100,000 images, uniformly sampled from the full training set to maintain diversity.Source: The effect of training dataset size · p. 8

MedSAM can improve the annotation efficiency

  • 82.37% and 82.95% reductions in annotation time were achieved for two experts using MedSAM rather than manual slice-by-slice annotation of 3D adrenal tumors.Experts marked long and short tumor axes every 3–10 slices, used MedSAM to generate segmentations, and manually revised the results until satisfied.Source: MedSAM can improve the annotation… · p. 8

Discussion

MedSAM supports diverse medical segmentation tasks with a single promptable model, outperforming the evaluated segmentation foundation model and rivaling or surpassing specialists. Training-set modality imbalance and ambiguous bounding-box prompts remain limitations.

Discussion

  • MedSAM uses over one million medical image-mask pairs and balances automation with customization through prompting, supporting diverse segmentation tasks without task-specific models.The authors identify potential applications in developing diagnostic and therapeutic tools and improving patient care.Source: Discussion · p. 9
  • MedSAM outperforms the evaluated state-of-the-art segmentation foundation model and rivals or surpasses specialist models, with internal and external evaluations demonstrating generalization to new data and tasks.Evaluations covered diverse segmentation targets.Source: Discussion · p. 9
  • MedSAM could accelerate 3D tumor annotation and enable tumor-volume measurement, a biomarker for disease progression and treatment response.Its delineation of anatomical structures and pathological regions enables quantitative biomarker computation.Source: Discussion · p. 9
  • MedSAM demonstrates adaptation of natural-image foundation models to medicine, potentially extending to biological segmentation, including cells in light microscopy [31] and organelles in electron microscopy.Source: Discussion · p. 9
  • Modality imbalance may impair performance on underrepresented imaging modalities, while ambiguous bounding-box prompts make vessel-like branching structures difficult to segment.CT, MRI, and endoscopy dominate training; arteries and veins can share a bounding box in fundus images. Learned medical features support fine-tuning for underrepresented modalities and intricate structures.Source: Discussion · p. 9

Methods · Dataset curation and pre-processing

The dataset combined publicly available medical images with expert segmentation annotations used directly for model development and validation. Pre-processing excluded unsuitable data and standardized image formats, intensities, and dimensions across modalities.

Dataset curation and pre-processing

  • Publicly available segmentation datasets supplied medical images and expert annotations, which were incorporated directly into model development and validation.Sources included the Cancer Imaging Archive (TCIA), Kaggle, Grand-Challenge, Scientific Data, CodaLab, and MICCAI segmentation challenges.Source: Dataset curation and pre-processing · pp. 9–10
  • CT and MR volumes were converted to NifTI, while grayscale and RGB images became PNG; exclusions removed incomplete images, branching segmentation targets, inaccurate annotations, and tiny volumes.The original 3D images used DICOM, nrrd, or mhd formats.Source: Dataset curation and pre-processing · p. 10
  • 1024×1024×3 was the standardized input size across images.Each 3D CT or MR slice was resized to 1024×1024 and its channel repeated three times; remaining 2D images were resized directly. Whole-slide pathology images used nonoverlapping sliding-window patches, with boundary patches zero-padded.Source: Dataset curation and pre-processing · p. 10
  • [0, 255] was the CT intensity range after Hounsfield-unit windowing using tissue-specific window widths and levels.Window width and level were (W:400, L:40) for soft tissues, (W:1500, L:-160) for lung, and (W:80, L:40) for brain.Source: Dataset curation and pre-processing · p. 10
  • 0.5th and 99.5th percentiles bounded intensity clipping for MR, X-Ray, ultrasound, mammography, and OCT images before rescaling to [0, 255].Intensity normalization addressed substantial differences between modalities to facilitate stable training.Source: Dataset curation and pre-processing · p. 10
  • [0, 255] RGB intensities remained unchanged; images outside that range underwent max-min normalization to [0, 255].This applied to endoscopy, dermoscopy, fundus, and pathology images.Source: Dataset curation and pre-processing · p. 10
  • Bicubic interpolation resized images, while nearest-neighbor interpolation resized masks to preserve precise boundaries and avoid artifacts.Source: Dataset curation and pre-processing · p. 10

Network architecture · Training protocol and experimental setting

MedSAM combines a ViT-Base image encoder, bounding-box prompt encoder, and lightweight mask decoder, with training and validation separated to prevent data leakage. In comparisons with task-specific U-Net models, MedSAM maintained consistent internal and external validation performance without additional fine-tuning, while U-Net performance declined externally.

Network architecture

  • MedSAM combines a ViT image encoder, bounding-box prompt encoder, and mask decoder that produces segmentation results and confidence scores.The 12-layer ViT-Base was chosen because larger encoders provided only marginal accuracy improvements at substantially greater computational cost; 1024×1024 images become 64×64 image embeddings. Bounding-box corners become paired 256-dimensional embeddings, and a lightweight two-transformer-layer decoder upsamples to 256×256 before sigmoid activation and bilinear resizing.Source: Network architecture · p. 11

Training protocol and experimental setting

  • 1,570,263 medical image-mask pairs were divided into 80% training, 10% tuning, and 10% validation sets, with scan-, video-, and slide-level separation preventing leakage.Pathology slides were separated before extraction of 1024×1024 patches. External datasets were entirely held out from training, testing generalization to unseen patients, imaging conditions, and potentially segmentation tasks.Source: Training protocol and experimental… · p. 11
  • 89,670,912 image-encoder parameters and 4,058,340 mask-decoder parameters were trainable, while the SAM-initialized ViT-Base model’s prompt encoder remained frozen.Bounding-box prompts were simulated from expert annotations with random perturbations of 0–20 pixels.Source: Training protocol and experimental… · p. 12
  • 150 epochs on 20 A100 (80G) GPUs trained MedSAM with a global batch size of 160, no data augmentation, and final-checkpoint selection.The objective was an unweighted sum of Dice and cross-entropy losses, described as robust across segmentation tasks. AdamW used β1 = 0.9, β2 = 0.999, an initial learning rate of 1e-4, and weight decay of 0.01.Source: Training protocol and experimental… · p. 12
  • 20 modality-specific U-Net and DeepLabV3+ [24] models provided specialist comparisons against SAM and MedSAM, which each used one model across modalities.Specialists were trained separately for each of the 10 modalities, with bounding boxes converted into binary masks and concatenated with images. U-Net experiments used default 2D nnU-Net configurations, training each model for 1000 epochs on one A100 GPU and selecting the last checkpoint.Source: Training protocol and experimental… · p. 12
  • 500 epochs on one A100 GPU trained each DeepLabV3+ specialist, using ResNet50, 224×224×3 inputs, and the last checkpoint.Images were resized following, and bounding-box masks supplied an additional input channel. Segmentation Models Pytorch (0.3.3) supported training and inference.Source: Training protocol and experimental… · p. 12
  • MedSAM maintained consistent internal and external validation performance without additional fine-tuning, whereas task-specific U-Net models performed well internally but declined significantly externally.The comparison covered CT liver cancer, MR abdominal organs, ultrasound nerve cancer, and endoscopic polyps. U-Net was selected because it outperformed DeepLabV3+ on most tasks.Source: Training protocol and experimental… · pp. 12–13

Loss function

MedSAM uses an unweighted sum of binary cross-entropy and Dice loss, chosen for robustness across medical image segmentation tasks.

  • The final loss L is the unweighted sum of binary cross-entropy and Dice loss.Source: Loss function · p. 13
  • Prior evidence of robustness across different medical image segmentation tasks motivated the combined loss choice.Source: Loss function · p. 13
  • The formulation denotes predicted segmentation by S and ground truth by G, with corresponding voxel values s_i and g_i.N is the number of voxels in image I.Source: Loss function · p. 13

Human annotation study

The study evaluated whether MedSAM could reduce adrenal-tumor annotation time compared with manual annotation. The assisted workflow combined sparse tumor-axis markers, automated segmentation, and expert refinement.

Human annotation study

  • 10 cases comprising 733 tumor slices tested annotation-time savings on adrenal tumors absent from MedSAM’s training and existing validation sets.Cases were randomly sampled from an adrenocortical carcinoma CT dataset.Source: Human annotation study · p. 13
  • Two radiologists independently performed manual slice-by-slice annotation, followed by MedSAM-assisted annotation after a one-week cooling period.The experts had 8 and 6 years of abdominal clinical experience, respectively; manual annotation time was recorded per case.Source: Human annotation study · pp. 13–14
  • Every 3-10 slices, experts independently marked the tumor’s long and short axes to initialize MedSAM-assisted segmentation.These sparse linear annotations followed a common practice in tumor response evaluation.Source: Human annotation study · p. 14
  • The automated pipeline enclosed axis markers in rectangular masks, interpolated masks for unlabeled slices, and converted them into bounding-box prompts for MedSAM.MedSAM received the bounding boxes together with the images to generate segmentations.Source: Human annotation study · p. 14
  • Assisted annotation time included initial marking, MedSAM inference, and manual refinement until experts were satisfied.Model runtime was recorded per case, and all manual annotation used ITK-SNAP.Source: Human annotation study · p. 14

Evaluation metrics

Following Metrics Reloaded [45], segmentation was evaluated using Dice Similarity Coefficient (DSC) for region overlap and Normalized Surface Distance (NSD) for boundary agreement.

  • Following Metrics Reloaded [45], the evaluation used Dice Similarity Coefficient and Normalized Surface Distance to quantify segmentation performance.Source: Evaluation metrics · p. 14
  • DSC measures region overlap between expert annotation masks and segmentation results.Source: Evaluation metrics · p. 14
  • NSD measures boundary agreement between expert annotation masks and segmentation results at a specified tolerance, set to 2 in this study.Source: Evaluation metrics · p. 14

Statistical analysis

Segmentation performance of MedSAM, SAM, U-Net, and DeepLabV3+ specialist models was compared using the Wilcoxon signed-rank test. Test results are marked in the DSC and NSD score tables.

  • The Wilcoxon signed-rank test assessed statistically superior segmentation performance among MedSAM, SAM, U-Net, and DeepLabV3+ specialist models.Source: Statistical analysis · p. 14
  • This non-parametric test supports paired-sample comparisons when data do not meet normal-distribution assumptions.Source: Statistical analysis · p. 14
  • Wilcoxon signed-rank test results are marked in the DSC and NSD score tables in Supplementary Tables 6–11.Source: Statistical analysis · p. 15

Software Utilized

The software implementation used Python 3.10 and Pytorch 2.0, with additional packages for data analysis and results visualization. Biorender was used to create Fig. 1.

  • All code was implemented in Python 3.10, with Pytorch 2.0 as the base deep learning framework.Source: Software Utilized · p. 15
  • Supporting packages provided data analysis and results visualization functionality.Packages included connected-components-3d (3.10.3), SimpleITK (2.2.1), nibabel (5.1.0), torchvision (0.15.2), numpy (1.24.3), scikit-image (0.20.0), scipy (1.10.1), pandas (2.0.2), matplotlib (3.7.1), opencv-python (4.8.0), ChallengeR (1.0.5), and plotly (5.15.0).Source: Software Utilized · p. 15
  • Biorender was used to create Fig. 1.Source: Software Utilized · p. 15

Data availability · Code availability

The study’s datasets are publicly accessible and permitted for research, while MedSAM’s training and inference scripts and trained model are publicly available.

Data availability

Code availability

Related papers

Loading 2304.12306v3…