PaperlayerView exact revision on arXiv

Source-linked AI summary

Segment Anything in Medical Images and Videos: Benchmark and Deployment

Jun Ma, Sumin Kim, Feifei Li, Mohammed Baharoon, Reza Asakereh, Hongwei Lyu, Bo Wang

arXiv:2408.03322v1eess.IVcs.CV

TL;DR

The utility of segmentation foundation models for medical data remains unclear, particularly for 3D images and videos. The paper benchmarks SAM2 against SAM1 and MedSAM, develops fine-tuning and deployment tools, and finds that performance varies across modalities and tasks.

  • Problem

    SAM2's medical performance, especially on 3D images and videos, was unclear, and its official interface supported only short videos and limited medical data formats.

    Source: §2.7 · p. 9

  • Method

    The paper benchmarks SAM2 across medical modalities and 2D, 3D, and video data, compares it with SAM1 and MedSAM, and develops fine-tuning and deployment interfaces.

    Source: §1 · p. 3, §2.1 · p. 3

  • Results

    SAM2's relative performance varies by task: it exceeds SAM1 on some 2D modalities and in 3D CT and MR, but trails SAM1 in PET, while video winners differ by modality.

    Source: §2.2 · p. 4, §2.3 · p. 5, §2.5 · p. 7

  • Takeaways & Limitations

    The authors provide 3D Slicer and Gradio tools to facilitate SAM2 deployment for medical image and video segmentation, which they consider important for clinical adoption.

    Source: Limitation and Future Work · p. 12

    SAM2-T still requires large GPU RAM for long-video segmentation, constraining deployment across broader clinical settings.

    Source: Limitation and Future Work · p. 12

Abstract

from arXiv · show

Recent advances in segmentation foundation models have enabled accurate and efficient segmentation across a wide range of natural images and videos, but their utility to medical data remains unclear. In this work, we first present a comprehensive benchmarking of the Segment Anything Model 2 (SAM2) across 11 medical image modalities and videos and point out its strengths and weaknesses by comparing it to SAM1 and MedSAM. Then, we develop a transfer learning pipeline and demonstrate SAM2 can be quickly adapted to medical domain by fine-tuning. Furthermore, we implement SAM2 as a 3D slicer plugin and Gradio API for efficient 3D image and video segmentation. The code has been made publicly available at \url{https://github.com/bowang-lab/MedSAM}.

1 Introduction

Segmentation foundation models offer broad generalization, but their performance and usability on medical images, especially 3D data and videos, remain uncertain. The paper benchmarks SAM2 against SAM1 and MedSAM and introduces fine-tuning and deployment tools for medical use.

1 Introduction

  • Specialist models target particular structures, diseases, or modalities, whereas foundation models are trained on diverse data and can generalize across objects and scenarios.The passage also describes zero-shot generalization to unseen domains.Source: §1 · p. 1
  • SAM was trained on natural images and failed to segment medical images, whose domain gap includes low resolution and limited image quality.The passage notes that SAM can be extended to the medical domain through transfer learning.Source: §1 · pp. 1–2
  • Prior work extended SAM toward 3D segmentation, while MedSAM's image-by-image prompting was described as inefficient for practical 3D and video use.Approaches cited include 3D adapters, 3D image encoders, mixture-of-experts models, and text-guided segmentation.Source: §1 · p. 2
  • SAM2's medical-domain performance was unclear, with only one prior medical dataset evaluated and the official interface limited to short videos and few medical data formats.The prior dataset concerned instrument segmentation in endoscopy videos.Source: §1 · p. 3
  • The paper benchmarks SAM2 across ten medical modalities and 2D, 3D, and video data, comparing it with SAM1 and MedSAM.It also develops a transfer-learning pipeline and integrates SAM2 into a 3D Slicer plugin and Gradio interface.Source: Figure 1 · p. 2, §1 · p. 3

2 Results

The evaluation spans 11 medical modalities across 2D images, 3D scans, and videos, comparing SAM2 with SAM1 and MedSAM. Results vary by modality and task; the paper also tests fine-tuning and provides interfaces for medical image and video segmentation.

2.1 Datasets and evaluation protocol

  • The benchmark covers 11 modalities, with all represented in 2D, CT, MR, and PET in 3D, and ultrasound and endoscopy in videos.The study compared SAM and SAM2 model sizes with MedSAM, using bounding-box prompts; for 3D, SAM2 propagated masks from the middle slice while SAM and MedSAM proceeded slice by slice.Source: §2.1 · pp. 3–4

2.2 Evaluation results on 2D image segmentation

  • MedSAM outperformed SAM1 and SAM2 in nine of 11 2D modalities; SAM2 exceeded SAM1 in MR, dermoscopy, and light microscopy, but trailed in PET and OCT.Their performance was comparable in CT, X-Ray, ultrasound, endoscopy, fundus, and mammography.Source: §2.2 · p. 4, Table 1 · p. 4, Figure 2 · p. 5

2.3 Evaluation results on 3D images

  • SAM2-Base improved over slice-by-slice SAM1 and MedSAM on 3D CT and MR, but all SAM1 models outperformed SAM2 on PET.The PET disadvantage was associated with over-segmentation in the middle slice that propagated to the remaining slices.Source: §2.3 · p. 5, Table 2 · p. 6, Figure 3 · p. 7

2.4 Effects of mask initialization on 3D segmentation performance

  • Using a MedSAM-generated middle-slice mask improved propagated 3D segmentation across CT, MR, and PET, with larger gains from ground-truth initialization.The comparison tested SAM2-generated, MedSAM-generated, and ground-truth middle-slice masks.Source: §2.4 · p. 5, Table 3 · p. 6

2.5 Evaluation results on video segmentation

  • SAM2-T achieved a DSC of 0.8537 on ultrasound videos, while SAM-B had the highest endoscopy-video DSC at 0.8397.Failure cases included first-frame segmentation failures and over-segmentation during propagation when boundaries were unclear or image contrast was low.Source: §2.5 · p. 7, Table 4 · p. 7, Figure 4 · p. 8

2.6 Transfer learning: a case study on adapting SAM2 to the medical domain

  • Fine-tuning SAM2-T on 2D abdominal CT slices improved DSC and NSD in 3D abdominal-organ segmentation with middle-slice initialization and slice-wise propagation.Inference used the fine-tuned model in 3D after training on 2D CT slices.Source: §2.6 · p. 8, Table 5 · p. 9

2.7 Deployment: 3D Slicer Plugin and Gradio API for effective medical data annotation

  • The authors developed a 3D Slicer plugin for 3D medical images and a Gradio API for videos, supporting prompts and mask propagation or tracking.The Gradio API allows video uploads without time limitations and can be deployed on users’ computing resources.Source: §2.7 · pp. 8–9

3 Discussion

SAM2's medical segmentation performance varies by modality and model size, while fine-tuning can improve task-specific results at a cost to its original versatility. The paper also identifies coverage and deployment constraints and provides tools for medical image and video annotation.

Model Variants: SAM1 versus SAM2

  • SAM2 outperforms SAM1 on 2D MRI, dermoscopy, light microscopy, and 3D CT/MRI, but SAM1 performs better on 2D OCT and PET.The 3D CT/MRI comparison concerns SAM2-B, which outperformed all SAM1 models by a large margin.Source: Model Variants: SAM1 versus SAM2 · pp. 9, 11

Scaling Laws: Model Size versus Performance

  • SAM2-L does not lead in 8 of 11 2D modalities or any 3D modalities and videos; SAM-T leads on 3D PET, light microscopy, and ultrasound video.The authors suggest training data characteristics and protocols may also matter for performance.Source: Scaling Laws: Model Size versus… · p. 11

General-purpose Model (SAM2) versus Adapted Model in Medical

  • SAM2's video capability improves 3D segmentation, but SAM2 remains inferior to MedSAM on most 2D modalities; fine-tuning can improve performance while compromising its original versatility.The authors recommend balancing task-specific gains against maintaining general segmentation ability.Source: General-purpose Model (SAM2) versus… · p. 11

Limitation and Future Work

  • The benchmark covers common modalities but could add 3D ultrasound and OCT; SAM2 video segmentation often fails on targets without clear boundaries, and long videos require substantial GPU RAM.The authors call for smaller models and faster inference for deployment across broader clinical settings.Source: Limitation and Future Work · pp. 11–12
  • The authors provide a fine-tuning pipeline and deploy SAM2 through a 3D Slicer plugin and Gradio API for medical image and video annotation.They describe these engineering implementations as important for clinical-practice adoption and make the code publicly available.Source: Figure 5 · p. 10, Limitation and Future Work · p. 12

Methods

The methods evaluate public medical datasets after modality-specific preprocessing, describe SAM2’s temporal and multiscale architecture, and fine-tune SAM2-Tiny on abdomen CT. Evaluation uses region- and boundary-overlap metrics.

Data sources and preprocessing

  • The benchmark uses public datasets, applies modality-specific intensity preprocessing to CT, MR, and PET, and converts all images to npz for batch inference.CT uses typical window-level and window-width cutoffs; MR and PET intensities are clipped to the 0.5th and 99.5th percentiles of non-zero intensities, then normalized to [0, 255].Source: Data sources and preprocessing · p. 12

Key differences between SAM1 and SAM2 in methodology

  • SAM2 replaces SAM1’s ViT with Hiera for multiscale features and uses memory attention to condition current-frame predictions on prior-frame information.The memory bank supports consistent segmentation across video frames using temporal context.Source: Key differences between SAM1 and SAM2… · p. 12

Fine-tunig protocol

  • Fine-tuning updates SAM2-Tiny’s image encoder and mask decoder on FLARE22 abdomen CT while freezing the domain-agnostic prompt encoder.Training used perturbed ground-truth bounding boxes, an unweighted Dice and cross-entropy loss sum, and manual early stopping when loss plateaued.Source: Fine-tunig protocol · p. 13

Evaluaiton metrics

  • Evaluation uses Dice Similarity Coefficient for region overlap and Normalized Surface Distance for boundary overlap.Source: Evaluaiton metrics · p. 13

Related papers

Loading 2408.03322v1…