PaperlayerView exact revision on arXiv

Source-linked AI summary

Medical SAM 2: Segment medical images as video via Segment Anything Model 2

Jiayuan Zhu, Abdullah Hamdi, Yunli Qi, Yueming Jin, Junde Wu

arXiv:2408.00874v2cs.CV

TL;DR

Medical image segmentation still faces limited generalization across targets and modalities, especially when applying 2D-oriented architectures to 3D data. MedSAM-2 treats medical images as video and uses a self-sorting memory bank to select informative embeddings without relying on temporal order. Across 14 benchmarks and 25 tasks, it consistently outperformed state-of-the-art models in 2D and 3D segmentation while reducing continuous user interaction.

  • Problem

    Medical segmentation models often struggle to generalize across targets or modalities, while many architectures are designed for 2D images despite frequent 3D medical data and SAM requires interaction for each image.

  • Method

    MedSAM-2 treats medical images as videos and uses a self-sorting memory bank that selects informative embeddings by confidence and dissimilarity, handling unordered 2D and 3D images.

  • Results

    Across 14 benchmarks and 25 tasks, MedSAM-2 consistently outperformed state-of-the-art models in both 2D and 3D medical image segmentation.

  • Takeaways & Limitations

    One-Prompt Segmentation extends a single prompt across multiple unordered 2D images, while the model reduces the need for continuous user interaction in clinical settings.

  • Takeaways & Limitations

    SAM 2 relies on temporal relationships between frames, limiting its applicability to unordered medical images and leaving generalization challenges insufficiently addressed.

Abstract

from arXiv · show

Medical image segmentation plays a pivotal role in clinical diagnostics and treatment planning, yet existing models often face challenges in generalization and in handling both 2D and 3D data uniformly. In this paper, we introduce Medical SAM 2 (MedSAM-2), a generalized auto-tracking model for universal 2D and 3D medical image segmentation. The core concept is to leverage the Segment Anything Model 2 (SAM2) pipeline to treat all 2D and 3D medical segmentation tasks as a video object tracking problem. To put it into practice, we propose a novel \emph{self-sorting memory bank} mechanism that dynamically selects informative embeddings based on confidence and dissimilarity, regardless of temporal order. This mechanism not only significantly improves performance in 3D medical image segmentation but also unlocks a \emph{One-Prompt Segmentation} capability for 2D images, allowing segmentation across multiple images from a single prompt without temporal relationships. We evaluated MedSAM-2 on five 2D tasks and nine 3D tasks, including white blood cells, optic cups, retinal vessels, mandibles, coronary arteries, kidney tumors, liver tumors, breast cancer, nasopharynx cancer, vestibular schwannoma, mediastinal lymph nodules, cerebral artery, inferior alveolar nerve, and abdominal organs, comparing it against state-of-the-art (SOTA) models in task-tailored, general and interactive segmentation settings. Our findings demonstrate that MedSAM-2 surpasses a wide range of existing models and updates new SOTA on several benchmarks. The code is released on the project page: https://supermedintel.github.io/Medical-SAM2/.

1. Introduction

Medical image segmentation remains limited by weak cross-target generalization, the mismatch between 2D architectures and 3D data, and the interaction burden of existing promptable models. MedSAM-2 reframes medical images as videos and adds a self-sorting memory bank to support unified 2D/3D segmentation with fewer prompts.

  • Medical segmentation supports diagnosis, treatment planning, and image-guided surgery, but models trained for specific targets or modalities often generalize poorly.
  • Architectures designed for 2D images face a gap when applied to 3D CT or MRI data.
  • SAM provides zero-shot promptable segmentation, but requiring interaction for every image is labor-intensive for large clinical datasets.
  • SAM 2 reduces interaction through video tracking but depends on temporal relationships, limiting use on unordered medical images and leaving generalization challenges unresolved.
  • MedSAM-2 treats 3D medical images and 2D image flows as videos, using a self-sorting memory bank to select confident embeddings and enable segmentation across images without temporal relationships.
  • MedSAM-2 was evaluated across 14 benchmarks and 25 tasks, with reported superior performance against fully supervised and SAM-based interactive models in 2D and 3D segmentation.

3. Method

MedSAM-2 adapts SAM 2 for unordered medical images by replacing temporal memory with a self-sorting memory bank. The unified pipeline supports 2D and 3D segmentation while using confidence, dissimilarity, and similarity to select relevant embeddings.

  • Self-sorting memory: Because medical-image order may be arbitrary, MedSAM-2 replaces temporal memory with a self-sorting memory bank.This design addresses unordered 2D images and varying slice orders or orientations in 3D imaging.
  • Memory update: The memory bank adds a previous embedding only when its predicted IoU confidence meets a threshold, then retains the K most dissimilar embeddings.Candidate embeddings are formed from the existing bank and the previous frame; embeddings with insufficient confidence leave the bank unchanged.
  • Memory resampling: Before attention, the bank is resampled with replacement so embeddings more similar to the current feature receive higher selection probability.The resampled bank contains K independently selected embeddings and is then used in the MedSAM-2 segmentation pipeline.
  • Unified 2D and 3D processing: The resulting unified framework supports One-Prompt Segmentation for 2D images and processes 3D volumes as slice sequences across multiple orientations.For 3D data, volumes are treated as sequences of 2D slices along defined orientations.

5. Reverse Coronal: X(5) = {xt = V(:, H −t+1, :)}H

MedSAM-2 treats medical images as video sequences and uses a self-sorting memory bank to handle unordered orientations, supporting unified 2D and 3D segmentation. It was evaluated across diverse datasets and tasks, including BTCV 3D segmentation and one-prompt 2D segmentation.

  • Reverse Coronal: MedSAM-2 orders embeddings from different orientations using mean direction features and confidence, jointly capturing 3D context.The self-sorting mechanism avoids relying on a predetermined direction order.
  • Reverse Coronal: During inference, MedSAM-2 aggregates predictions from multiple orientations to produce the final 3D volume segmentation.Aggregation can use pixel-wise averaging or majority voting.
  • Reverse Coronal: For independent 2D images, MedSAM-2 creates pseudo-video sequences and propagates segmentation from one prompted template across the image set.The self-sorting memory bank associates prompts with intrinsic frame features rather than temporal order.
  • Dataset: The One-Prompt dataset contains 78 datasets spanning multiple medical domains, modalities, organs, and clinician-annotated prompts.The dataset includes examples from lung, eye, brain, and abdominal imaging.
  • Dataset: The default split uses 64 datasets for training and 14 for testing, with 11 of the test tasks reserved for zero-shot evaluation.The test set includes challenge tasks and structures such as tumors, vessels, mandibles, and abdominal organs.
  • Implementation: Human prompting was simulated by 15 users spanning laypersons, junior clinicians, and senior clinicians on approximately 10% of test images.This setup was intended to reflect clinical training or semi-automated annotation scenarios.

5. Results

MedSAM-2 performs strongly across 2D and 3D medical segmentation benchmarks, including under reduced prompting and one-prompt evaluation. Its self-sorting memory bank produces more diverse embeddings and supports robust performance with less interaction.

  • 3D medical images: MedSAM-2 achieves a Dice score of 89.0% on BTCV, outperforming all compared methods.The comparison includes task-specific, diffusion-based, interactive, and auto-tracking models.
  • 2D medical images: 2.5% and 2.9%: MedSAM-2 improves Dice scores on optic disc and brain tumor segmentation over previous best models.It maintains its lead over interactive models that require prompts for each image.
  • One-Prompt Segmentation: MedSAM-2 consistently achieves higher average performance and lower variance than one-prompt and few-shot baselines across tested tasks.Performance was evaluated five times using different prompted images and input sequences.
  • Memory-bank analysis: 1.43: total mutual information decreases from 2.54 during memory-bank evolution, indicating increasingly diverse and representative embeddings.The analysis uses the ISIC dataset and interprets lower mutual information as reduced redundancy.
  • Prompt-frequency analysis: 2% versus 7.5%: on 3D images, the performance gap between 5% and full prompting is smaller for MedSAM-2 than for SAM 2.On 2D images, the corresponding gaps are 3.5% for MedSAM-2 and 33.1% for SAM 2.

6. Conclusion

MedSAM-2 treats 2D and 3D medical images as video-like tracking problems and uses a self-sorting memory bank to handle unordered images. Across 14 benchmarks and 25 tasks, it outperforms state-of-the-art models while reducing continuous user interaction.

  • Conclusion: MedSAM-2 is a generalized auto-tracking model for both 2D and 3D medical image segmentation.It treats medical images as videos and incorporates a self-sorting memory bank to handle unordered images.
  • Conclusion: One-Prompt Segmentation generalizes from a single prompt to similar structures across multiple images without temporal relationships.This capability follows from the model's self-sorting memory mechanism.
  • Conclusion: Across 14 benchmarks and 25 tasks, MedSAM-2 consistently outperforms state-of-the-art models in both 2D and 3D segmentation.The reported advantage is accompanied by reduced need for continuous user interaction.

A. Why Does Self-Sorting Work?

The self-sorting memory bank is motivated by information theory: selecting informative and diverse embeddings can increase mutual information and improve segmentation for unordered medical images.

  • Mutual information and entropy provide the information-theoretic framework for analyzing the self-sorting memory bank.The analysis focuses on the diversity of stored embeddings and their information about segmentation outputs.
  • Given the input image, increasing mutual information between the selected memory bank and the predicted mask is expected to improve the prediction.The image features and query remain constant, leaving the selected memory bank as the variable factor.
  • High-confidence embeddings can increase mutual information with the ground truth by reducing the conditional entropy of the ground truth given the memory bank and input.The mechanism treats confidence as an indicator that an embedding carries more information about the output.
  • Selecting embeddings across all past frames rather than only recent frames increases variability in the memory bank and can reduce redundant information.This is especially relevant when image content changes rapidly and substantially.
  • The resulting use of informative and diverse embeddings supports unordered medical-image segmentation and improves performance across diverse tasks after standard segmentation-loss training.

B. Experimental Details

The experiments evaluate segmentation quality with overlap and boundary metrics, emphasizing both region agreement and boundary discrepancies while accounting for outliers.

  • Evaluation Metrics: IoU and Dice Score are used to assess medical-image segmentation performance.
  • Evaluation Metrics: IoU measures the overlap between predicted and ground-truth segmentation divided by their union.
  • Evaluation Metrics: Dice Score measures segmentation similarity as twice the overlap divided by the total predicted and ground-truth areas.
  • Evaluation Metrics: A Dice Score of 1 indicates perfect agreement, and the metric is robust to variability in segmented-object size.
  • Evaluation Metrics: Hausdorff Distance measures the greatest nearest-point boundary discrepancy between prediction and ground truth.HD95 instead uses the 95th percentile of distances, making it less sensitive to outliers.

C.1. Data Preprocessing

The preprocessing pipeline converts heterogeneous medical images into standardized representations through format conversion, intensity normalization, resizing, mask processing, and augmentation.

  • Format and intensity standardization: CT and MRI data are converted to NIfTI, while grayscale and RGB modalities are standardized into NIfTI or PNG formats respectively.For multi-target tasks, each segmentation target is handled individually.
  • Format and intensity standardization: Image intensities are normalized independently for each image to harmonize the substantially different value ranges across modalities.
  • Spatial standardization: Images are cropped to non-zero regions and resized to 256 × 256, using bicubic interpolation for images and nearest-neighbor interpolation for masks.For 3D data, slice extraction generally uses the two highest-resolution axes.
  • Mask processing: Multi-class masks are split into individual class masks, connected components are separated when necessary, and targets smaller than 0.153% of the image are excluded.The exclusion threshold corresponds to fewer than 100 pixels at 256 × 256 resolution.
  • Data augmentation: Training uses stochastic spatial and intensity augmentations, including rotation, scaling, noise, blur, contrast and intensity changes, gamma correction, and flipping.Examples include rotation and scaling probabilities of 0.15 and spatial flipping probability of 0.5.
Loading 2408.00874v2…