PaperlayerView exact revision on arXiv

Source-linked AI summary

Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions

Yichi Zhang, Zhenrong Shen, Rushi Jiao

arXiv:2401.03495v1eess.IVcs.CV

TL;DR

Medical image segmentation requires models that generalize across modalities and targets, but SAM’s transfer from natural images remains uncertain. This survey synthesizes zero-shot evaluations and adaptation methods, finding that direct application is generally unsatisfactory on multi-modal and multi-target medical datasets. The review identifies adaptation strategies and larger medical datasets as important directions for improving medical segmentation foundation models.

  • Problem

    It remains unclear whether SAM can effectively segment medical images because natural and medical images differ substantially.

  • Method

    The paper surveys recent SAM applications to medical segmentation, covering zero-shot benchmarking, adaptation methods, challenges, and future research directions.

  • Results

    Direct SAM application yields limited and highly variable generalization across medical datasets and tasks, often requiring substantial human interaction for moderate performance.

  • Takeaways & Limitations

    Adaptation strategies can improve SAM and achieve competitive performance with task-specific models, while broader medical datasets may support future universal segmentation foundation models.

  • Takeaways & Limitations

    Fine-tuning on a specific medical dataset can improve performance, but generalization remains constrained on unseen tasks.

Abstract

from arXiv · show

Due to the inherent flexibility of prompting, foundation models have emerged as the predominant force in the fields of natural language processing and computer vision. The recent introduction of the Segment Anything Model (SAM) signifies a noteworthy expansion of the prompt-driven paradigm into the domain of image segmentation, thereby introducing a plethora of previously unexplored capabilities. However, the viability of its application to medical image segmentation remains uncertain, given the substantial distinctions between natural and medical images. In this work, we provide a comprehensive overview of recent endeavors aimed at extending the efficacy of SAM to medical image segmentation tasks, encompassing both empirical benchmarking and methodological adaptations. Additionally, we explore potential avenues for future research directions in SAM's role within medical image segmentation. While direct application of SAM to medical image segmentation does not yield satisfactory performance on multi-modal and multi-target medical datasets so far, numerous insights gleaned from these efforts serve as valuable guidance for shaping the trajectory of foundational models in the realm of medical image analysis. To support ongoing research endeavors, we maintain an active repository that contains an up-to-date paper list and a succinct summary of open-source projects at https://github.com/YichiZhang98/SAM4MIS.

1. Introduction

Medical image segmentation is clinically important, but it remains unclear whether foundation models such as SAM can transfer effectively from natural to medical images. This survey reviews zero-shot evaluations, adaptation methods, challenges, and future directions.

  • Medical image segmentation identifies anatomical structures, lesions, and tissues to support diagnosis, treatment planning, and disease monitoring.Accurate segmentation provides volumetric and shape information for downstream clinical applications.
  • Foundation models offer zero-shot and few-shot generalization, motivating their adaptation across diverse medical imaging modalities.
  • SAM extends prompt-driven foundation modeling to image segmentation, but its medical-image applicability remains uncertain because natural and medical images differ substantially.
  • The survey organizes recent work into zero-shot evaluation with different prompts and methodological adaptation, then outlines challenges and future directions.It also maintains an updated paper list and open-source project summary.

2. Background

Foundation models learn transferable representations from large-scale data, while SAM applies this paradigm to promptable image segmentation. Its architecture combines image embeddings, prompt embeddings, and mask decoding, trained with interactive and automated data-engine processes.

  • 2.1. Foundation Models: Foundation models are large-scale, general-purpose language and vision models trained on massive datasets to learn transferable representations.
  • 2.2. Segment Anything Model: SAM is trained on the SA-1B dataset and uses a transformer-based architecture designed for zero-shot image segmentation.
  • 2.2. Segment Anything Model: SAM’s image encoder extracts image embeddings, its prompt encoder integrates points, boxes, masks, and other interactions, and its mask decoder predicts segmentation masks.
  • 2.2. Segment Anything Model: The mask decoder uses transformer layers with dynamic mask prediction and IoU-score regression heads to produce masks at reduced resolution.
  • 2.2. Segment Anything Model: SAM is trained with focal and Dice losses on geometric prompts using a data engine that combines interactive annotation, manual labeling, and confident pseudo-masks.

3. SAM’s Zero-shot Evaluations on Medical Image Segmentation

Zero-shot SAM performance varies widely across medical datasets, modalities, and targets. It can perform competitively on some well-circumscribed structures, but generally struggles with difficult targets and often requires substantial prompting.

  • Zero-shot medical segmentation studies examine SAM across diverse anatomical targets, pathological structures, and imaging modalities, including CT, MRI, X-ray, ultrasound, and endoscopy.
  • 3.2. MRI Image Segmentation: SAM can match or exceed a brain-extraction baseline on some MRI scans, yet remains behind current SOTA methods for brain-tumor segmentation without fine-tuning.
  • 3.5. Endoscopic Image Segmentation: SAM performs poorly on challenging endoscopic scenes, including overlapping instruments and images containing blood, reflection, blur, or shade.
  • Box prompts generally outperform point prompts because they provide more accurate location information, although large boxes can confuse segmentation when similar instances are nearby.
  • 3.7. Summary: Across datasets, SAM typically requires substantial human interaction for moderate performance, with limited and highly variable generalization across tasks.
  • 3.7. Summary: SAM is competitive for well-circumscribed objects but often fails on targets with weak boundaries, low contrast, small size, or irregular shapes.

4. Adapting SAM to Medical Image Segmentation

Recent efforts adapt SAM to medical image segmentation by modifying its training strategy and selectively fine-tuning components. These approaches target SAM’s limited direct-transfer performance across diverse medical data.

  • Adaptation efforts fine-tune different SAM modules or train SAM-like architectures to accommodate diverse medical imaging characteristics and complexities.The survey identifies adaptation as an alternative to zero-shot transfer, covering both 2D and 3D modalities.
  • MedSAM freezes the prompt encoder while fine-tuning the image encoder and mask decoder for medical image segmentation.
  • Medical SAM Adapter adapts SAM in a parameter-efficient way for medical image segmentation.

4.1. Fine-tuning on Medical Images

Fine-tuning improves SAM’s medical segmentation performance, but researchers increasingly favor parameter-efficient strategies because updating the full model is computationally demanding. The reviewed methods include full fine-tuning, adapters, and prompt-generation pipelines.

  • 4.1. Fine-tuning on Medical Images: Fine-tuning SAM on medical images includes both full fine-tuning and parameter-efficient fine-tuning.
  • 4.1.1. Full Fine-tuning: 81.25% to 88.79%: fine-tuning SAM for skin cancer segmentation substantially improves the DSC score.
  • 4.1.1. Full Fine-tuning: Polyp-SAM fine-tunes all SAM components and achieves DSC scores above 88% on five public datasets.
  • 4.1.2. Parameter-efficient Fine-tuning: Full parameter updates are time-consuming, computationally intensive, and less feasible for widespread deployment.
  • 4.1.2. Parameter-efficient Fine-tuning: Med-SA freezes pretrained SAM parameters and integrates LoRA modules, with experiments spanning 17 tasks across five modalities showing superiority over SAM and previous SOTA methods.
  • 4.1.2. Parameter-efficient Fine-tuning: SAMed updates only a small fraction of SAM parameters while achieving an 81.88% DSC score on the Synapse multi-organ segmentation dataset.
  • 4.1.2. Parameter-efficient Fine-tuning: MedLSAM uses a few-shot localization process to auto-generate 3D bounding boxes enclosing anatomical structures as prompts.
  • 4.1.2. Parameter-efficient Fine-tuning: SAM-Med2D demonstrates significantly superior performance and generalization capability compared to SAM on nine MICCAI 2023 challenge datasets.

4.2. Auto-prompting Adaptation

Auto-prompting adaptations address SAM’s dependence on high-quality manual prompts by generating prompts through localization, auxiliary encoders, or uncertainty estimation. These methods aim to make medical segmentation more automatic and reliable.

  • 4.2. Auto-prompting Adaptation: Existing SAM adaptations often require high-quality points, boxes, or masks, frequently generated from ground truth during testing.
  • 4.2. Auto-prompting Adaptation: Accurate and reliable prompts require domain-specific medical expertise, which may not be available.
  • 4.2.1. Prompts Auto-generation: Localization frameworks generate input prompts by identifying regions of interest or 3D bounding boxes around anatomical structures.
  • 4.2.1. Prompts Auto-generation: DeSAM decomposes the mask decoder into PRIM for generating mask embeddings and PIMM for fusing image embeddings with those mask embeddings.
  • 4.2.2. Learnable Prompts: AutoSAM trains an auxiliary prompt encoder to extract image features as conditional surrogate prompts, eliminating manual prompts.
  • 4.2.2. Learnable Prompts: UR-SAM estimates and uses uncertainty to rectify segmentation results and enhance auto-prompting reliability.
  • 4.2.3. Enhancing Reliability Against Prompts with Uncertainty: EviPrompt uses training-free uncertainty estimation to automatically generate SAM prompts for medical image segmentation.

4.3. Framework Modification

Framework-modification approaches integrate SAM into new training schemes or alter its components for medical segmentation. The reviewed examples add domain-specific prompts and encoders or use SAM to support learning when annotations are scarce.

  • 4.3. Framework Modification: Researchers modify SAM’s framework or integrate it into novel training schemes to construct medical image segmentation models.
  • 4.3. Framework Modification: SemiSAM uses SAM as an additional supervision branch to assist learning in a semi-supervised framework.
  • 4.3. Framework Modification: SAM-Path introduces trainable class prompts and a pretrained pathology encoder to incorporate domain-specific knowledge in digital pathology.
  • 4.3. Framework Modification: SAM-Path improves DSC by 27.52% relative to vanilla SAM with manual prompts on the CRAG dataset.
  • 4.3. Framework Modification: High annotation costs motivate semi-supervised and weakly supervised learning, where SAM can guide segmentation as a pseudo-label generator.

4.4. Towards 3D Medical Images

SAM adaptation for 3D medical images addresses the loss of depth-related spatial context caused by applying its original 2D architecture slice by slice. Approaches range from lightweight 2D-to-3D modifications to fully learnable volumetric architectures.

  • Motivation: Directly applying SAM’s 2D architecture to 3D images can discard depth-related spatial context needed for accurate segmentation.This limitation motivates architectural modifications for volumetric medical image segmentation.
  • 2D-to-3D adaptation: Med-SA captures spatial and depth correlations separately through its bifurcated Space-Depth Transpose attention mechanism.
  • 2D-to-3D adaptation: 16.96% of the original model’s parameters are tunable in 3DSAM-adapter, which significantly outperforms nnU-Net on kidney, pancreas, and colon tumor datasets.The reported improvements are 8.25% for kidney tumor, 29.87% for pancreas tumor, and 10.11% for colon cancer.
  • 2D-to-3D adaptation: MA-SAM injects tunable 3D adapters into each image-encoder transformer block and fine-tunes them with the mask decoder across volumetric and video medical data.Experiments on 10 datasets report consistent improvements over various state-of-the-art 3D approaches without prompts.
  • Fully 3D architecture: SAM-Med3D replaces SAM’s original components with a fully learnable 3D image encoder, 3D prompt encoder, and 3D mask decoder.It is trained on 21K medical images and 131K masks spanning 247 categories, then evaluated on 15 public datasets.

4.5. Summary

SAM adaptation research spans fine-tuning, automated prompting, framework modification, and extension from 2D to 3D medical images. These directions target the diverse modalities and structures encountered in medical segmentation.

  • Adaptation directions: Fine-tuning strategies tailor SAM parameters specifically for medical image segmentation.
  • Adaptation directions: Auto-prompting approaches automate prompt mechanisms to improve SAM’s flexibility and robustness.
  • Adaptation directions: Framework modification methods refine SAM’s architecture or integrate it into new training frameworks for medical segmentation.
  • Adaptation directions: Extending SAM to 3D medical images addresses its original limitation of primarily handling 2D data.

5. Discussion and Conclusion

Direct SAM application remains inconsistent across multimodal and multitarget medical datasets, motivating adaptation, larger datasets, richer prompts, and clinical integration. The reviewed efforts point toward clinically applicable medical foundation models while retaining important performance and annotation constraints.

  • Discussion and Conclusion: Directly applying SAM without adaptation produces significantly varying performance across medical datasets and tasks.The survey identifies difficulty achieving consistent and accurate zero-shot segmentation across multimodal and multitarget datasets.
  • Discussion and Conclusion: SAM often performs suboptimally on irregular, weak-boundary, small, or low-contrast objects where exceptionally high accuracy is required.
  • Discussion and Conclusion: Fine-tuning can improve SAM on a specific medical dataset, but generalization remains constrained on unseen tasks.
  • Building Large-Scale Medical Datasets: SA-Med2D-20M contains 4.6M 2D medical images and 19.7M corresponding masks spanning the human body.The dataset is presented as a large-scale resource for developing universal medical segmentation foundation models.
  • Building Large-Scale Medical Datasets: Medical segmentation datasets are costly to annotate because they require domain expertise, especially for labor-intensive slice-by-slice delineation of 3D volumes.
  • Supporting Medical Image Annotation: SAM-generated masks can accelerate annotation by providing coarse segmentations that experts manually revise, supported by automatic and assistive annotation modules.
  • Incorporating Scribble and Text Prompts: Scribbles, clicks, boxes, and text prompts extend interaction and inject clinical knowledge for targets and tasks across modalities and anatomies.Box prompts often outperform point prompts, while combined scribble and positional prompts suit non-compact structures such as vessels, intestines, and bones.
  • Assisting in More Clinical Applications: Clinical integrations include gaze-based segmentation, ultrasound guidance, label-free diabetic-retinopathy diagnosis, and MRI synthesis or super-resolution.These applications include human-in-the-loop and real-time systems as well as semantic supervision for anatomical preservation.

Declaration of Competing Interest

The authors report no competing interests.

  • Declaration of Competing Interest: The authors have no conflict of interest to disclose.
Loading 2401.03495v1…