Source-linked AI summary
Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation
Chongcong Jiang, Tianxingjian Ding, Chuhan Song, Jiachen Tu, Ziyang Yan, Yihua Shao, Zhenyi Wang, Yuzhang Shang, Tianyu Han, Yu Tian
TL;DR
Medical segmentation models often generalize poorly across heterogeneous domains, while promptable SAM3 can be unstable on medical images without privileged spatial cues. The paper fully fine-tunes SAM3 on diverse text–image–mask data, yielding consistent gains and a semantic-only foundation model, although computational, volumetric, prompt-language, and validation limitations remain.
Problem
Medical segmentation remains difficult to generalize across modalities and clinical sites, while promptable models face unstable performance under medical domain shift and reliance on spatial cues.
Method
Medical SAM3 fully adapts SAM3 on 33 heterogeneous datasets using paired images, masks, and clinical-label-derived text prompts while preserving prompt-driven segmentation.
Results
Medical SAM3 achieves consistent performance gains across diverse organs, modalities, and dimensionalities, including average Dice rising from 54.0% to 77.0% and average IoU from 43.3% to 67.3%.
Takeaways & Limitations
Robust medical promptability depends on holistic representation adaptation and reliable semantic grounding rather than prompt engineering or privileged geometric cues alone.
Takeaways & Limitations
Full high-resolution adaptation is computationally demanding, while planar processing may underuse volumetric continuity and broader validation remains necessary.
Abstract
from arXiv · showhide
Promptable segmentation foundation models such as SAM3 have demonstrated strong generalization capabilities through interactive and concept-based prompting. However, their direct applicability to medical image segmentation remains limited by severe domain shifts, the absence of privileged spatial prompts, and the need to reason over complex anatomical and volumetric structures. Here we present Medical SAM3, a foundation model for universal prompt-driven medical image segmentation, obtained by fully fine-tuning SAM3 on large-scale, heterogeneous 2D and 3D medical imaging datasets with paired segmentation masks and text prompts. Through a systematic analysis of vanilla SAM3, we observe that its performance degrades substantially on medical data, with its apparent competitiveness largely relying on strong geometric priors such as ground-truth-derived bounding boxes. These findings motivate full model adaptation beyond prompt engineering alone. By fine-tuning SAM3's model parameters on 33 datasets spanning 10 medical imaging modalities, Medical SAM3 acquires robust domain-specific representations while preserving prompt-driven flexibility. Extensive experiments across organs, imaging modalities, and dimensionalities demonstrate consistent and significant performance gains, particularly in challenging scenarios characterized by semantic ambiguity, complex morphology, and long-range 3D context. Our results establish Medical SAM3 as a universal, text-guided segmentation foundation model for medical imaging and highlight the importance of holistic model adaptation for achieving robust prompt-driven segmentation under severe domain shift. Code and model will be made available at https://github.com/AIM-Research-Lab/Medical-SAM3.
1 Introduction
Medical segmentation remains difficult to scale across heterogeneous modalities, anatomies, pathologies, and clinical sites because specialist models depend on dense annotation and dataset-specific optimization. Medical SAM3 addresses this gap by holistically adapting SAM3 for universal, text-guided segmentation without privileged spatial prompts.
- Medical segmentation supports disease quantification, treatment planning, and longitudinal follow-up at pixel or voxel level.
- Dataset-specific models struggle to adapt across modalities, anatomies, pathologies, and clinical sites, especially under distribution shift.
- Dense annotation requirements and specialist optimization limit scalability and deployment in rare conditions and heterogeneous real-world settings.
- Promptable foundation models aim to provide a unified interface through prompt-based interaction while reducing task-specific retraining.
- Medical images differ substantially from natural scenes, causing unstable zero-shot or lightly adapted performance for promptable models.
- Medical SAM3 holistically adapts SAM3 using heterogeneous 2D and 3D datasets with paired masks and text prompts, avoiding privileged spatial prompts.
2 Related Works
Prior work spans specialist medical segmentation, text-guided mask prediction, and promptable foundation models. Medical SAM3 builds on these directions while using SAM3’s detector–tracker design for semantic-driven segmentation of 2D and slice-based 3D medical images.
- Specialist Medical Image Segmentation: Specialist medical segmentation relies primarily on fully supervised CNN and transformer models, including 3D architectures for volumetric CT and MRI.
- Text Guided and Open Vocabulary Segmentation: Text-guided segmentation aligns dense visual features with language representations for open-vocabulary masks and grounded target specification.
- Promptable Segmentation Foundation Models: SAM introduced segmentation conditioned on spatial prompts, while SAM 2 added memory for streaming image and video settings.
- Promptable Segmentation Foundation Models: Medical SAM3 accepts text and 2D or slice-based 3D images, then merges detector masks with optional tracker-propagated masks.
- Promptable Segmentation Foundation Models: Medical promptable models extend these ideas to volumetric data through native 3D models, memory mechanisms, and universal prompt-driven multitask learning.
3 Method
Medical SAM3 fully fine-tunes SAM3 without architectural changes on unified text–image–mask samples from heterogeneous medical datasets. Its method combines planar processing, stratified tuning, text-only semantic alignment, and set-prediction losses for instance discovery and precise segmentation.
- 3 Method: Medical SAM3 updates all SAM3 parameters without architectural modifications and follows the detector–tracker design for 2D and 3D inputs.
- 3 Method: The method harmonizes heterogeneous medical modalities in a common 2D feature space to improve applicability across clinical workflows.
- 3.1 Unified Input Formulation: Each training sample is a text-driven triplet (I, M, t), pairing an image, segmentation mask, and clinical-label-derived text prompt.
- 3 Method: Training maintains a 1008 × 1008 resolution to preserve high-frequency spatial priors and synchronize positional embeddings with the perception backbone.
- 3.2 Stratified Tuning: Layer-wise Learning Rate Decay mitigates the natural-to-medical domain gap while reducing catastrophic forgetting during full adaptation.
- 3.2 Stratified Tuning: With L = 12 and γ = 0.85, shallow layers retain general visual primitives while deeper layers specialize in medical semantics.
- 3 Method: Text-only training removes dependence on manual bounding boxes and encourages intrinsic spatial awareness linking clinical concepts to pixel-level morphology.
- 3.4 Set-Prediction Objective: A multi-task objective jointly supervises instance discovery and semantic segmentation using predicted queries and matched ground-truth instances.
4 Experiments
Medical SAM3 is evaluated through a unified text-driven protocol built from diverse medical datasets, standardized concept prompts, and comparisons with the original SAM3. The experiments assess robustness across internal held-out tasks and externally excluded tasks.
- Datasets: Each training sample is structured as an (image, mask, text) triplet using clinical labels or manually curated prompts.This representation pairs segmentation masks with text prompts derived from native clinical nomenclature.
- Datasets: 33 datasets across 10 imaging modalities provide 76,956 images and 263,705 mask annotations for training and validation.The corpus spans modalities including radiography, ultrasound, endoscopy, pathology, fundus, dermoscopy, microscopy, and electron microscopy.
- Evaluation: Evaluation covers 10 internal validation tasks and 7 external segmentation tasks excluded from model development to test robustness under domain shift.The external tasks are entirely withheld from the development pipeline.
- Datasets: A unified vocabulary maps dataset-specific labels to canonical anatomy or pathology concepts for consistent prompting across datasets.Single-class datasets use one global concept, while multi-class datasets map label indices to dataset-defined clinical terms.
- Experimental Settings: Medical SAM3 is compared with the official unadapted SAM3 checkpoint after full parameter fine-tuning from the same initialization.Both models use text-only prompts, and evaluation selects the highest-confidence mask for each prompt before resolving multi-class overlaps.
5 Results
Medical SAM3 consistently outperforms the original SAM3 on internal and external medical segmentation benchmarks under text-only prompting. Improvements are especially substantial for domain-shifted, thin, low-contrast, and visually complex targets.
- Overall Comparison: Medical SAM3’s red radar area exceeds SAM3’s blue area across internal validation and external generalization for both Dice and IoU.The radar comparison aligns with the quantitative results reported in Table 3.
- Internal Validation: 77.0% average Dice and 67.3% average IoU versus 54.0% and 43.3% for SAM3 were achieved across 10 internal held-out tasks.Medical SAM3 improves on every internal task, with especially large gains for small, thin, or low-contrast targets.
- Internal Validation: 43.8% versus 16.3% Dice for breast cancer tissue and 88.2% versus 68.9% Dice for GlaS’15 gland segmentation demonstrate gains in digital pathology.The results indicate adaptation to stain and texture variations, while PAPILA reaches 99.4% Dice and 98.7% IoU.
- External Validation: 73.9% average Dice and 64.4% average IoU versus 11.9% and 8.0% for SAM3 were obtained across seven externally excluded tasks.Endoscopic polyp segmentation reached 87.9% Dice on CVC and 86.1% Dice on ETIS after baseline failure.
- Qualitative Results: Qualitatively, Medical SAM3 recovers cleaner retinal-vessel boundaries, coherent endoscopic masks, and more accurate dermoscopic lesion extents than SAM3.SAM3 commonly produces empty, fragmented, noisy, or over-segmented masks with spurious background activations.
6 Discussion and Conclusion
Medical SAM3 frames universal medical segmentation as an alignment and adaptation problem: reliable semantic grounding under domain shift matters more than prompt interfaces alone. Its discussion emphasizes holistic adaptation, deployment-aware evaluation, standardized prompting, and remaining efficiency, volumetric, language, and validation challenges.
- Semantic grounding, rather than prompt-interface availability, is the main bottleneck for universal medical segmentation under domain shift.Geometric cues can reduce segmentation to boundary refinement, whereas text-only prompting requires mapping clinical concepts to spatially precise masks.
- Consistent gains across modalities indicate that robust text grounding can emerge from holistic representation adaptation instead of prompt engineering alone.The proposed perspective links language concepts with localization-relevant visual features, especially for difficult targets.
- Deployment-consistent evaluation should separate semantic-only settings from interactive settings that provide privileged spatial hints.Otherwise, comparisons may be confounded by localization priors unavailable in realistic semantic-only deployment.
- Clinical deployment remains bounded by computational cost, underused volumetric continuity, limited prompt compositionality, and insufficient multi-center reliability validation.The authors identify parameter-efficient adaptation, native 3D prompting, richer prompts, uncertainty estimation, and broader validation as needed directions.
- Medical SAM3 supports a semantic-driven paradigm in which robust promptability is primarily an alignment and adaptation challenge.Future progress is described as combining scalable multi-domain training, richer clinical language handling, and efficiency-oriented adaptation.