Source-linked AI summary
MedSAM3: Delving into Segment Anything with Medical Concepts
Anglin Liu, Rundong Xue, Xu R. Cao, Yifan Shen, Yi Lu, Xiang Li, Qianqian Chen, Jintai Chen
TL;DR
Medical segmentation needs more generalizable models because existing approaches require extensive manual annotation and struggle across clinical applications. MedSAM-3 fine-tunes SAM 3 on curated medical concept labels for text- and visual-prompted segmentation, then adds an MLLM-based agent for iterative reasoning and refinement. The paper reports state-of-the-art generalist medical segmentation, with fine-tuning providing the foundation for agentic improvements.
Problem
Existing medical segmentation methods lack generalizability and require extensive, time-consuming manual annotation for new clinical applications.
Method
MedSAM-3 fine-tunes SAM 3 on paired medical images and concise concept phrases, while the MedSAM-3 Agent adds MLLM-based planning and iterative feedback.
Results
MedSAM-3 establishes state-of-the-art generalist medical segmentation across diverse modalities, while the agent improves usability through iterative feedback.
Takeaways & Limitations
Domain adaptation provides the fundamental segmentation quality, while the agent uses reasoning to correct errors and optimize prompts.
Takeaways & Limitations
Fine-tuning is constrained by limited medical-dataset scale and scarce high-quality data containing rich clinical terminology and domain-specific descriptions.
Abstract
from arXiv · showhide
Medical image segmentation is fundamental for biomedical discovery. Existing methods lack generalizability and demand extensive, time-consuming manual annotation for new clinical application. Here, we propose MedSAM-3, a text promptable medical segmentation model for medical image and video segmentation. By fine-tuning the Segment Anything Model (SAM) 3 architecture on medical images paired with semantic conceptual labels, our MedSAM-3 enables medical Promptable Concept Segmentation (PCS), allowing precise targeting of anatomical structures via open-vocabulary text descriptions rather than solely geometric prompts. We further introduce the MedSAM-3 Agent, a framework that integrates Multimodal Large Language Models (MLLMs) to perform complex reasoning and iterative refinement in an agent-in-the-loop workflow. Comprehensive experiments across diverse medical imaging modalities, including X-ray, MRI, Ultrasound, CT, and video, demonstrate that our approach significantly outperforms existing specialist and foundation models. We will release our code and model at https://github.com/Joey-S-Liu/MedSAM3.
1. Introduction
MedSAM-3 adapts SAM 3 for clinically aligned, concept-driven medical segmentation, addressing limited generalizability and costly manual annotation. It combines semantic prompting with an agentic extension and is reported to establish state-of-the-art generalist medical segmentation while streamlining annotation.
- Motivation: Existing specialist models lack generalizability and require extensive, time-consuming manual annotation for each new clinical application.Medical segmentation remains important for diagnosis, treatment planning, and disease monitoring.
- Motivation: SAM 3 introduces open-vocabulary concept prompts, creating an opportunity to align medical segmentation with conceptual clinical language.Examples include descriptions of tumors, edema, or enlarged lymph nodes rather than only geometric cues.
- Contribution: The MedSAM-3 Agent extends the model with an MLLM-based agent-in-the-loop framework for complex clinical instructions and iterative refinement.The framework is presented as a way to improve usability through reasoning and feedback.
- Results: Experiments across diverse medical imaging modalities demonstrate state-of-the-art generalist medical segmentation and substantially streamlined clinical annotation.The paper frames these results as supporting more intelligent, collaborative medical AI systems.
2. Related Works
Medical segmentation has progressed from specialist CNN and Transformer systems toward foundation models, but semantic understanding remains limited. Existing text-guided approaches are constrained by fixed vocabularies or insufficient reasoning for complex, open-ended clinical concepts.
- Medical Image Segmentation: Specialist CNN, Transformer, and hybrid architectures generally require task- or modality-specific development and often rely on geometric prompting.Earlier systems include U-Net variants and models extended to three-dimensional volumetric data.
- Semantic Guidance: Recent text-guided systems attempt semantic prompting but remain limited by fixed vocabularies or inadequate reasoning over open-ended clinical concepts.These limitations motivate more agentic architectures.
3. Methodology
MedSAM-3 unifies visual and concept prompting for medical image and video segmentation, using a SAM 3-based architecture fine-tuned with curated medical concept labels. Its MedSAM-3 Agent adds multimodal planning, iterative feedback, and refinement for complex segmentation workflows.
- Enabling Medical Concepts in SAM 3: MedSAM-3 supports both Promptable Visual Segmentation and Promptable Concept Segmentation for medical images and videos.Visual prompts include points, boxes, or masks, while concept prompts use short medical noun phrases.
- Architecture: The model uses a dual encoder-decoder transformer with a detector for images and a tracker-memory module for video segmentation.The tracker conditions current-frame features on prior predictions through streaming memory.
- Supervised Fine-Tuning with Medical Concepts: Fine-tuning freezes the image and text encoders while adapting detector components on paired medical images and concise, curated concept phrases.Concept phrases are limited to three words and selected from official dataset documentation or repository descriptions.
- Scalable Medical Segmentation Agent: The MedSAM-3 Agent uses an MLLM or medical VLM to plan multistep workflows, invoke MedSAM-3, inspect outputs, and revise actions iteratively.The process continues until the agent determines that the user’s goal is satisfied or no valid mask exists.
4. Experiments and Results
Experiments evaluate MedSAM-3 across diverse medical datasets and prompting settings, showing stronger performance with text-and-image inputs than text alone. The MedSAM-3 Agent further improves BUSI segmentation through iterative multimodal feedback.
- Datasets: The evaluation spans medical datasets covering multiple imaging modalities and dimensions, including 2D, 3D, and video.The dataset collection includes modalities such as X-ray, MRI, ultrasound, OCT, fundus, dermoscopy, histopathology, nuclear imaging, infrared, endoscopy, and CT.
- Experimental settings: The experiments compare classical 2D and 3D segmentation networks with SAM-based methods under text-only and text-plus-image prompting conditions.For 2D settings, text-plus-image inputs combine a short target phrase with a bounding-box reference; 3D experiments use text phrases only.
- Experimental settings: MedSAM-3 was fine-tuned on BUSI, RIM-ONE(Cup), ISIC 2018, and Kvasir-SEG, with adaptation focused exclusively on the SAM 3 detector module.The experiments also included MedSAM as a prompt-based medical segmentation comparator.
- MedSAM-3 Performance: MedSAM-3 T+I achieves the best results on all four benchmarks, whereas text-only MedSAM-3 T has limitations across every dataset.The comparison is summarized in Figure 4 and Table 2.
- MedSAM-3 Performance: Qualitative results show coherent segmentation across BUSI, RIM-ONE (Cup), ISIC 2018, and Kvasir-SEG, including challenging low-contrast or irregular-boundary regions.The reported improvements use only a small amount of domain-specific fine-tuning data.
- MedSAM-3 Agent Performance: The MedSAM-3 Agent improves BUSI Dice from 0.7772 to 0.8064 after Gemini 3 Pro orchestrates query interpretation and three rounds of iterative feedback.The agent was evaluated on the BUSI test set.
5. Discussion
Discussion shows that off-the-shelf SAM 3 transfers inconsistently to medical segmentation, while domain-specific adaptation and agentic refinement improve medical concept alignment and reliability.
- SAM 3 Performance: SAM 3 performs inconsistently across medical tasks, with large gaps versus established baselines in 2D, video, and 3D segmentation.Its performance is highly uneven across 2D/video datasets and consistently lower than specialist methods on the evaluated 3D datasets.
- SAM 3 Performance: Bounding-box guidance substantially improves SAM 3 Dice scores, sometimes approaching or surpassing conventional supervised methods.The result indicates that geometric cues compensate for limited fine-grained medical semantic understanding.
- SAM 3 Limitations: SAM 3 can misalign concepts with anatomical regions, such as segmenting lungs for “liver” and non-lesion regions for “lesion.”These failures illustrate systematic concept–region misalignment rather than isolated segmentation errors.
- MedSAM-3 and Agentic Adaptation: Curated medical concept fine-tuning improves MedSAM-3’s concept alignment and segmentation reliability across modalities, while broader concept-annotated datasets remain needed.The discussion describes the gains as meaningful but constrained by limited medical-data scale and scarce high-quality clinical terminology annotations.
- MedSAM-3 and Agentic Adaptation: The MedSAM-3 Agent improves complex-instruction handling through iterative reasoning, prompt refinement, and error correction, but depends critically on a semantically aligned base model.Agentic refinement is effective when MedSAM-3 supplies a stable medical foundation and less effective when the underlying segmentation is misaligned.
6. Conclusion
The conclusion presents MedSAM-3 as a domain-adapted extension of SAM 3 for medical concept grounding, with an agentic framework that improves usability through iterative feedback. It identifies concept granularity and text–image alignment as areas for future work.
- Conclusion: MedSAM-3 extends SAM 3 through domain-specific fine-tuning for medical concept grounding, outperforming the original model on complex medical semantics and temporal consistency.The conclusion positions MedSAM-3 as the foundation for reliable medical segmentation before agentic refinement.
- Conclusion: The MedSAM-3 Agent uses iterative feedback to improve usability, while reasoning and prompt optimization raise the performance ceiling above the backbone.The conclusion distinguishes foundational segmentation quality from the agent’s subsequent error correction and prompt refinement.
- Future Work: Future work targets limitations in concept granularity and text–image alignment to scale MedSAM3 toward more clinical applications.The authors also state that code and models will be released to support the community.