Source-linked AI summary
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, Yixin Wang
TL;DR
Medical MLLMs often lack broad task and modality coverage while falling short of specialized models. MMedAgent addresses this gap with a medical tool-using agent and instruction-tuning dataset, achieving superior performance across medical tasks and efficiently integrating new tools.
Problem
Medical MLLMs have limited generality, often lack specialization, and medical tool-using agents remain insufficiently explored.
Method
MMedAgent uses an instruction-tuning dataset to learn selection and aggregation of six medical tools solving seven tasks across five modalities.
Results
MMedAgent surpasses open-source baselines and GPT-4o across many medical tasks; its overall relative score is 109.48 versus 60.68 for LLaVA-Med.
Takeaways & Limitations
The agent can handle broad medical tasks across modalities while efficiently integrating new tools and retaining previously learned tool-use capabilities.
Takeaways & Limitations
The work is limited to seven tasks across five modalities and may benefit from more specialized tools and stronger medical generalist backbones.
Abstract
from arXiv · showhide
Multi-Modal Large Language Models (MLLMs), despite being successful, exhibit limited generality and often fall short when compared to specialized models. Recently, LLM-based agents have been developed to address these challenges by selecting appropriate specialized models as tools based on user inputs. However, such advancements have not been extensively explored within the medical domain. To bridge this gap, this paper introduces the first agent explicitly designed for the medical field, named \textbf{M}ulti-modal \textbf{Med}ical \textbf{Agent} (MMedAgent). We curate an instruction-tuning dataset comprising six medical tools solving seven tasks across five modalities, enabling the agent to choose the most suitable tools for a given task. Comprehensive experiments demonstrate that MMedAgent achieves superior performance across a variety of medical tasks compared to state-of-the-art open-source methods and even the closed-source model, GPT-4o. Furthermore, MMedAgent exhibits efficiency in updating and integrating new medical tools. Codes and models are all available.
1 Introduction
MMedAgent is introduced as a medical AI agent that selects and combines specialized tools to address the limited generality and specialization of existing medical MLLMs. It spans multiple tasks and modalities, supported by medical tool adaptations and instruction-tuning data.
- Existing medical MLLM generalists handle limited task and modality ranges and typically lack the expert-level responses of specialized models.
- MMedAgent is introduced as the first AI agent specifically designed for the medical domain.
- The agent extends a LLaVA-Med backbone across grounding, segmentation, classification, report generation, and retrieval-augmented generation tasks.
- The instruction-tuning dataset teaches tool selection and aggregation of tool outputs for user requests.
- MMedAgent incorporates adaptive medical tools, including specialized datasets that adapt grounding and segmentation tools to medicine.
- MMedAgent surpasses previous state-of-the-art multimodal medical language models across a range of tasks.
2 Related Work
Prior medical MLLMs are often specialized to particular modalities or tasks, while broader systems face scalability and specialization limitations. Multi-modal agents offer tool-based planning, but MMedAgent extends this paradigm to a broader range of medical modalities.
- 2.1 Medical MLLMs: Medical MLLMs have commonly been localized to specific imaging domains or tailored to particular tasks because medical data and tasks are diverse.
- 2.1 Medical MLLMs: Broad medical MLLMs face limited scalability when adding skills and lack specialization for specific tasks.
- 2.2 AI Agent: Multi-modal AI agents use perception and decision-making to achieve general-purpose goals, with LLMs acting as planners that call different tools.
- 2.2 AI Agent: Fine-tuned-planner agents learn to understand instructions, make decisions, and call tools or APIs; MMedAgent belongs to this type.
- 2.2 AI Agent: Existing multi-modal agents have largely focused on natural domains, whereas MMedAgent covers MRI, CT, X-ray, and histology.
3 MMedAgent
MMedAgent combines an instruction-tuned multimodal LLM with specialized medical tools to plan actions, execute tasks, and aggregate results across diverse modalities. Its six tools cover seven medical tasks, including grounding, segmentation, classification, report generation, retrieval, and VQA.
- 3.1 Workflow: MMedAgent uses an instruction-tuned multimodal LLM as both action planner and results aggregator alongside a collection of specialized medical tools.The agent generates tool calls, receives tool outputs, and produces final answers using the original query and image.
- 3.1 Workflow: The four-step workflow maps user inputs to tool instructions, executes the selected tool, and aggregates its output into the final answer.Training uses an autoregressive objective over the generated tool instruction and final answer.
- 3.2 Instruction Tuning: The instruction-tuning data teaches MMedAgent to determine whether tools are needed, select API calls, and combine tool outputs with its response.The unified dialogue format represents these stages through Thoughts, Actions, and resulting outputs.
- 3.3 Medical Tools: MMedAgent integrates six tools across seven tasks: grounding, two segmentation settings, classification, medical report generation, retrieval-augmented generation, and VQA.The incorporated tasks span medical imaging applications, while RAG operates on natural language without images.
- 3.3.1 Grounding: Grounding uses a fine-tuned Grounding DINO to localize medical objects, supporting multimodal detection across medical imaging domains.The model is adapted using medical and common natural-image datasets to retain general object-detection ability.
- 3.3.2 Other Tasks: Table 2 compares MMedAgent with baseline models using task-level relative scores and an overall absolute score.The table distinguishes models that cannot perform particular tasks and includes a tool-augmented LLaVA-Med comparison.
4 Experimental Settings
MMedAgent is instruction-tuned from LLaVA-Med 60K-IM with LoRA using 48K generated examples and a two-GPU training setup.
- 4 Experimental Settings: MMedAgent is initialized with LLaVA-Med 60K-IM and instruction-tuned with LoRA for 15 epochs.Training uses rank 128, batch size 48, AdamW, and a cosine schedule peaking at 2e-4.
- 4 Experimental Settings: The instruction-tuning dataset contains 48K examples, including augmented VQA data and task-specific detection data.The training run takes approximately 72 hours on two 80G NVIDIA A100 GPUs.
5 Experimentals
Experiments evaluate MMedAgent across diverse medical tasks, open-ended dialogue, VQA, and tool scalability. MMedAgent outperforms strong baselines, including GPT-4o on several tasks, while efficiently selecting and incorporating tools.
- 5.1 Experimental Results: 109.48 overall relative score makes MMedAgent outperform all evaluated state-of-the-art MLLMs, reaching 1.8 times LLaVA-Med’s 60.68.MMedAgent retains its advantage over LLaVA-Med when the latter incorporates internal tool outputs.
- 5.1 Experimental Results: Scores above 100% on organ grounding, disease grounding, and MRG indicate that MMedAgent surpasses GPT-4o on these tasks.The results span diverse medical tasks and imaging modalities.
- 5.1 Experimental Results: Figure 3 shows MMedAgent activating appropriate tools to produce accurate, in-depth, and visualized results where LLaVA-Med provides simple or insufficient responses.The comparison covers classification, grounding, segmentation, MRG, and RAG tasks.
- 5.2 Open-ended Medical Dialogue: MMedAgent improves on both conversation and description questions and surpasses LLaVA-Med across every listed modality except MRI in open-ended medical dialogue.The evaluation covers X-ray, MRI, Histology, Gross, and CT images.
- 5.3 Tool Utilization: Tool-selection accuracy reaches 100% after 15 training epochs, demonstrating reliable activation of appropriate tools.This accuracy underpins performance across the evaluated tasks.
- 5.3 Tool Utilization: When extending the system with a simulated new tool, selection accuracy reaches 100% within 2K steps without harming selection of existing tools.Replacing an outdated tool can avoid additional retraining, while adding a new task requires additional instruction-tuning data.
A.3 Medical Grounding DINO
The paper describes the implementation and supporting materials for MMedAgent, including medical grounding data, a FastChat-based deployment, evaluation, and user-facing interfaces.
- A.3 Medical Grounding DINO: The medical grounding DINO fine-tuning data are documented through an appendix dataset reference and a dataset-overview table.The supplied materials identify Appendix Table 5 as the source for the fine-tuning datasets and Table 5 as the overview.
- A.3 Medical Grounding DINO: MMedAgent runs in FastChat with a web UI, model workers, medical tools, and a controller coordinating their activities.The complete 7B system and associated tools can run on one Nvidia A100 (80GB) GPU.
- A.3 Medical Grounding DINO: The instruction-tuning dataset-generation prompts and grounding-task pipeline are presented in the appendix figures.The cited materials refer to Appendix Figure 5 for prompt representation and Figure 5 for the grounding-task pipeline.
- A.3 Medical Grounding DINO: MMedAgent’s interface accepts images and text questions, then selects tools for tasks such as abdominal-CT organ segmentation and X-ray report generation.The interface example also displays the agent’s thought progress.
- A.3 Medical Grounding DINO: The evaluation pipeline compares an evaluated assistant, such as MMedAgent or LLaVA-Med, with GPT-4o as the second assistant.The evaluation prompts are shown in Appendix Figure 7.