PaperlayerView exact revision on arXiv

Source-linked AI summary

MMedAgent-Plus: Scaling Multimodal Medical Agents across Modalities, Tasks and Workflows

Ruoxin Wang, Yuanting Pan, Binxu Li, Ruiyang Ji, Yaqian Chen, Jiayuan Ding, Qifan Wang, Lifu Huang, Mingjie Li, Yu Zhang, Ehsan Adeli, Kilian Pohl, Zhe Xu, Zihao Lin, Haoyu Dong, Yixin Wang

arXiv:2610.27233v1cs.CLcs.AI

TL;DR

Medical workflows need interdependent analyses, while specialist and generalist models are not designed to coordinate them. MMedAgent-Plus uses a planner to invoke specialist tools and synthesize their outputs; most evaluated tasks have recall of at least 0.94, with Registration and Segmentation as exceptions.

  • Problem

    Specialist models typically operate in isolation, while complex medical workflows require diverse analyses whose intermediate findings guide later decisions.

    Source: §1 · p. 2

  • Method

    MMedAgent-Plus uses a planner that incorporates dialogue history and tool outputs to generate structured tool calls, execute specialists, and select subsequent steps or produce a final response.

    Source: §4.1 · p. 15

  • Results

    Most evaluated tasks achieve recall of at least 0.94, except Registration at 0.36 and Segmentation at 0.88.

    Source: §2.1 · p. 8

  • Takeaways & Limitations

    Sequential tool orchestration allows intermediate outputs to inform subsequent operations in workflows involving multiple interdependent analyses.

    Source: §1 · p. 2, §3.1 · p. 11

    Evaluated multi-tool workflows use two-to-three-tool chains from public benchmarks and do not cover substantially longer sequences or prospectively collected clinical workflows.

    Source: §3.4 · p. 14

Abstract

from arXiv · show

Medical workflows often require multiple interdependent analyses, but existing specialist and generalist AI models are not designed to coordinate them. We introduce MMedAgent-Plus, a multimodal medical tool-use agent that combines a vision-language planner with 18 leading specialist models spanning 12 tasks and 8 modalities. Through a unified tool interface, the planner selects specialists, executes them in sequence, and synthesizes responses grounded in their outputs. The planner is instruction-tuned on 208,928 samples from 32 public medical benchmarks, covering single- and multi-tool interactions across single and multiple rounds. We separately evaluate tool orchestration, response synthesis, and end-to-end execution on held-out samples. MMedAgent-Plus achieves tool-call accuracies of 91.0%, 98.1%, and 80.7% in single-tool multi-round, multi-tool multi-round, and multi-tool single-round settings, respectively. Given correct tool outputs, MMedAgent-Plus achieves scores of 14.4-14.8 out of 15 for faithfulness, completeness, and conciseness across 600 evaluation samples. In end-to-end execution without pre-supplied tool outputs, MMedAgent-Plus correctly invokes the required specialist in 90.6% of 350 evaluation cases spanning seven tasks. Its final-answer accuracy reaches 0.98, matching the dedicated leading specialist and exceeding the strongest baseline (0.28).

1 Main

MMedAgent-Plus addresses the gap between specialist models that work in isolation and clinical workflows that require multiple interdependent analyses. It combines a vision-language planner with specialist tools to select, sequence, and synthesize analyses across tasks and modalities.

1 Main

  • Specialist models excel at individual tasks but typically lack the ability to determine needed analyses, coordinate execution, and integrate results into a coherent workflow.Clinical examples include characterizing lesions before reporting and selecting tests based on symptoms and medical history.Source: §1 · p. 2
  • Unified medical models broaden task and modality coverage, but that coverage alone does not provide sequential tool selection or adaptation to intermediate results.Existing tool-augmented agents have often been developed or evaluated within particular modalities or clinical domains.Source: §1 · p. 2
  • MMedAgent-Plus combines a vision-language planner with 18 specialist models covering 12 medical tasks and 8 modalities.The specialists were selected as the leading publicly available models for their tasks at construction time, based on reported benchmark performance.Source: §1 · p. 3
  • Its instruction-tuning corpus contains 208,928 samples from 32 public medical benchmarks, spanning single- and multi-tool dialogues across single and multiple rounds.The dialogue settings represent iterative individual analyses, progressive workflows, successive requests, and compositional workflows.Source: §1 · p. 3
  • 91.0%, 98.1%, and 80.7% tool-call accuracy is achieved in single-tool multi-round, multi-tool multi-round, and multi-tool single-round interactions, respectively.The paper also reports improved response faithfulness and completeness over four recent vision-language models.Source: §1 · p. 3

2 Results

The Results assess tool routing, response synthesis, and end-to-end execution separately. MMedAgent-Plus performs strongly across dialogue settings, while routing errors and task-specific output gaps remain.

2 Results

  • The evaluation measures tool-orchestration accuracy on 1,500 held-out dialogues, synthesis quality on 600 cases with correct specialist outputs, and end-to-end performance on 350 single-tool cases.The 95% Wilson confidence-interval margins of error for tool-orchestration accuracy are within 4 percentage points across all three settings.Source: §2 · pp. 3, 5

2.1 MMEDAGENT-PLUS demonstrates effective tool orchestration across diverse dialogue settings

  • Tool-call accuracy is 91.0% in single-tool multi-round, 98.1% in multi-tool multi-round, and 80.7% in multi-tool single-round dialogues.Multi-tool multi-round whole-case accuracy is 95.8%; multi-tool single-round whole-case accuracy is 77.1%.Source: §2.1 · pp. 5–6, Figure 2 · p. 6
  • Routing errors cluster in a few tools: recall is 0.36 for UniGradICON, 0.64 for CONCH, and 0.76 for MedSAM; Registration recall is 0.36 and Segmentation recall is 0.88.UniGradICON and MedSAM errors often omit specialist invocation, while 36% of CONCH cases are routed to LLaVA-Med.Source: Figure 3 · p. 7, §2.1 · pp. 7–8

2.2 MMEDAGENT-PLUS achieves high-quality tool-grounded responses

  • Response quality is highest across all three dialogue settings; in multi-tool single-round, MMedAgent-Plus scores 14.77/15 versus 10.89 for Qwen3-VL-8B-Instruct.Scores are evaluated for faithfulness, completeness, and conciseness.Source: Table 1 · p. 8, §2.2 · p. 8
  • Tool-grounded responses outperform direct-answer baselines across evaluated modalities and most tasks, but textual scores do not fully measure segmentation task completion.For fundus images, all baselines score below 50%, while MMedAgent-Plus approaches 100%.Source: §2.2 · pp. 8–9

2.3 MMEDAGENT-PLUS preserves specialist-level performance

  • Across 350 end-to-end cases, required-specialist invocation accuracy is 90.6%; QA accuracy is 0.98, matching the specialist and exceeding the strongest baseline at 0.28.The end-to-end evaluation requires specialist selection and execution before the final response.Source: §2.3.1 · p. 9, Figure 4 · p. 10, §2.3.3 · p. 11

2.3.2 Response Quality under LLM-as-a-judge

  • MMedAgent-Plus has the highest mean response-quality score across five end-to-end tasks, with the largest margins in report generation, NER, and multimodal classification.In QA/rationale generation, its conciseness score of 4.14 is below MedGemma-1.5-4B-IT at 5.00 and GPT-5.4-mini at 4.92.Source: Figure 4 · p. 10, §2.3.2 · p. 10

2.3.3 Task-Specific Tool Output Quality

  • End-to-end outputs preserve specialist-level QA accuracy at 0.98 and specialist RadGraph F1 at 0.64, but pathology micro-F1 falls from 0.72 to 0.38 across all cases.For segmentation, Dice includes all 50 cases with missing masks scored zero, whereas HD95 includes only cases with generated masks.Source: Figure 4 · p. 10, §2.3.3 · p. 11

3 Discussion

MMedAgent-Plus coordinates specialist tools through a vision-language planner, extending medical AI beyond isolated predictions. Results show strong orchestration and end-to-end performance, while the evaluated workflows remain less complex than clinical practice.

3.1 Tool orchestration extends medical AI beyond isolated specialist models

  • M coordinates 18 specialists across 12 tasks and eight modalities, extending MMedAgent’s five modalities and six tasks.Across eight modality axes, M achieves 96.1% versus 78.2% for Qwen3-VL-8B; in multi-tool single-round response synthesis, its total final response score is 14.77 versus 10.89 for the strongest baseline.Source: §3.1 · pp. 11–12

3.2 Orchestration errors reflect tool ambiguity and implicit workflow requirements

  • Tool-call accuracy peaks at 98.1% in multi-tool multi-round dialogues, where preceding requests and intermediate outputs may clarify workflow state; overlapping tools and implicit steps remain error sources.Multi-tool single-round dialogues reach 80.7% tool-call accuracy and 77.1% whole-case accuracy, with missed invocations more prevalent than unnecessary selections.Source: §3.2 · pp. 12–13

3.3 Residual errors arise from specialist selection and response synthesis

  • M invokes the required specialist in 90.6% of end-to-end cases, but both routing failures and post-invocation synthesis losses contribute to pathology classification performance loss.Among 35 correctly routed pathology cases, micro-F1 is 0.69 for the specialist and 0.51 for M.Source: §3.3 · pp. 13–14

3.4 Extending tool orchestration to adaptive workflows and clinical practice

  • Current workflows are benchmark-built chains of two to three tools; longer output-dependent sequences and cases with multiple valid tool paths remain unevaluated.Extending the library to new capabilities requires additional instruction tuning, and longer adaptive workflows and prospective clinical validation remain necessary before clinical deployment.Source: §3.4 · pp. 14–15

4.1 System Overview

MMedAgent-Plus coordinates a vision-language planner with specialist tools to carry out medical tasks and workflows. The planner uses dialogue history and returned tool outputs to guide later tool selections or final responses.

4.1 System Overview

  • The planner decides whether tools are needed, selects specialists, and coordinates their execution for a requested task or workflow.It incorporates intermediate specialist outputs into subsequent steps across dialogue turns.Source: §4.1 · p. 15

4.2 Backbone Model and Interaction Framework

The planner is adapted for medical tool orchestration by fine-tuning its vision-language projector and language decoder while keeping the vision encoder frozen. It produces structured tool calls and natural-language responses using the dialogue context and prior tool outputs.

4.2 Backbone Model and Interaction Framework

  • The planner generates JSON tool calls naming a specialist and its input parameters, such as MedSAM box coordinates.During multi-round interactions, it uses dialogue history and outputs such as masks, bounding boxes, and reports to inform later tool selection and response generation.Source: §4.2 · p. 15
  • Fine-tuning updates the vision-language projector and language decoder while leaving the vision encoder frozen.Training minimizes causal language modeling loss over sequences of structured tool calls and natural-language responses.Source: §4.2 · p. 15
  • The loss is defined over output tokens conditioned on multimodal context, enabling the planner to learn tool selection, sequencing, and response generation.The context includes user input, dialogue history, images, and available tool outputs.Source: §4.2 · p. 15

4.3 Dataset Curation

The authors curate an instruction-tuning corpus for specialist selection and workflow orchestration, drawing on public medical benchmarks and constructing dialogue examples in three interaction settings.

4.3.1 Data Sources

  • The corpus contains 208,928 samples from 32 public medical datasets and benchmarks, covering text, imaging, and report-generation tasks.Covered modalities include X-ray, CT, MRI, ultrasound, retinal imaging, and pathology.Source: §4.3.1 · p. 16, §4.3.2 · p. 16
  • Sources span tasks from question answering and classification to detection, segmentation, image transformation, and report generation.They include benchmark resources associated with PMC-LLaMA, LLaVA-Med, MedSAM, UniGradICON, and LLaVA-Rad, among others.Source: §4.3.1 · p. 16

4.3.2 Instruction Dataset Construction

  • The corpus includes 97,055 single-tool multi-round, 90,000 multi-tool multi-round, and 21,873 multi-tool single-round samples.The settings train invocation across turns, sequential tool selection using prior context, and selection of a complete tool sequence from one request.Source: §4.3.2 · pp. 16–17, §4.3 · p. 16
  • Multi-tool multi-round examples model dependent workflows using prior specialist findings and independent requests requiring fresh tool selection; generated chains are checked for structural, modality, and interface compatibility.Single-round multi-tool examples require selecting the full two- or three-tool sequence before specialist outputs are available.Source: §4.3.2 · pp. 16–17

4.3.3 Dialogue Generation and Data Quality Control

  • New dialogues use fixed request and response templates created with GPT-4, while medical inputs and reference outputs come from source benchmarks.The corpus also retains original instruction data for six inherited tools; generated multi-tool dialogues undergo compatibility checks, and construction uses a fixed random seed.Source: §4.3.3 · p. 17

4.4 Tool Integration

The tool library combines specialist models behind a standardized interface that accommodates heterogeneous inputs and outputs. Its tools cover text, visual understanding, image processing, and medical report generation across multiple tasks and modalities.

4.4 Tool Integration

  • The library integrates 18 specialist models spanning 12 medical tasks and 8 input modalities, with a shared interface for tool invocation and compatible multi-step workflows.The tools are grouped into text understanding and generation, visual understanding, image processing, and report generation.Source: §4.4 · pp. 17–18
  • Table 3 groups single-tool multi-round samples by specialist, while reporting multi-tool settings by their total sample counts.The caption directs readers to Supplementary Tables S6–S7 for tool-chain configurations.Source: Table 3 · p. 18

4.4.3 Image Processing

  • Image-processing tools cover detection, segmentation, registration, reconstruction, and super-resolution across medical imaging tasks.The library includes Grounding DINO, MedSAM, uniGradICON, and HealthGPT for these functions.Source: §4.4 · p. 18, §4.4.3 · p. 20

4.4.1 Text Understanding and Generation

  • Text tools provide question answering and rationale generation, radiology named-entity recognition, and retrieval-augmented generation from radiology knowledge and structured databases.The cited tools are PMC-LLaMA, RaTE-NER, and ChatCAD-R.Source: §4.4.1 · p. 19

4.4.2 Visual Understanding

  • Visual-understanding tools use LLaVA-Med for medical image question answering and BiomedCLIP for cross-modal image classification, with CONCH providing specialized pathology classification.CONCH matches image features against text descriptions in a shared image-text embedding space.Source: §4.4.2 · p. 19

4.4.3 Image Processing

  • For grounding-segmentation, Grounding DINO localizes a text-prompted region and MedSAM refines it into a pixel-level mask.Other specialized tools address ultrasound and fundus segmentation, cell segmentation in whole-slide images, and tumor detection and subtyping.Source: §4.4.3 · pp. 19–20

4.4.4 Medical Report Generation

  • LLaVA-Rad generates radiology reports, while SpecialistVLMs handles retinal OCT reports using domain-specific images and expert annotations.The report-generation tools cover radiology and retinal OCT reporting.Source: §4.4 · p. 18, §4.4.4 · p. 21

4.5 Implementation Details

The planner is fine-tuned from Qwen2.5-VL-7B and deployed alongside specialist tools as separate services behind a gateway. Evaluation covers tool orchestration, response synthesis, and end-to-end task performance using complementary metrics and baselines.

4.5.1 Model Training / 4.5.2 Deployment

  • The planner is initialized from Qwen2.5-VL-7B, trained with a frozen vision encoder, and deployed with each specialist as a separate service coordinated by a central gateway.Training updates the vision-language projector and language decoder; the gateway dispatches structured tool calls and returns outputs for subsequent selection or response generation.Source: §4.5.2 · p. 21, §4.5.1 · p. 21

4.5.3 Evaluation Protocol

  • The evaluation protocol measures individual tool selections, exact dialogue-level tool sequences, response quality, and task-specific end-to-end performance.Whole-case accuracy requires the complete tool sequence, including identities and order, to match; response quality is judged on faithfulness, completeness, and conciseness.Source: §4.5.3 · pp. 21, 22, 23, 25, 26
  • Comparisons with direct-answer multimodal models assess response quality, but observed gaps cannot be attributed solely to specialist-tool access.Baselines receive the original request and images without specialist outputs; end-to-end comparisons exclude segmentation tools from direct-answer response-quality comparisons because the baselines do not generate masks.Source: §4.5.3 · p. 25

5 Data availability

The authors provide the instruction annotations, training and test splits, and dataset reconstruction instructions; underlying medical images remain with their original repositories.

5 Data availability

  • Instruction annotations and training/test splits are available from the MMedAgent organization, while underlying images must be obtained from their original repositories under their licenses.The authors do not own or redistribute the images and provide instructions for downloading source data and reconstructing the dataset.Source: §5 · p. 26

6 Code availability

Training and evaluation code and fine-tuned model weights are available through the project’s repositories and Hugging Face organization.

6 Code availability

  • Training and evaluation code and fine-tuned model weights are released through the project repositories and the MMedAgent Hugging Face organization.Deployment code, some instruction-generation scripts, task-specific metric implementations, and evaluation sample indices are slated to be added upon publication.Source: §6 · p. 26

7 Supplementary

The supplementary material documents the specialist library, compatibility and execution interfaces, instruction corpus construction, dialogue schema, and evaluation split protocol. It also describes training-data provenance and the limits of linguistic novelty in one held-out subset.

S1 System and Tools / S1.1 Tool Cards / S1.2 Tool-Modality Compatibility

  • The 18 specialist tools run as independent containerized services, with G-Seg sequentially invoking Grounding DINO and MedSAM and timed-out calls returning structured errors.Tool availability is modality-specific, and unsupported tool-modality combinations are excluded during dataset construction.Source: S1.1 Tool Cards · p. 33, S1.2 Tool-Modality Compatibility · p. 33

S1.3 Interface Compatibility Rules

  • The shared interface resolves tool names and parameter aliases, binds images to calls, returns structured outputs or errors, and validates compatibility between successive tools.Execution alternates planner generation and tool execution up to a maximum round limit; workflows exceeding it are truncated and counted as orchestration failures.Source: S1.3 Interface Compatibility Rules · pp. 33–34

S2 Dataset / S2.1 Data Sources / S2.2 Request and Response Templates / S2.3 Data Provenance

S2.4 Tool Chains

  • Multi-tool chains include dependent two-call workflows and independent requests across turns, while single-round chains specify two or three tools before any specialist output is available.The independent multi-round pattern spans 342, 5,195, and 11,460 distinct chains for dialogues with two, three, and four invocations, respectively.Source: S2.4 Tool Chains · pp. 35–36

S2.5 Instruction schema

  • Instruction records serialize user requests, planner calls, and specialist outputs in an ordered dialogue, with the three settings representing single-tool, multi-tool multi-round, and multi-tool single-round interactions.In the single-round multi-tool setting, the planner emits the complete sequence before execution and tool returns are combined into one message.Source: S2.5 Instruction schema · pp. 36–37
  • The supplementary figures show representative held-out records for the seven end-to-end specialist tools and a UniGradICON example with multiple image paths.Request, planner, and specialist-output fields are highlighted in distinct colors; text-only tools use a non-clinical placeholder image path without a thumbnail.Source: S2.5 Instruction schema · p. 37, Figure S1 · p. 37, Figure S2 · p. 37

S2.6 Dataset Split Protocol

  • Dialogue-level splits reserve approximately 10% of each setting for evaluation, and samples exceeding the 8,192-token context limit are excluded.The held-out pools contain 9,128 single-tool multi-round, 10,000 multi-tool multi-round, and 2,430 multi-tool single-round dialogues.Source: S2.6 Dataset Split Protocol · pp. 37, 40
  • In the examined multi-tool multi-round subset, training and held-out partitions share no source images or image-request pairs, but 67.0% of held-out requests also appear verbatim in training.Thus, this subset tests orchestration on unseen source images rather than entirely unseen request formulations.Source: S2.6 Dataset Split Protocol · p. 40

S3 Training Details

The supplement documents the planner’s training setup and the protocols used to assess response quality, task metrics, and direct-answer baselines. Evaluation uses held-out data, and the baseline comparison has limits on attributing performance differences specifically to specialist-tool access.

S3 Training Details

  • The planner is initialized from Qwen2.5-VL-7B-Instruct, with its vision encoder frozen and projector and language decoder fully fine-tuned; training uses masked assistant-token loss and one epoch of AdamW optimization.Training uses a cosine learning-rate schedule, an effective global batch size of 256, and approximately 816 optimization steps; no validation set is used for checkpoint selection, and the final checkpoint is evaluated.Source: S3 Training Details · pp. 40–41

S4.1 LLM-as-a-judge

  • Final responses are scored separately for faithfulness, completeness, and conciseness using criterion-specific GPT-5.4-mini judge calls with integer scores from 1 to 5.For faithfulness, the judge assesses consistency with specialist output; completeness and conciseness use the reference answer to identify relevant information and expected scope.Source: S4.1 LLM-as-a-judge · pp. 41–42, Figure S9 · p. 42

S4.2 Task-specific metrics

  • Task-specific metrics include ROUGE-L and BLEU-4 for text, rule-based CheXpert F1 and RadGraph F1 for reports, and exact string-and-type matching for RaTE-NER.Classification uses micro- and macro-averaged F1; segmentation reports Dice and HD95, with missing masks scored differently across those metrics.Source: S4.2 Task-specific metrics · p. 43

S4.3 Baseline models and prompts

  • Direct-answer baselines receive user requests and images but no specialist outputs, dialogue history, or reference answers; model and input differences limit attribution of performance gaps solely to tool access.For multi-image requests, LLaVA-Med receives only the first image, whereas Qwen3-VL-8B-Instruct and MedGemma receive all available images.Source: S4.3 Baseline models and prompts · pp. 43–44

S5 Qualitative Examples

Representative held-out examples show MMedAgent-Plus using specialist outputs to produce task-specific answers and reports, while direct-answer baselines sometimes omit required labels or contradict the examples’ stated findings. The examples span question answering, radiology, entity recognition, image classification, and pathology classification.

Biomedical question answering

  • In three biomedical questions, MMedAgent-Plus gives the stated glaucoma finding, a concise erythrocyte-treatment answer, and the explicit negative gestational-diabetes association required by the references.The examples contrast these outputs with baselines that contradict evidence, give verbose responses despite the correct yes/no conclusion, or omit the negative association.Source: Figure S10 · p. 45, Figure S11 · p. 53, Figure S12 · p. 54

Radiology report generation

  • Across chest-radiograph examples, MMedAgent-Plus produces accurate or faithful LLaVA-Rad reports despite a mislabeled modality, low lung volumes, or a pacemaker with chronic interstitial markings.The shown baselines misidentify the modality or report findings such as a mass, normal study, hiatal hernia, pneumothorax, or pleural effusion.Source: Figure S13 · p. 55, Figure S14 · p. 56, Figure S15 · p. 57

Named entity recognition

  • Three named-entity examples show MMedAgent-Plus reproducing RaTE-NER’s typed spans, including a single Disease span and a Non-Disease label for negated intracranial hemorrhage.Direct-answer baselines instead return unstructured lists, split or paraphrase spans, or skip extraction.Source: Figure S16 · p. 58, Figure S17 · p. 59, Figure S18 · p. 60

Image classification

  • Image-classification examples show MMedAgent-Plus identifying immunohistochemistry staining, lung CT, and a COVID line chart through BiomedCLIP where direct-answer baselines miss the required labels or misclassify the image.Source: Figure S19 · p. 61, Figure S20 · p. 62, Figure S21 · p. 63

Pathology classification

Related papers

Loading 2610.27233v1…