PaperlayerView exact revision on arXiv

Source-linked AI summary

MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaboration

Yucheng Zhou, Lingran Song, Jianbing Shen

arXiv:2506.19835v1cs.CL

TL;DR

Unified multimodal medical LLMs face costly knowledge updates and limited modularity across diagnostic tasks. MAM addresses these challenges by coordinating specialized LLM-based medical roles, and evaluations across text, image, audio, and video datasets show improvements of 18% to 365% over baseline models. The framework also reveals that discussion benefits depend on the number of rounds and that relevant retrieval content is associated with higher diagnostic accuracy.

  • Problem

    Unified multimodal medical LLMs have costly knowledge updates and limited modularity and flexibility across diverse diagnostic tasks.

  • Method

    MAM assigns General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director roles to collaborating LLM-based agents for multimodal diagnosis.

  • Results

    MAM consistently outperforms modality-specific LLMs across multimodal medical datasets, with performance improvements ranging from 18% to 365% over baseline models.

  • Takeaways & Limitations

    MAM supports modular knowledge updates and integrates specialized expertise across diverse medical tasks and modalities.

  • Takeaways & Limitations

    MAM is constrained by underlying LLM biases, knowledge gaps, and reasoning inaccuracies, and lacks real-world clinical validation.

Abstract

from arXiv · show

Recent advancements in medical Large Language Models (LLMs) have showcased their powerful reasoning and diagnostic capabilities. Despite their success, current unified multimodal medical LLMs face limitations in knowledge update costs, comprehensiveness, and flexibility. To address these challenges, we introduce the Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis (MAM). Inspired by our empirical findings highlighting the benefits of role assignment and diagnostic discernment in LLMs, MAM decomposes the medical diagnostic process into specialized roles: a General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director, each embodied by an LLM-based agent. This modular and collaborative framework enables efficient knowledge updates and leverages existing medical LLMs and knowledge bases. Extensive experimental evaluations conducted on a wide range of publicly accessible multimodal medical datasets, incorporating text, image, audio, and video modalities, demonstrate that MAM consistently surpasses the performance of modality-specific LLMs. Notably, MAM achieves significant performance improvements ranging from 18% to 365% compared to baseline models. Our code is released at https://github.com/yczhou001/MAM.

1 Introduction

Medical diagnosis involves heterogeneous data that burdens clinicians, while unified multimodal medical LLMs remain costly to update and insufficiently modular. MAM addresses these issues by assigning specialized roles to collaborating LLM agents and evaluates the framework across multimodal datasets.

  • Motivation: Heterogeneous medical data, including text, images, cardiac sounds, and surgical video, creates substantial cognitive demands for clinicians performing diagnosis.The paper motivates LLM-assisted diagnostic workflows because growing medical data also provides training resources.
  • Limitations of Unified Models: Unified multimodal medical LLMs require costly full-model retraining for knowledge updates and lack flexibility across diverse diagnostic tasks.These limitations motivate an alternative to a single model expected to perform sufficiently across medical tasks.
  • MAM Framework: MAM decomposes diagnosis into General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director agents with distinct responsibilities.The agents cover triage, domain expertise, image analysis, information retrieval and management, and coordination of diagnostic opinions.
  • MAM Framework: The modular design supports more granular knowledge updates and maintenance without global retraining while integrating existing medical models and specialized knowledge.This design is intended to support collaboration resembling human medical teams.
  • Evaluation: Across publicly available multimodal medical datasets, MAM consistently outperforms modality-specific LLMs and is further examined through ablation, consistency, and sensitivity analyses.The evaluations cover multiple data modalities and investigate discussion rounds and role counts.

2 Related Work

Prior work develops general, medical, multimodal, and multi-agent LLM systems for complex reasoning and domain tasks. These studies motivate applying collaborative agents to medical diagnosis, while Table 1 compares direct and role-assigned prompting.

  • LLM-Based Multi-Agent Systems: LLM-based multi-agent systems have been applied to mathematical problem solving, software engineering, and finance through collaborative reasoning and decision-making.Examples include MathChat, MAGIS, and FinCon.
  • LLM-Based Multi-Agent Systems: Multi-agent systems are presented as adaptable to reasoning and decision-making, which are relevant to medical tasks requiring interdisciplinary knowledge and multiple steps.The related work connects these capabilities to medical-domain needs.
  • Medical LLMs: Medical LLMs have been developed for conversational healthcare, Chinese medical assistance, and multi-stage domain-specific training.The cited examples include DISC-MedLLM and Zhongjing.
  • Multimodal Medical LLMs: Multimodal medical models address biomedical images, chest radiographs, and ophthalmology through systems such as LLaVA-Med, XrayGPT, and OphGLM.These systems specialize in different medical visual tasks and domains.
  • Prompting Comparison: Table 1 compares Direct prompting without role assignment against Assigned Roles prompting across multimodal medical tasks.The table marks performance improvements using blue numbers.

3 Empirical Study

The empirical study tests role assignment and diagnostic discernment across publicly available medical datasets spanning text, image, audio, and video. Results indicate that assigned roles improve performance and that LLMs can select correct diagnoses beyond random expectation.

  • Experimental Setting: The study uses publicly available Brain Tumor, DeepLesion, Heartbeat, MedQA, MedVidQA, and NIH Chest X-rays datasets spanning multiple medical modalities.The listed datasets include case counts for several text, image, and audio tasks.
  • Assigned Roles: Assigned Roles prompting improves performance over Direct prompting across all evaluated datasets, with gains ranging from 5.0% on PMC-VQA to 38.5% on PubMedQA.The study reports benefits even for datasets with high baseline performance and particularly for tasks requiring deeper medical context.
  • Diagnostic Discernment: Diagnostic discernment evaluates whether LLMs can select the correct diagnosis from three plausible alternatives after filtering for instances containing at least one correct option.Expectation represents random-selection accuracy, whereas Reasoning measures explicit correct-diagnosis selection.
  • Diagnostic Discernment: Reasoning accuracy exceeds random-selection expectation, particularly on complex visual question-answering tasks such as PathVQA and PMC-VQA.The positive delta across datasets indicates that models can evaluate and refine diagnostic outputs.

4 Method

MAM organizes multimodal diagnosis through five specialized LLM-based roles that classify, refer, decompose, retrieve, discuss, and synthesize medical cases. The workflow uses specialist and radiologist opinions, iterative review, and consensus-based final diagnosis.

  • Role Assignment: MAM assigns five LLM-based roles: General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director.The roles respectively support triage and referral, domain opinions, image analysis, information retrieval and summarization, and diagnostic synthesis.
  • Initial Triage and Referral: The General Practitioner classifies each multimodal case and refers it to an appropriate Specialist Team.Referral is determined from the classified disease type.
  • Problem Decomposition and Information Retrieval: The Specialist Team decomposes and anonymizes the medical problem, while the Medical Assistant retrieves and summarizes relevant medical information.Retrieval uses the decomposed problem as its basis, and the resulting information is summarized for the diagnostic process.
  • Diagnostic Opinion Generation and Discussion: Specialists and the Radiologist independently generate diagnostic opinions, which the Director coordinates into a synthesized report.The Director's report incorporates specialist opinions, radiologist input, and the information summary.
  • Consensus and Final Diagnosis: The Specialist Team votes on the Director's report, and the process iterates until consensus is reached before deriving the final diagnosis.Consensus is checked from specialist endorsement votes; the Director then derives the final diagnosis from the report.

5 Experiments

MAM is evaluated across text, image, audio, and video medical diagnosis datasets, where it consistently outperforms modality-specific competitors. Ablation results attribute gains to role specialization, discussion, and retrieval.

  • Main Results: MAM consistently outperforms strong competitors across text, image, audio, and video medical datasets.The evaluations cover MedQA, PubMedQA, multiple image datasets, Heartbeat, SoundDr, and MedVidQA.
  • Main Results: 18% to 365%: MAM’s performance improvements over baseline models span this range across the evaluated modalities.
  • Image-based Evaluation: MAM achieves top accuracy on PathVQA, PMC-VQA, DeepLesion, NIH Chest X-rays, and Brain Tumor image datasets.The passage highlights particularly substantial gains on DeepLesion and NIH Chest X-rays.
  • Ablation Study: Ablation results show consistent performance improvements as agent roles, inter-agent discussion, and information retrieval are incrementally added.The complete +Retrieval configuration achieves the highest performance across datasets.
  • Ablation Study: The improvement from +Discussion to +Retrieval highlights the Medical Assistant’s role in enhancing diagnostic accuracy through relevant medical knowledge.

5.4 Consistency

MAM’s consistency analysis examines whether its predictions preserve correct Direct predictions while correcting errors. Discussion experiments show that early rounds can help, whereas excessive discussion may reduce gains or introduce noise.

  • Consistency: Consistency measures the percentage of cases where MAM aligns with a correct prediction from the Direct approach.The metric assesses retention and reinforcement of correct baseline predictions while correcting errors.
  • Consistency: MAM’s performance and consistency are positively correlated across datasets.Datasets where MAM performs well generally show high consistency with correct Direct predictions.
  • Consistency: MAM generally outperforms Direct despite occasional deviations from the baseline’s correct predictions.The authors interpret this as refinement through collaborative, knowledge-augmented processing rather than simple replication.
  • Discussion Time and Performance: Early discussion rounds improve Brain Tumor performance, but extending discussions beyond a few rounds does not consistently produce further gains.MedQA and PathVQA often peak within the first two or three rounds.
  • Discussion Time and Performance: Excessive discussion may introduce noise or dilute accurate initial opinions, making limited discussion a balance between collaboration and over-discussion.PMC-VQA declined in the final round, indicating dataset-specific sensitivity.

5.6 Impact of Role Number

Role granularity affects MAM’s diagnostic performance, with moderate specialization outperforming both a single-agent setup and an overly complex five-role configuration. Retrieval quality also remains a key constraint, although relevant retrieved information improves diagnostic accuracy.

  • Role Granularity: 3 roles significantly improved results over 1 role, while 5 roles decreased performance across datasets.Performance followed an inverted U-shape, suggesting that excessive roles may introduce redundancy or overhead.
  • Retrieval Recall: 12.1% to 34.0%: retrieval recall varied across datasets, from NIH to Heartbeat.The reported recall measures whether retrieved medical documents contain information necessary to answer diagnostic questions correctly.
  • Retrieval Impact: 46.2%: NIH “Answer Correct” exceeded 38.6% for “Discussion” without retrieval.“Answer Correct” measures final-answer accuracy when retrieved content contains the necessary information.
  • Case Workflow: The framework uses expert dialogue, web retrieval, Director synthesis, and expert voting before producing a final diagnosis.In the illustrated DeepLesion case, the framework produced correct predictions while the baseline was incorrect.

6 Conclusion

MAM addresses limitations of unified multimodal medical LLMs through modular collaboration among specialized LLM-based agents. Evaluations across multimodal medical datasets show superiority over modality-specific LLMs, with reported improvements of 18% to 365%.

  • Conclusion: MAM assigns General Practitioner, Specialist Team, Radiologist, Medical Assistant, and Director roles to distinct LLM-based agents.The structure is intended to enhance knowledge updates, leverage specialized expertise, and adapt to diverse medical tasks and modalities.
  • Conclusion: 18% to 365%: MAM outperformed modality-specific LLMs across extensive multimodal medical-dataset evaluations.The evaluated modalities include text, image, audio, and video.
  • Future Work: Future work will integrate advanced knowledge retrieval and evaluate MAM in real-world clinical settings.

Limitations

MAM’s performance is constrained by the capabilities of the underlying LLMs used for each agent role, and the study lacks real-world clinical validation. These limitations may be addressed through flexible base-model switching and future clinical validation studies.

  • Model Dependence: Underlying LLM biases, knowledge gaps, and reasoning inaccuracies may propagate through MAM and compromise diagnostic outcomes.
  • Model Dependence: MAM’s architecture permits switching base models, which could mitigate some underlying-model limitations in future applications.
  • Clinical Validation: The study lacks real-world clinical validation because comprehensive evaluation requires substantial resource allocation and human expertise.

A Case Study

Figure 4 compares the baseline model with MAM on the same sample input from the DeepLesion dataset.

  • A Case Study: Figure 4 compares the baseline model and MAM using the same DeepLesion sample input.

B Prompt

The prompts organize multimodal medical diagnosis by classifying inputs, assigning disease-relevant roles, gathering assessments and studies, and coordinating review toward a final answer. They cover image, audio, video, and text question types, alongside meeting summaries, search synthesis, and answer validation.

  • Input classification: The framework classifies images by modality and body region, including X-Ray, CT, MRI, pathology, biomedical, brain, bone, abdomen, mediastinum, liver, lung, kidney, soft tissue, and pelvis.
  • Input classification: Audio and video prompts assign categories such as cardiovascular or respiratory audio and sports, rehabilitation, or emergency video.
  • Input classification: Text questions are mapped to medical domains including anatomy, medicine, microbiology, radiology, surgery, and unknown.
  • Role assignment: A General Practitioner prompt identifies relevant Specialist Doctors, Radiologic Technologists, and other specialists for the modality, disease type, and patient question.
  • Diagnostic collaboration: Role prompts request assessment of health, modality-specific studies, patient history, symptoms, diagnostic reasoning, possible answers, and a concluding recommendation.
  • Diagnostic collaboration: Moderator and assistant prompts summarize discussions, record agreements and disagreements, review conclusions, synthesize search results, describe modalities, and check answer reasonableness or reasoning errors.
Loading 2506.19835v1…