Source-linked AI summary
MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making
Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, Hae Won Park
TL;DR
Complex medical decision-making requires careful collaboration over diverse information, but the best way to leverage LLMs remains open. MDAgents dynamically assigns solo or group LLM collaboration according to task complexity and outperformed prior solo and group methods in seven of ten benchmarks. Moderator review combined with external medical knowledge further improved accuracy, while the framework remains limited by multiple-choice evaluation and non-interactive diagnosis.
Problem
Medical decision-making involves diverse information and complex clinical reasoning, while the best way to leverage LLMs for such tasks remains open.
Method
MDAgents dynamically assigns LLMs to solo roles or collaborative teams according to the medical query’s complexity.
Results
MDAgents achieved the best performance in 7 out of 10 medical benchmarks, while combining moderator review with external medical knowledge raised accuracy from 71.8% to 80.3%.
Takeaways & Limitations
Adaptive collaboration provides a framework for matching medical decision-making structure to task complexity across text-only and multimodal benchmarks.
Takeaways & Limitations
The framework is currently confined to multiple-choice question answering and does not model interactive, patient-centered diagnosis involving patients and caregivers.
Abstract
from arXiv · showhide
Foundation models are becoming valuable tools in medicine. Yet despite their promise, the best way to leverage Large Language Models (LLMs) in complex medical tasks remains an open question. We introduce a novel multi-agent framework, named Medical Decision-making Agents (MDAgents) that helps address this gap by automatically assigning a collaboration structure to a team of LLMs. The assigned solo or group collaboration structure is tailored to the medical task at hand, emulating real-world medical decision-making processes adapted to tasks of varying complexities. We evaluate our framework and baseline methods using state-of-the-art LLMs across a suite of real-world medical knowledge and medical diagnosis benchmarks, including a comparison of LLMs' medical complexity classification against human physicians. MDAgents achieved the best performance in seven out of ten benchmarks on tasks requiring an understanding of medical knowledge and multi-modal reasoning, showing a significant improvement of up to 4.2% (p < 0.05) compared to previous methods' best performances. Ablation studies reveal that MDAgents effectively determines medical complexity to optimize for efficiency and accuracy across diverse medical tasks. Notably, the combination of moderator review and external medical knowledge in group collaboration resulted in an average accuracy improvement of 11.8%. Our code can be found at https://github.com/mitmedialab/MDAgents.
1 Introduction
Medical decision-making integrates diverse, multimodal information and collaborative clinical judgment, making it a promising but demanding application for LLMs. The introduction motivates an adaptive framework for varying medical-query complexity.
- Medical decision-making combines imaging, EHRs, physiological signals, genetic information, and current medical research.
- Clinicians may refer or triage patients to specialists when cases exceed primary-care or initial-care complexity.
- LLMs can process medical literature and clinical information while supporting probabilistic and causal reasoning.
- The introduction presents solo and group clinical roles, including primary care clinicians, multidisciplinary teams, and integrated care teams.
- MDAgents organizes medical queries through complexity checking, recruitment, analysis and synthesis, and final decision-making.
2 Related Works
Prior work applies LLMs to medical question answering, research, prediction, diagnosis, and generative clinical tasks. MDAgents is positioned as a framework spanning key dimensions of LLM-based medical decision-making.
- Medical LLMs have been evaluated on exams, biomedical research, clinical risk prediction, diagnosis, reporting, image description, differential construction, and diagnostic dialogue.
- Detailed examples of low-, moderate-, and high-complexity cases are provided in Appendix E.1.
- MDAgents is described as the only compared method performing all key dimensions of LLM decision-making.
- Inference-time methods such as prompt engineering and retrieval-augmented generation complement medical pre-training and fine-tuning.
3 MDAgents: Medical Decision-making Agents
MDAgents adapts collaboration to query complexity by selecting solo clinicians, multidisciplinary teams, or integrated care teams. Its pipeline then analyzes, synthesizes, and converts those inputs into a final answer.
- Medical Complexity Check: The framework first classifies each query as low, moderate, or high complexity.
- Expert Recruitment: Recruitment assigns a single Primary Care Clinician to low-complexity cases and MDT or ICT teams to moderate- or high-complexity cases.
- Analysis and Synthesis: Solo analysis uses Chain-of-Thought and Self-Consistency, while MDTs form consensus and ICTs synthesize information for the most complex cases.
- Decision-making: The decision-maker synthesizes inputs according to complexity, using direct responses for low-complexity queries and interaction histories for moderate cases.
- Evaluation: Table 2 reports accuracy across medical benchmarks for Solo, Group, and Adaptive settings, with GPT-4(V) used for most benchmarks and Gemini-Pro(Vision) for MedVidQA.
4 Experiments and Results
Across ten medical benchmarks, MDAgents adapt collaboration to query complexity and outperform fixed Solo and Group settings while improving efficiency. Ablations show benefits from complexity selection, moderator review, external knowledge, calibrated agent counts, and collaborative convergence.
- Benchmark Results: MDAgents achieved the best performance in 7 of 10 medical benchmarks, significantly outperforming both Solo and Group methods.The advantage extended across text-only and text-image datasets.
- Solo vs. Group Setting: Solo methods reached 83.9% on MedQA, whereas Group methods reached 91.9% on SymCat and performed better on several multimodal datasets.Solo methods were stronger on simpler MedQA, while Group methods led on more complex SymCat and multimodal tasks.
- Impact of Complexity Selection: Adaptive complexity selection reached 81.2% accuracy on text-only queries, exceeding low, moderate, and high static assignments.The corresponding static accuracies were 64.2%, 71.6%, and 65.8%.
- Impact of Complexity Selection: Complexity assignments varied by modality: 64% of text-only queries were high complexity, compared with 55% low complexity for image-plus-text and 87% low complexity for video-plus-text.The authors note that MedVidQA contains relatively less complex medical knowledge and requires evaluation on more challenging video datasets.
- Moderator Review and RAG: Combining moderator review with MedRAG increased average accuracy from 71.8% to 80.3%, an 11.8% improvement.MedRAG alone reached 75.2% and moderator review alone reached 77.6%.
- Efficiency and Robustness: Adaptive collaboration peaked with 3 agents at 83.5% accuracy while using 9.3 API calls, fewer than the Solo and Group comparisons.The Solo and Group settings used 6.0 and 20.3 API calls, respectively.
- Efficiency and Robustness: Adaptive performance improved at higher temperature, while consensus entropy declined across text-only, text-image, and text-video modalities.Text-plus-video converged rapidly; text-only and text-plus-image showed more gradual entropy declines.
5 Conclusion
MDAgents dynamically structures LLM collaboration for medical decision-making by assigning solo or group roles according to task complexity. Across ten benchmarks, it outperformed previous solo and group methods on seven, while case studies and ablations examined collaboration and component contributions.
- MDAgents dynamically assigns LLMs to independent roles or collaborative groups according to medical task complexity.The framework is designed to emulate consultation processes and structure collaboration for complex medical decision-making.
- MDAgents outperformed previous solo and group methods on 7 out of 10 medical benchmarks.
- Case-study agents converged on the correct diagnosis despite initially divergent perspectives.The case study examines how differing expert opinions are synthesized into a more accurate diagnosis.
- Ablation studies identified the individual agent contributions and strategy interactions associated with the framework’s performance.
Limitations and Future Works
The paper identifies limitations in medical specialization, patient-centered interaction, and diagnostic safety. Future work proposes medical-focused models, interactive multi-stakeholder diagnosis, regret-aware decision-making, and corrective safeguards.
- The authors identify these limitations as directions for future research.
- The framework does not use foundation models specifically trained on medical data, such as Med-Gemini, AMIE, or Med-PaLM 2.The paper proposes these models to improve medical terminology and communication among agents.
- The framework currently operates within multiple-choice question answering and does not model interactive, patient-centered real-world diagnosis.Future iterations aim to incorporate patients, caregivers, and physicians in a multi-stakeholder system.
- Potential risks include medical hallucinations and inaccurate or misleading information.Proposed mitigations include self-correction, rule-based rewards, confidence scores, and uncertainty estimates.
A Dataset Information
The evaluation spans medical question-answering datasets that vary in complexity, modality, and response format, alongside prompting, recruitment, decision, and consensus-analysis procedures. Complexity ranges from low text-only tasks to moderate image- or video-based tasks, with entropy used to track agent agreement.
- Dataset Information: The benchmark suite varies in question complexity, with greater complexity associated with multiple modalities or lengthy diagnostic tasks.
- Dataset Information: MedQA and PubMedQA are categorized as low-complexity text-based tasks, while DDXPlus is a low-complexity text-only diagnostic multiple-choice dataset.MedQA uses five-option USMLE questions; PubMedQA uses yes/no/maybe answers grounded in PubMed abstracts.
- Dataset Information: PMC-VQA and PathVQA are categorized as moderate-complexity visual question-answering datasets requiring medical expertise with image inputs.PMC-VQA uses multiple-choice questions, while the evaluation uses yes/no questions from PathVQA.
- Dataset Information: MedVidQA contains health-related questions with visual answers from validated videos and GPT-4-generated multiple-choice options.
- Consensus Dynamics: Entropy measures consensus progression from the distribution of unique answers, with lower entropy indicating higher consensus.Entropy trends across modalities provide insight into how quickly and effectively MDAgents reach a unified decision.
- Method and Evaluation: The framework prompts agents to assess query difficulty, recruit experts, communicate, and produce a final decision.The final decision maker reviews expert opinions and applies a specified decision method such as majority voting or ensemble refinement.
D.1 Accuracy on entire MedQA 5-options Dataset
Experiments on the entire MedQA five-option dataset with GPT-4o mini show that MDAgents outperforms single-agent and other multi-agent methods. The approach achieves 83.6% accuracy across the full dataset.
- The experiment used all MedQA five-option questions rather than the subsampled main-experiment set.
- 83.6% accuracy was achieved by MDAgents on the entire MedQA five-option dataset with GPT-4o mini.
- MDAgents outperformed both single-agent and other multi-agent methods across the full MedQA dataset.
D.2 Estimated Costs for Full Test Set Experiments
The framework’s added collaboration and knowledge mechanisms improve performance but increase computational cost. Structured moderator review with retrieved medical knowledge performs best, while discussion is especially valuable for high-complexity image-plus-text cases.
- Estimated costs: MDAgents incurs higher costs because of its multi-agent structure, with the reported performance improvements presented as justifying the expense for critical medical decision-making tasks.
- Knowledge enhancement: Moderator’s Review combined with MedRAG yielded the best performance in knowledge-enhancement experiments.Role assignment alone provided a foundation, while specific knowledge retrieval and structured review significantly enhanced domain-expertise simulation.
- Collaborative settings: 59.0% accuracy was achieved by parallel collaboration with discussion on high-complexity image+text tasks.Discussion exceeded no-discussion settings for both sequential and parallel collaboration: 45.0% vs. 39.0% and 59.0% vs. 56.0%, respectively.
- Adaptive framework: Benchmark complexity and recruited experts are aligned in examples such as MIMIC-CXR, which predominantly recruits radiologists, pulmonologists, and biomedical-informatics experts.
- Adaptive framework: The framework assigns collaboration structures according to query complexity, using a PCC for low complexity and MDT or ICT structures for moderate or high complexity.Its stages include complexity checking, expert recruitment, analysis and synthesis, and decision-making.
E.1.2 Medical Cases from MedQA Dataset
The MedQA cases illustrate how medical complexity determines whether a case can be handled by a primary-care clinician, a single specialist, or a multidisciplinary team. Examples span low, moderate, and high complexity decisions.
- MedQA examples are used to illustrate why complexity checking matters for selecting individual or teamed decision-making.
- Low complexity: A low-complexity gastric-cancer case is described as answerable by a primary-care clinician without consulting a gastroenterologist.
- Moderate complexity: A moderate-complexity pediatric case is described as manageable by a pediatric endocrinologist alone, potentially following referral from a primary-care clinician.
- High complexity: A high-complexity case involving neurological symptoms and vision problems requires neurologist consultation with ophthalmology.The associated question concerns a 63-year-old woman with persistent diplopia, ptosis, and a poorly reactive enlarged pupil.
E.2 Cases Studies with MDAgents
The case studies demonstrate how MDAgents organizes agents and reasoning steps for moderate- and high-complexity medical scenarios. The framework supports information gathering, discussion, moderation, and final decisions across image-plus-text and text-only cases.
- The case studies cover a moderate-complexity PMC-VQA image+text example and a high-complexity DDXPlus text-only example.
- Pathology example: The pathology example highlights that distinguishing hemorrhage from fibrosis and other conditions can be subtle and subject to interpretation.
- Pathology example: The displayed example answer identifies hemorrhage as the answer.
- Moderate-complexity case: The framework’s moderate-complexity workflow includes complexity checking, expert recruitment, initial assessment, collaborative discussion, and review with final decision-making.
- The examples portray an environment where agents collaborate, gather information, moderate responses, and make final decisions in complex medical scenarios.
- High-complexity case: The high-complexity workflow includes complexity checking, expert recruitment, information gathering and report generation, and final decision.
F Medical Complexity Comparison with Human Physicians
The study compares LLM medical-complexity ratings with physician judgments using 50 MedQA questions and three physician raters. Physician agreement was moderate, whereas LLM–physician correlations were low and varied across models, motivating adaptive collaboration.
- Inter-rater Reliability: 0.269 and 0.280 were the physician-rating agreement values for ICC2k and ICC3k, respectively.The reported intervals were [−0.14, 0.55] for ICC2k and [−0.15, 0.57] for ICC3k.
- Interpretation: Moderate physician agreement highlights subjectivity in medical-complexity assessment.MDAgents therefore uses dynamic collaboration structures rather than relying on one fixed complexity assessment.
- Annotation Protocol: The annotation study was conducted with a specialized interface presenting questions consistently for physician and LLM ratings.Physician majority opinions were defined by the mode, or by the mean in cases of complete disagreement.
- LLM Comparison: LLM–physician complexity-rating correlations ranged from −0.090 to 0.110.The results indicate that current LLMs may not fully capture nuances used by physicians when assessing complexity.
2. Limitations
The paper discusses limitations and future improvement areas, while also reporting methodological and reproducibility details for its experiments. Its review materials emphasize that scope, assumptions, and implementation details should be made explicit.
- Limitations: The paper includes a limitations section addressing the framework’s scope and areas for future improvement.This is explicitly identified in the review justification.
- Reproducibility: The paper states that its experimental datasets, setups, and methodologies are described sufficiently to reproduce the main results.The review cites comprehensive experimental details as support for reproducibility.
- Reproducibility: The reproducibility guidance requires clear reporting of architecture, model access, or procedures for recreating the algorithm, depending on the contribution.The applicable requirements differ for architectures, models, and algorithms.
- Open Materials: The paper provides a code-repository link and supplemental instructions for reproducing the experiments.The review explicitly cites both resources.
- Experimental Details: Reproducibility materials include data splits, sample counts, and numbers of seeds.The review identifies these details as relevant to understanding and reproducing the results.
7. Experiment Statistical Significance
The review materials report that the paper addresses statistical reporting and responsible research considerations. They also describe API-based inference and discuss societal risks, safeguards, and asset attribution.
- Statistical Reporting: The paper reports error bars and statistical-significance information for its experimental results.The review states that these materials clarify variability and reliability.
- Statistical Reporting: The statistical-reporting guidance requires explaining variability sources, error-bar calculations, and assumptions.It also distinguishes standard deviation, standard error, and confidence-interval conventions.
- Compute Setting: The experiments primarily use API calls to GPT-3.5, GPT-4 (V), and Gemini-Pro (Vision).The review states that compute-worker, memory, and execution-time details are managed by the API providers.
- Broader Impacts: The paper discusses positive and negative societal impacts, including improved medical decision-making and risks from misuse or bias.The review explicitly identifies both categories of impact.
- Safeguards: The paper addresses medical hallucinations and false knowledge through responsible-use guidelines for the framework and publicly available datasets.These safeguards are described in the review justification.