PaperlayerView exact revision on arXiv

Source-linked AI summary

MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making

Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, Hae Won Park

arXiv:2404.15155v3cs.CLcs.AIcs.LG

TL;DR

Medical decision-making is collaborative and often involves integrating multimodal information, yet how best to use LLMs for complex medical tasks remains an open question. MDAgents assigns solo or group collaboration based on task complexity and outperformed solo and group methods on 7 of 10 benchmarks.

  • Problem

    How best to use LLMs for complex medical decision-making remains open, given that clinicians collaboratively interpret complex, multimodal information.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    MDAgents assesses query complexity and assigns either a solo clinician agent or a multidisciplinary or integrated care team to answer it.

    Source: §3 · p. 3

  • Results

    7 of 10 medical benchmarks showed significantly better performance for MDAgents than solo and group methods (p < 0.05).

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    11.8% average accuracy improvement over baseline accompanied combined moderator review and external medical knowledge in group collaboration.

    Source: §4.3 · p. 9

    The framework is limited to multiple-choice question answering and does not capture the interactive, patient-centered nature of real-world diagnostics.

    Source: Limitations and Future Works · p. 18

Abstract

from arXiv · show

Foundation models are becoming valuable tools in medicine. Yet despite their promise, the best way to leverage Large Language Models (LLMs) in complex medical tasks remains an open question. We introduce a novel multi-agent framework, named Medical Decision-making Agents (MDAgents) that helps address this gap by automatically assigning a collaboration structure to a team of LLMs. The assigned solo or group collaboration structure is tailored to the medical task at hand, emulating real-world medical decision-making processes adapted to tasks of varying complexities. We evaluate our framework and baseline methods using state-of-the-art LLMs across a suite of real-world medical knowledge and medical diagnosis benchmarks, including a comparison of LLMs' medical complexity classification against human physicians. MDAgents achieved the best performance in seven out of ten benchmarks on tasks requiring an understanding of medical knowledge and multi-modal reasoning, showing a significant improvement of up to 4.2% (p < 0.05) compared to previous methods' best performances. Ablation studies reveal that MDAgents effectively determines medical complexity to optimize for efficiency and accuracy across diverse medical tasks. Notably, the combination of moderator review and external medical knowledge in group collaboration resulted in an average accuracy improvement of 11.8%. Our code can be found at https://github.com/mitmedialab/MDAgents.

1 Introduction

Medical decision-making requires clinicians to integrate complex, multimodal information, but generalist LLM collaboration does not reflect its adaptive, tiered process. MDAgents assigns collaboration according to task complexity to emulate this process.

1 Introduction

  • Medical decision-making integrates multimodal data and new research, while LLMs can synthesize medical literature and clinical information and support probabilistic and causal reasoning [1] [39].The introduction cautions that medical applications require careful design rather than a one-size-fits-all solution.Source: §1 · p. 1
  • MDAgents outperformed previous solo and group methods on 7 of 10 medical benchmarks, with performance-efficiency trade-offs as agent numbers varied.The authors also report greater robustness than solo and group methods across hyperparameters and ablations showing that MDAgents identifies an appropriate complexity level for each instance.Source: §1 · p. 2
  • The framework checks medical complexity, recruits agents accordingly, synthesizes their analyses, and returns a final decision.It is designed to emulate hierarchical diagnosis, from an individual clinician to collaborative teams.Source: §1 · p. 2, Figure 1 · p. 2
  • Generalist multi-agent systems have seen limited evaluation in health applications and do not effectively integrate systematic medical decision-making.The proposed framework adapts collaboration to case complexity, from uncomplicated cases to those requiring multiple specialist discussions or sequential consultations.Source: §1 · p. 2

2 Related Works

Prior work applies medical LLMs to knowledge and clinical tasks using domain-specific training and inference-time techniques. Multi-agent methods offer collaborative reasoning but often use fixed configurations, motivating MDAgents’ complexity-adaptive collaboration.

Language Models in Medical Decision-Making

  • Medical LLMs address exam questions, research, prediction, diagnosis, and generation through domain-specific training or inference-time methods such as prompting and retrieval.General-purpose models and frameworks including Medprompt, ensemble refinement, MEDIQ, and UoT build on latent medical knowledge, reasoning, and information-seeking.Source: §2 · p. 2

Multi-Agent Collaboration

  • Multi-agent approaches include role-playing, debate, voting, multidisciplinary collaboration, discussion, and negotiation, but commonly rely on fixed agent counts and interaction settings.The paper argues static configurations may be suboptimal across tasks and computationally costly; MDAgents dynamically assigns collaboration strategy at inference time based on query complexity. Table 1 states MDAgents alone performs all key dimensions of LLM decision-making.Source: §2 · p. 3, Table 1 · p. 3

3 MDAgents: Medical Decision-making Agents

MDAgents adapts medical decision-making to query complexity by routing cases to a solo clinician, a multidisciplinary team, or an integrated care team. Its workflow assesses complexity, recruits appropriate agents, coordinates analysis, and synthesizes a final answer.

3.1 Agent Roles

  • The moderator triages each query and oversees selection of a solo agent, MDT, or ICT, while the recruiter assembles generalist or specialist agents for the chosen pathway.General physicians handle routine cases; specialists contribute expertise to more complex scenarios and team decisions.Source: §3.1 · p. 4

3.2 Medical Complexity Classification (Line 1 of Algorithm 1, Appendix C.2)

  • A moderator classifies queries as low, moderate, or high complexity, corresponding to a single PCP, an MDT, or an ICT, respectively.The levels distinguish predictable, straightforward needs from cases requiring interdisciplinary coordination and extensive combined expertise.Source: §3.2 · p. 4, §3.4 · p. 5

3.3 Expert Recruitment (Line 3, 7, 17 of Algorithm 1)

  • The recruiter assigns medical expertise and roles to LLMs, configuring them to work independently or collaborate in groups or multiple teams according to the moderator’s assessment.Frequently recruited agents for each benchmark are provided as a reference in Appendix Figure 9.Source: §3.3 · p. 4

3.4 Medical Collaboration and Refinement

  • Low-complexity queries receive a single PCP’s few-shot response directly, while moderate-complexity queries use an MDT to integrate perspectives across specialties.The low-complexity answer requires no iterative refinement; moderate cases proceed through structured team collaboration.Source: §3.4 · p. 5
  • For moderate cases, MDT agents exchange messages for up to R rounds, assess consensus, and receive moderator feedback if they disagree.Agents indicate participation and preferred interlocutors; discussions can continue with access to prior conversations when consensus is not reached.Source: §3.4 · p. 5
  • High-complexity cases use an ICT that progresses from initial assessment through diagnostic teams to a final review team, with each team producing a report.The successive reports build into a cohesive synthesis of specialist findings.Source: §3.4 · p. 5

3.5 Decision-making

  • The decision-maker uses the initial response for low-complexity cases, interaction history for moderate cases, and agent reports for high-complexity cases to produce the final answer.When applicable, synthesis uses temperature ensembles to support a robust decision reflecting model consensus.Source: §3.5 · p. 5

4 Experiments and Results

MDAgents adapts collaboration complexity to medical tasks, outperforming baseline settings on most tested benchmarks while balancing accuracy, efficiency, and consensus. Ablations and robustness experiments examine complexity selection, moderator review, MedRAG, agent count, and temperature.

4.1 Setup

  • Experiments compared Solo, Group, and Adaptive settings on ten datasets, using 50 test samples per dataset and 3-shot prompts for low-complexity cases.Moderate- and high-complexity cases used zero-shot prompts; average inference times were 14.7s, 95.5s, and 226s for low, moderate, and high complexity, respectively.Source: §4.1 · p. 6

4.2 Results

  • The complexity classifier selected the highest-accuracy complexity level with probability a = 0.81 ± 0.29, versus 0.11 ± 0.28 for the middle and 0.08 ± 0.16 for the lowest.Across 25 medical problems repeated 10 times at each complexity, the results suggest the classifier aligned task difficulty with the most effective solution level.Source: §4.2 · p. 8
  • Solo methods reached 83.9% on MedQA versus 81.3% for Group, while Group reached 91.9% on SymCat versus 88.7% for Solo.Solo outperformed Group on four benchmarks, while Group outperformed Solo on six, including multimodal tasks Path-VQA, MedVidQA, and MIMIC-CXR.Source: §4.2 · p. 8
  • MDAgents significantly outperformed Solo and Group methods (p < 0.05), achieving the best performance on 7 of 10 benchmarks.The reported gains include 5.2% over the best Solo result and 9.5% over the best Group result on DDXPlus.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 Ablation Studies

  • Combined MedRAG and moderator review raised average accuracy to 80.3% from the 71.8% baseline, an 11.8% increase.MedRAG alone reached 75.2% (up 4.7%), and moderator review alone reached 77.6% (up 8.1%).Source: Table 3 · p. 9, §4.3 · p. 9
  • Adaptive accuracy on text-only queries was 81.2%, versus 64.2% at low, 71.6% at moderate, and 65.8% at high fixed complexity.The method classified 64% of text-only queries as high complexity, 55% of image-plus-text queries as low, and 87% of video-plus-text queries as low; the video result is limited by MedVidQA’s relatively less complex medical knowledge.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

4.4 Impact of Number of Agents in Group Setting.

  • Adaptive achieved peak accuracy of 83.5% with three agents and used 9.3 API calls, compared with 6.0 for Solo and 20.3 for Group with five agents.Increasing the number of Group agents did not improve performance; the Adaptive approach calibrated agent count to reduce calls while maintaining high accuracy.Source: §4.4 · p. 9, Figure 6 · p. 10

4.5 Robustness of MDAgents with different parameters.

  • Adaptive performance improved at higher temperature (T=1.2) versus lower temperature (T=0.3), and agent opinions converged across text-only, text-plus-image, and text-plus-video inputs.Text-plus-video showed faster convergence, while the other modalities had a more gradual entropy decline; evaluation across a wider temperature range remains future work.Source: §4.5 · pp. 9–10, Figure 6 · p. 10, Figure 7 · p. 10

5 Conclusion

MDAgents dynamically assigns LLMs to solo or group collaboration based on task complexity, emulating clinical decision-making. It outperformed previous solo and group methods on 7 of 10 medical benchmarks.

5 Conclusion

  • MDAgents outperformed previous solo and group methods in 7 of 10 medical benchmarks.The conclusion also reports that a case study showed agents converging on the correct diagnosis despite initially divergent perspectives.Source: §5 · p. 10
  • MDAgents adaptively assigns LLMs to independent roles or groups according to task complexity.The framework is designed to reflect nuanced clinical consultation and decision processes.Source: §5 · p. 10
  • Ablation studies identify the contributions and interactions of agents and strategies that drive the framework’s success.Source: §5 · p. 10
  • The case study illustrates how agents synthesize differing expert opinions and reach the correct diagnosis despite initially divergent perspectives.Source: §5 · p. 10

Limitations and Future Works

Despite promising performance, MDAgents has limitations in medical specialization, patient-centered interaction, and safety. The authors propose future work to address these gaps.

Medical Focused Foundation Models

  • Future work could incorporate medical-focused models Med-Gemini [63], AMIE, and Med-PaLM 2 [67] to improve agents’ shared medical vocabulary and collaboration.The authors propose these models to support more medically accurate content and communication among agents.Source: Limitations and Future Works · p. 18

Patient-Centered Diagnosis

  • Because MDAgents currently handles only multiple-choice questions, future work would add interactive patient and caregiver engagement and regret-aware decisions that seek more information under uncertainty.Feedback from prior outcomes could help minimize diagnostic regret and refine responses in complex, multi-stakeholder scenarios.Source: Limitations and Future Works · p. 18

Potential Risks and Mitigations

  • Potential medical hallucinations and misleading information could be mitigated with self-correction [42], rule-based safety rewards [56], and confidence or uncertainty estimates.The proposed self-correction mechanism uses reinforcement learning, while reward structures guide adherence to safety and accuracy guidelines.Source: Limitations and Future Works · p. 18

A Dataset Information … D Additional Results

The appendices describe the benchmark datasets and their complexity rationale, define entropy as a measure of agent consensus, and provide prompts for single-agent, multi-agent, and final-decision workflows.

A Dataset Information

  • The benchmarks vary in complexity: text-only, brief questions are rated low, image-based QA and pathology questions moderate, while multimodal or lengthy diagnostic tasks are generally more complex.Examples include MedQA, PMC-VQA, and Path-VQA.Source: A Dataset Information · pp. 18–19
  • Dataset descriptions span text, image, video, and clinical-case question answering, with the appendix also describing synthetic symptom-condition data and linked chest-radiograph questions.MedVidQA questions use video answers and GPT-4-generated multiple-choice options; the supplied text gives sample descriptions rather than a complete dataset table.Source: A Dataset Information · p. 20

B Entropy Calculation for Consensus Dynamics

  • Lower entropy indicates greater agreement among agents, and entropy trends across modalities show how quickly MDAgents reaches a unified decision.Entropy is calculated from the probabilities of unique answers; the supplied passages describe its purpose but do not provide trend values.Source: B Entropy Calculation for Consensus… · pp. 20–21

C Prompt Templates

  • Single-agent templates cover few-shot and chain-of-thought answering, ensemble refinement, and Medprompt-style questions.The templates structure medical multiple-choice queries with context, answer choices, and, for some variants, reasoning paths or similarity-selected examples.Source: §C.1 · p. 21
  • The multi-agent prompts first assess query complexity, then ask a recruiter to select expert roles and specify their communication structure.Complexity categories distinguish questions answerable by a general physician, those requiring team consultation, and those requiring multidisciplinary specialist effort.Source: §C.2 · p. 22

D Additional Results

  • The additional-results prompts let agents request discussion with another expert, then have a final decision maker review agent opinions and produce a final answer.The final-answer prompt may use decision methods such as majority voting or ensemble refinement, or synthesize reports into a refined summary.Source: D Additional Results · pp. 22–23

D.1 Accuracy on entire MedQA 5-options Dataset

On the entire MedQA 5-options dataset, MDAgents outperformed both single-agent and other multi-agent methods with GPT-4o mini. The evaluation extends the paper’s subsampled experiments to the full dataset.

D.1 Accuracy on entire MedQA 5-options Dataset

  • The full-dataset evaluation with GPT-4o mini expands on the subsampled MedQA 5-options experiments in Table 2.It reports accuracy results for various methods.Source: §D.1 · p. 23, Table 5 · p. 23
  • 83.6% accuracy: MDAgents outperformed both single-agent and other multi-agent methods on the full MedQA 5-options dataset.The evaluation used GPT-4o mini.Source: Table 5 · p. 23, §D.1 · p. 23

D.2 Estimated Costs for Full Test Set Experiments … E.1.1 Real-World Medical Cases

These sections report that MDAgents’ costs are higher because of its multi-agent design, while RAG and structured collaboration improve performance. They also describe complexity-based agent structures and illustrate low-, moderate-, and high-complexity medical cases.

D.2 Estimated Costs for Full Test Set Experiments

  • The authors estimate full-test-set GPT-4 (Vision) costs in USD and state that MDAgents’ multi-agent approach incurs higher costs, which they consider justified for critical medical decision-making.The supplied passages do not provide the table’s cost values.Source: §D.2 · p. 23, Table 6 · p. 23

D.3 Impact of Knowledge Enhancement with RAG

  • Moderator’s Review combined with MedRAG yielded the best performance, while role assignment alone provided a foundation for expert-like performance.The experiments investigated adding specific knowledge through Retrieval-Augmented Generation and structured reviews.Source: §D.3 · p. 24, Table 7 · p. 24

D.4 Complexity Assignment and Collaborative Settings

  • 59.0% was the highest accuracy for parallel collaboration with discussion; discussion raised accuracy from 39.0% to 45.0% sequentially and from 56.0% to 59.0% in parallel.The comparison concerns high-complexity image+text tasks.Source: §D.4 · p. 24, Table 8 · p. 24
  • The adaptive framework recruits a Primary Care Clinician for low-complexity queries, a multidisciplinary team with iterative discussion and moderator feedback for moderate queries, and an Integrated Care Team for high-complexity queries.For moderate cases, the moderator makes the final decision; for high-complexity cases, teams provide reports for a final decision.Source: §D.4 · p. 30

E.1 Medical Decision Making Case Studies

  • Medical decision-making in challenging cases requires both individual expertise and collaboration across specialties to seek the best possible outcomes.Source: §E.1 · p. 31

E.1.1 Real-World Medical Cases

  • The real-world examples span a low-complexity diabetes dosage adjustment, a moderate-complexity dengue diagnosis by collaborating clinicians, and a high-complexity case involving pulmonary effusion and a viral infection.The high-complexity case involves a patient with chronic heart failure and assessment by cardiology and infectious disease specialists.Source: §E.1.1 · p. 31

E.1.2 Medical Cases from MedQA Dataset · E.2 Cases Studies with MDAgents

The MedQA cases show how medical complexity shapes whether care can be handled by a PCP, a single specialist, or a multidisciplinary team. MDAgents case studies illustrate collaboration workflows for moderate and high complexity across image-plus-text and text-only scenarios.

E.1.2 Medical Cases from MedQA Dataset

  • The MedQA examples illustrate that checking case complexity helps determine whether individual or teamed decision-making is appropriate for management.They range from low- to high-complexity cases.Source: §E.1.2 · p. 31
  • In the low-complexity gastric-cancer case, a PCP is expected to use basic scientific and pathophysiological knowledge to answer the question without consulting a gastroenterologist.Management based on the disease’s manifestations is described as the gastroenterologist’s role.Source: §E.1.2 · pp. 31–32
  • For the moderate-complexity case of a girl with excess hair growth, an ovarian mass, and elevated estrogen, a pediatric endocrinologist can diagnose and plan treatment alone.The patient could have been referred by a PCP who regularly sees her.Source: §E.1.2 · pp. 31–32
  • The high-complexity case pairs neurological symptoms with vision problems, requiring a neurologist to consult ophthalmology for further evaluation.The example describes persistent double vision and a drooping eyelid, and asks for the next best test.Source: §E.1.2 · p. 32

E.2 Cases Studies with MDAgents

  • MDAgents’ moderate-complexity image-plus-text example uses five steps: complexity check, expert recruitment, initial assessment, collaborative discussion, and review with a final decision.Figure 11 marks correct answers in green and incorrect answers in red.Source: §E.2 · p. 32, Figure 11 · p. 33
  • The case studies show MDAgents providing an environment for agents to collaborate, gather information, moderate, and make final decisions in complex medical scenarios.Source: §E.2 · p. 33
  • The high-complexity text-only example uses four steps: complexity check, expert recruitment, information gathering and report generation, then a final decision.Figure 12 marks the correct answer in green.Source: §E.2 · p. 32, Figure 12 · p. 34

F Medical Complexity Comparison with Human Physicians · NeurIPS Paper Checklist · 1. Claims

Physicians showed moderate agreement, while LLM complexity ratings correlated only weakly with physician judgments; the section frames these gaps as support for MDAgents’ adaptive collaboration. The checklist states that the abstract and introduction describe the framework, its adaptive structure, and experimental performance gains.

F Medical Complexity Comparison with Human Physicians

  • The study compared ratings on 50 MedQA questions spanning USMLE steps 1–3, with three physicians assigning complexity scores of -1, 0, or 1.The physician group comprised two PGY-2 internal medicine trainees and one general physician.Source: F Medical Complexity Comparison with… · pp. 34–35
  • Physician agreement was moderate: ICC2k was 0.269 [-0.14, 0.55] and ICC3k was 0.280 [-0.15, 0.57].The authors interpret this agreement level as evidence of subjectivity in medical-complexity assessment, even among trained professionals.Source: F Medical Complexity Comparison with… · p. 35
  • The authors argue that subjective physician judgments and weak LLM alignment support dynamic collaboration rather than relying on one fixed complexity assessment.They also note the continued importance of human expertise and propose that collaboration can offset individual LLM limitations.Source: F Medical Complexity Comparison with… · pp. 35–36
  • LLM correlations with physician majority opinions ranged from -0.090 to 0.110, indicating weak alignment and variation across models.Physician majority opinions were defined by the mode of ratings, or the mean in cases of complete disagreement.Source: F Medical Complexity Comparison with… · pp. 35–36, Table 13 · p. 36

NeurIPS Paper Checklist · 1. Claims

  • The checklist justification says the abstract and introduction state MDAgents’ adaptive decision-making structure and experimental performance improvements.Source: §1 · p. 37

1. Claims

  • The checklist guidance says claims should match results, clarify scope and assumptions, and distinguish attained findings from aspirational goals.Source: §1 · p. 37

2. Limitations … 6. Experimental Setting/Details

The paper discusses limitations and reports no theoretical results requiring proofs; it also describes its reproducibility provisions and experimental details.

2. Limitations

  • The paper includes a limitations section discussing the framework’s scope and areas for future improvement.The supplied passages do not specify particular limitations or future improvements.Source: §2 · p. 37

3. Theory Assumptions and Proofs

  • The paper reports no theoretical results requiring assumptions or proofs.The theory checklist response is marked NA because the paper includes no such results.Source: §3 · p. 38

4. Experimental Result Reproducibility

  • The paper provides details on datasets, experimental setups, and methodologies to support reproduction of its main results.The reproducibility response is marked Yes.Source: §4 · p. 38

5. Open access to data and code

  • The paper links a code repository and provides supplemental instructions for reproducing the experiments.Source: §5 · p. 39

6. Experimental Setting/Details

  • The paper’s experimental details include data splits, sample counts, and numbers of seeds.The setting/details response says these particulars are specified for transparency and reproducibility.Source: §6 · p. 39

7. Experiment Statistical Significance

The paper reports error bars and information about the statistical significance of its experimental results.

7. Experiment Statistical Significance

  • The paper reports error bars and statistical-significance information for its experimental results.The supplied passage does not specify how the error bars were calculated or what variability they capture.Source: §7 · p. 39

8. Experiments Compute Resources … 14. Crowdsourcing and Research with Human Subjects

The checklist reports API-based experiments with reproducibility details, discusses potential benefits and risks, and marks several responsible-research disclosures as compliant or not applicable.

8. Experiments Compute Resources

  • The experiments primarily used API inference, with providers managing compute resources and experiment-specific API-call counts and configurations reported for reproducibility.The APIs were GPT-3.5, GPT-4 (V), and Gemini-Pro (Vision).Source: §8 · p. 40

9. Code Of Ethics

  • The paper states that its research conforms to the NeurIPS Code of Ethics.Source: §9 · p. 40

10. Broader Impacts

  • The paper discusses improved medical decision-making as a potential societal benefit and misuse or AI bias as potential risks.Source: §10 · p. 41

11. Safeguards

  • The paper discusses medical hallucinations and false knowledge as limitations and provides responsible-use guidelines for MDAgents and public datasets.Source: §11 · p. 41

12. Licenses for existing assets

  • The paper states that it credits existing assets and identifies their licenses and terms of use.Source: §12 · p. 41

13. New Assets

  • The paper reports that it introduces no new assets.Source: §13 · p. 42

14. Crowdsourcing and Research with Human Subjects

  • The paper reports no crowdsourcing or human-subject research, marking both participant-instruction and IRB disclosures as not applicable.Source: §14 · pp. 42–43

Related papers

Loading 2404.15155v3…