Source-linked AI summary
A Structured Debate-Mixture-of-Agents Framework for Complex Clinical Diagnostic Decision Support
Chang Xia, Leilei Ouyang, Huimin Wang, Yong Zhao, Kang Li
TL;DR
LLMs’ one-shot response format does not match the iterative reasoning required for complex clinical diagnosis. This paper introduces DMoA, a structured role-based multi-agent workflow, and reports improved diagnostic performance over single-model baselines while identifying model, structure, and token-budget effects.
Problem
Complex diagnosis requires iterative reasoning across uncertain, atypical, and heterogeneous clinical cases, whereas current LLMs remain limited by one-shot response patterns.
Method
DMoA organizes diagnosis into generation, critique, revision, and aggregation across role-specific agents rather than relying on one model’s single response.
Results
DMoA improved diagnostic accuracy and safety beyond single models and less structured collaborative settings across complex clinical tasks.
Takeaways & Limitations
The findings support further investigation of structured multi-agent systems for complex medical diagnosis.
Takeaways & Limitations
Rare-disease samples remain limited, cases came from published reports rather than prospective workflows, and performance remains constrained by base-model capability.
Abstract
from arXiv · showhide
Large language models (LLMs) show potential for medical tasks, but their single-turn question-answer format does not reflect how clinical diagnosis is performed in practice. As a result, they remain limited in complex diagnostic settings. We developed Debate-Mixture-of-Agents (DMoA), a novel multi-agent framework that structures role-based interaction to support iterative diagnostic reasoning. Base models and DMoA were evaluated on 297 rare disease cases and 1,719 challenging cases. Across both datasets, DMoA improved most likely diagnosis accuracy by 10.21 percentage points and safety rate by 11.36 percentage points over GPT-4o baseline. Ablation experiments showed that the gains were not simply due to the use of more models or longer outputs, but also reflected the contribution of the structured workflow. Further analyses examined how framework design, base model choice, and token budget affected performance. DMoA performed better with a 4*2 structure, stronger base models, and a larger token budget. These findings demonstrate the potential of DMoA for clinical tasks and suggest further investigation of multi-agent frameworks.
I. INTRODUCTION
Clinical diagnosis is iterative and especially difficult in rare and challenging cases, while single-model LLMs compress multiple reasoning components into one response. DMoA addresses this gap with role-constrained, multi-agent diagnostic reasoning.
- Clinical diagnosis is a dynamic, iterative reasoning process rather than a single-step decision.
- Rare diseases and published challenging cases represent distinct settings with high uncertainty, atypical presentations, and heterogeneous clinical evidence.
- Single-model architectures may be less suited to complex diagnosis because they compress multiple diagnostic reasoning components into one response pathway.
- DMoA extends Mixture-of-Agents with role-constrained organization and debate-based interaction for complex diagnostic reasoning.
- The study evaluates DMoA on rare-disease and challenging-case diagnosis while analyzing accuracy, safety, ablations, scaling, token budget, deployment, and cost.
A. Study design
The study evaluates DMoA on clinical case datasets and supplementary knowledge questions, using multiple role-specific agents to generate, critique, revise, and aggregate diagnostic outputs.
- Datasets and tasks: The evaluation uses rare-disease and challenging-case datasets, plus a supplementary set of foundational clinical knowledge questions.
- Datasets and tasks: The primary diagnostic cases contain patient histories, examinations, tests, and other clinical details, with reference diagnoses extracted from the original reports.
- Data source: All materials came from published, publicly identifiable sources; no real patients were enrolled or contacted.
- Evaluation outputs: Each model receives identical instructions to produce the most likely diagnosis and three possible diagnoses for diagnosis–reference comparison.
- DMoA framework: Each DMoA Debater uses a Presenter and Rebutter: the Presenter generates an initial analysis, the Rebutter critiques it, and the Presenter revises the diagnosis.
- DMoA framework: The Aggregator synthesizes all revised Debater outputs and returns the final top-1 and top-3 diagnoses.
E. Ablation Studies
The ablation study tests whether DMoA’s performance gains can be reproduced by prompt engineering alone, without the explicit multi-agent framework.
- The ablation evaluates whether a single model can simulate the DMoA workflow through a stepwise debate prompt alone.
F. Diagnosis Results Evaluation Method
The study evaluates diagnostic correctness through reference matching and diagnostic safety through a multidimensional rating framework, with automated evaluation checked by independent LLM judges.
- Accuracy: A diagnosis is accurate when the most likely diagnosis or any of three possible diagnoses exactly matches the reference diagnosis.
- Safety: Safety evaluation uses a five-point scale covering correctness, critical misses, false alarms or overdiagnosis, demographic applicability or bias, and uncertainty calibration for three-diagnosis outputs.
- Safety: A diagnostic output is classified as unsafe when any evaluated safety dimension receives a score below 3.
- Reliability: GPT-4o Mini performs automatic evaluation, while 1,000 outputs per diagnostic format are re-evaluated by GPT-4o and Gemini-2.5 Flash for cross-evaluator agreement.
G. Lightweight Model Combination Analysis
The study evaluated whether lightweight models could be used within DMoA, including configurations that combine lightweight and general-scale models. It also examined foundational clinical knowledge performance across four task dimensions.
- Lightweight models were defined as models with fewer than 10 billion parameters.
- The lightweight setting used lightweight models for the Presenter, Rebutter, and Aggregator and compared them with corresponding single-model baselines.
- The mixed-scale setting used a general-scale model for the Aggregator and lightweight models for the Presenter and Rebutter.
- Additional analyses examined DMoA structure, patient-information partitioning across Debaters, and token-budget constraints.
- The supplementary clinical-knowledge experiment evaluated accuracy across four task dimensions and overall performance.
J. Statistical Analysis
The study used paired statistical tests, error categorization, reliability assessment, and cost analysis to evaluate DMoA and its diagnostic outputs.
- Paired comparisons between single-model baselines, DMoA, and ablation or configuration settings used one-sided McNemar’s tests.
- Supplementary Q&A comparisons between DMoA and corresponding single-model settings were also evaluated with one-sided McNemar’s tests.
- Diagnostic errors were classified by proximity to the reference diagnosis, including near-miss and clinically informative alternative diagnoses.
- Reliability was assessed using three replicate runs across all 96 experiments and quantified with Fleiss’ kappa.
- The study calculated average cost per case for GPT-4o mini, GPT-3.5, GPT-4o, and other model combinations.
- The DMoA workflow used Presenter, Rebutter, and Aggregator roles, with revised Debater outputs synthesized into the final diagnosis.
B. Tool Agents and Panel
The paper describes user-facing agents and a panel supporting structured patient-information preparation and human oversight, alongside DMoA performance across both clinical datasets.
- Tool Agents and Panel: The Information Editor supports manual case entry and converts unstructured descriptions into structured clinical fields.
- Tool Agents and Panel: The Doctor Supervisor provides a human-in-the-loop function during the diagnostic process.
- Tool Agents and Panel: A user panel was developed, with its images and instructions provided in Supplementary Note 1.
- Performance: GPT-4o had the strongest overall single-model diagnostic performance across rare-disease and challenging-case diagnosis.
- Performance: In the rare-disease subset, GPT-4o achieved 28.62% accuracy and 58.92% safety rate for the most likely diagnosis under general prompts.
- Performance: In the challenging-case subset, GPT-4o achieved 14.50% accuracy and 63.46% safety rate for the most likely diagnosis.
- Performance: DMoA consistently outperformed corresponding single-model baselines in diagnostic accuracy and safety across both sub datasets.
- Performance: For GPT-4o on rare diseases, 4×2 DMoA achieved 35.20% versus 28.62% most-likely-diagnosis accuracy and 62.63% versus 58.92% safety rate.
E. Component Validation of the DMoA Topology
Ablation experiments tested whether DMoA’s gains could be reproduced by prompting, removing rebuttal, or using unstructured dialogue. The full topology performed better than these reduced configurations.
- Ablation design: The ablation study compared full DMoA with prompt-only emulation, MoA without rebuttal, and MAC-like dialogue without cross-Debater aggregation.
- Prompt-only emulation: Prompt-only emulation improved GPT-4o top-1 accuracy from 16.58% to 20.10% but remained below the 4×2 DMoA framework.
- Rebutter ablation: Removing the Rebutter reduced GPT-4o top-1 accuracy from 26.79% to 25.26% and top-1 safety rate from 74.15% to 72.95%.
- Evaluation scope: The component comparisons covered accuracy and safety across whole, rare-disease, and challenging-case datasets.
- Rebutter ablation: The Rebutter contributed to diagnostic refinement before aggregation and was not redundant.
- Unstructured dialogue: MAC-like dialogue achieved 24.26% top-1 accuracy versus 26.79% for DMoA and 72.86% top-1 safety versus 74.15%.
F. Case-level Analysis of Rebuttal-guided Revision
DMoA’s structured debate workflow supported diagnostic refinement and performance depended on configuration choices, while some added components did not consistently help.
- Case-level revision: DMoA progressively refined a case diagnosis by identifying missing evidence, reframing the injury, and aggregating revised hypotheses.The Rebutter highlighted absent trauma, repetitive microtrauma, growth plate involvement, and imaging findings before the Presenter and Aggregator revised the diagnosis.
- Base-model combinations: A mixed setting with GPT-3.5-Turbo as aggregator and lightweight models in other roles showed no clear advantage over the GPT-3.5-Turbo single model.For rare disease most likely diagnosis, accuracy was 19.53% versus 22.22% and safety rate was 53.66% versus 59.80%.
- Configuration factors: The 4×2 structure achieved the highest GPT-4o-based performance among tested structures, reaching 35.20% accuracy and 62.63% safety rate for most likely diagnosis.For three possible diagnoses, it also achieved the highest accuracy and safety rate, at 50.84% and 71.72%.
- Configuration factors: Reducing the token budget from 16,000 to 4,000 decreased most likely diagnosis accuracy from 35.20% to 30.98% and safety rate from 62.63% to 59.26%.Information partitioning did not improve performance, with the partitioned 3×2 setting showing lower accuracy and similar safety rate than the non-partitioned setting.
- Ablation findings: Adding a reward model lowered most likely diagnosis accuracy from 35.20% to 31.65% while leaving three possible diagnoses accuracy nearly unchanged at 50.17% versus 50.84%.Most likely diagnosis safety rate increased, but three possible diagnoses safety rate remained essentially unchanged.
I. Supplementary Experiments on Clinical Knowledge Q&A
Supplementary evaluations examined DMoA in clinical knowledge question answering, error characterization, configuration effects, and test–retest reliability.
- Clinical knowledge Q&A: DMoA showed little or no advantage over single models in basic clinical knowledge question answering, with both exceeding 70% accuracy across four question types.The evaluated categories covered foundational science, practice-based learning, professionalism and ethics, and systems-based practice and patient safety.
- Error analysis: Diagnostic accuracy errors were grouped into four levels according to their distance from the correct diagnosis.The table percentages represent the proportion of all cases.
- Configuration analysis: Figure 6 examined lightweight-model settings, token budget, information partitioning, and DMoA structure using bars and lines representing percentages.These configuration factors were evaluated as part of the supplementary experiments.
- Reliability analysis: Across three repeated test rounds, 76 of 96 single-model and DMoA settings showed substantial agreement with Fleiss’ kappa values of 0.61–0.80.The reported differences between testing rounds were small.
L. Cost Analysis
The cost analysis reports average per-case costs for the 4×2 DMoA framework across two datasets and two GPT base-model choices.
- GPT-4o mini: Using GPT-4o mini, average cost per case for three possible diagnoses was $0.0089 in rare disease and $0.0094 in challenging cases.These figures apply to the 4×2 DMoA framework.
- GPT-4o: Using GPT-4o, average cost per case was $0.1377 in rare disease and $0.1307 in challenging cases.These figures also apply to three possible diagnoses in the 4×2 DMoA framework.
- Comparison: The reported costs varied by base model and dataset while using the same 4×2 DMoA framework and three-possible-diagnoses task.The analysis compares GPT-4o mini and GPT-4o across rare disease and challenging case datasets.
IV. DISCUSSION
The discussion frames DMoA as a structured, iterative alternative to one-shot diagnosis, with benefits that are most apparent in complex, uncertain cases rather than basic medical question answering. It also identifies workflow structure, model capability, configuration, evaluation criteria, and deployment context as important determinants of performance and interpretation.
- Framework and rationale: DMoA organizes diagnosis into generation, challenge, revision, and aggregation, distributing reasoning across role-specific stages instead of one model’s single response.The framework is intended to address the gap between one-shot model outputs and the structured workflow of clinical diagnosis.
- Framework and rationale: DMoA’s gains depended on structured modules: imitating its instructions with one model or using unstructured multi-round dialogue performed below the full framework.The analysis attributes the difference to explicit internal critique and workflow structure rather than prompting or agent count alone.
- Task dependence: DMoA produced marginal or no improvement on basic clinical question answering, indicating that its advantages are more apparent in complex diagnostic tasks than in factual knowledge questions.The discussion characterizes its amplification of medical factual knowledge as limited.
- Design factors: The 4 × 2 configuration performed best, while stronger base models and compatible model capabilities were associated with more effective collaboration.Increasing structural complexity did not always improve performance, and large capability gaps between roles were not necessarily beneficial.
- Evaluation and limitations: The study used exact reference matching for accuracy and evaluated five safety dimensions, underscoring that diagnostic performance should not be judged by accuracy alone.The discussion also cautions against direct comparison with clinician results because the metrics, cases, and protocols differ.