PaperlayerView exact revision on arXiv

Source-linked AI summary

Role-Specialized Mixture-of-Agents with Open-Weight LLMs for Clinical Prediction

Jun Hou, Yi Fang, Xuan Wang

arXiv:2608.22176v1cs.AIcs.LG

TL;DR

Clinical prediction needs locally deployable, training-free alternatives to proprietary LLMs because privacy and compliance concerns limit closed models. This paper evaluates a role-specialized open-weight Mixture-of-Agents with fixed retrieval, finding that a small integrator with large analysts matches closed-model mortality F1 while favoring higher sensitivity, with weaker effects for readmission.

  • Problem

    Privacy and compliance concerns create a need for open-weight, locally hosted, training-free clinical LLM systems that can approach closed-source performance.

  • Method

    The system uses two large contrastive analysts and a final integrator, while fixing retrieval resources and exemplar pools to isolate role and model effects.

  • Results

    The pipeline matches API-based Claude-3.5 few-shot prompting on MIMIC-IV mortality F1 (+1.4 points) while improving Sensitivity by 45.3 points; replacing only the integrator raises Sensitivity from 13.0 to 64.8.

  • Takeaways & Limitations

    Model placement across roles is a key factor in open-weight clinical prediction, especially for high-recall screening without closed APIs or task-specific fine-tuning.

  • Takeaways & Limitations

    The evaluation uses MIMIC without external or prospective validation, and the advantage is a threshold difference expressed as higher Sensitivity rather than a uniform gain.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly applied to clinical prediction tasks such as in-hospital mortality and readmission from electronic health records (EHRs). Privacy and compliance constraints motivate systems that can be deployed locally, which has increased interest in open-weight multi-agent designs. However, most medical multi-agent systems are evaluated as a single block, leaving unclear which agent role contributes to prediction and whether retrieval drives observed gains. We study a role-specialized Mixture-of-Agents (MoA) that combines medical knowledge retrieval with contrastive similar-patient reasoning. By varying the role design while holding the retrieval setup fixed, we localize the main effect to the final integrator. Pairing large open-weight analysts with a small open-weight integrator matches closed-model prompting on F1 for mortality prediction while flagging substantially more true high-risk patients. Mechanism analysis shows the role assignment directly yields a high-recall operating point without threshold tuning. The effect is task-dependent, with smaller gains for readmission because the available records correlate weakly with this longer-horizon outcome. These results position role design as a key factor in privacy-constrained, training-free clinical LLM prediction.

1 Introduction

The paper addresses the need for locally deployable, training-free clinical LLM systems by testing whether role-specialized open-weight agents can approach closed-source performance. Its controlled design identifies the integrator as the main source of the mortality gain and links the effect to a shifted decision threshold.

  • Privacy and compliance concerns motivate open-weight, locally hosted, training-free clinical LLM systems that can approach closed-source performance.
  • Existing medical multi-agent systems are often evaluated as single blocks, leaving unclear which role drives prediction and whether open-weight models can match closed-source models without training.
  • The proposed role-specialized Mixture-of-Agents uses two contrastive analysts and a final integrator while keeping knowledge sources fixed on MIMIC datasets.
  • The large-analyst, small-integrator configuration matches a closed-model few-shot mortality F1 while identifying substantially more high-risk patients.
  • Role isolation localizes the gain to the integrator, whose threshold shift improves mortality sensitivity but not readmission because pre-discharge records carry less signal for post-discharge outcomes.

2 Related Work

Related work spans knowledge-augmented EHR prediction and collaborative medical multi-agent reasoning. This paper differs by studying training-free, open-weight role assignment within a fixed Mixture-of-Agents workflow.

  • Knowledge-augmented EHR prediction: Knowledge-augmented EHR methods incorporate ontologies, medical knowledge, or retrieved and generated knowledge to improve clinical prediction.
  • Knowledge-augmented EHR prediction: KARE distills closed-source LLM reasoning into a small language model for state-of-the-art EHR prediction, whereas this paper studies training-free open-weight role design.
  • Medical multi-agent reasoning: Medical multi-agent systems use role-playing, adaptive collaboration, debate, or independent-output aggregation to support medical reasoning and EHR modeling.
  • Role assignment: The paper holds the Mixture-of-Agents workflow fixed while assigning contrastive exemplar readings to specialized analysts and reconciliation to the integrator.

3 Methods

The pipeline distributes clinical prediction across two label-blind contrastive analysts and a final integrator, using medical knowledge and patient exemplars retrieved from separate databases. The integrator combines the readings and converts its probability into a binary prediction at a 0.5 threshold.

  • Role-specialized pipeline: The pipeline distributes outcome prediction across three specialized roles: a risk analyst, a protective analyst, and a final integrator.
  • Retrieval: Medical knowledge is retrieved for the target record or an integrator query, while patient retrieval supplies one positive-outcome and one negative-outcome exemplar.
  • Integrator: The integrator reconciles the two analyst readings and performs additional focused retrieval before producing the final probabilistic prediction.
  • Contrastive analysts: The risk analyst compares the target record with the positive exemplar, and the protective analyst compares it with the negative exemplar without making an early prediction.
  • Prediction rule: The integrator outputs positive-class probability p̂ and negative-class probability 1 − p̂, then thresholds p̂ at 0.5 to obtain the binary prediction ŷ.

4 Experiments & Results

Experiments use fixed retrieval resources and exemplar pools to isolate role and model effects across MIMIC mortality and readmission tasks. The mixed large-analyst, small-integrator assignment improves mortality sensitivity and F1 relative to relevant prompting baselines, while readmission remains limited by weak record signal.

  • Experimental setup: The evaluation uses KARE train/test splits across MIMIC-III and MIMIC-IV mortality and 15-day readmission tasks, with full test sets for the main results.
  • Experimental controls: Retrieval resources, retrievers, and exemplar pools remain fixed across configurations, so the main varied factors are role topology and model assignment.
  • Role isolation: Replacing only the integrator with the small model raises MIMIC-IV mortality Sensitivity from 13.0 to 64.8 and yields the best macro-F1 and within-cell AUROC on both mortality cells.
  • Main results: The mixed assignment matches Claude-3.5 few-shot prompting on MIMIC-IV mortality F1 by +1.4 points while improving Sensitivity by 45.3 points.
  • Main results: Table 1 compares analyst, retrieval, and integrator assignments using F1, Sensitivity, Specificity, and within-cell AUROC across both mortality cells.
  • Main results: Table 2 reports Accuracy, F1, and Sensitivity across four task-dataset cells and training regimes, including training-free RAG-plus-exemplar configurations.
  • Ablations: Component ablations compare single versus multi-agent topology, MedRAG retrieval, and KARE exemplars across all four cells.

5 Model Analysis

Model placement, rather than retrieval alone, determines the pipeline’s operating point: large analysts paired with a small integrator produce the strongest mortality results, while topology enables retrieved exemplars to help.

  • Model size in the pipeline: AUROC 0.603 on MIMIC-IV mortality is achieved by the mixed assignment with large analysts and a small integrator.This assignment avoids the conservative all-large system and the low-discrimination all-small system, while giving the best F1, Sensitivity, and within-cell AUROC on mortality.
  • Model size in the pipeline: The mixed model assignment shifts sensitivity through the integrator rather than through analyst placement or retrieval changes.The analysis varies model placement while keeping role instructions fixed, isolating assignment effects.
  • Topology and exemplars: Retrieved patient exemplars improve single-agent performance but remain below multi-agent configurations unless split across contrastive analyst roles.On MIMIC-IV mortality, large single-agent F1 rises from 37.7 with single+RAG to 52.8 with single+RAG+Sim, but the small single agent reaches Sensitivity 94.7 at Specificity 23.7.
  • Topology and retrieval: The multi-agent topology carries the gain, while medical-knowledge retrieval is close to neutral.With all-large models, removing MedRAG changes F1 from 54.4 for multi+RAG+Sim to 51.3 for multi+Sim.
  • Risk separation: Tjur’s D measures separation between mean predicted risk for actual positives and negatives, while AUROC and bootstrap intervals compare pipeline conditions against null values.Figure 2 uses Tjur’s D and AUROC for CoT, single+RAG+Sim, and multi+RAG+Sim on MIMIC-IV mortality.

6 Error Analysis

The integrator’s lower threshold flags more patients, improving mortality sensitivity, but the same condition-count heuristic is poorly matched to readmission because pre-discharge records contain weak signal.

  • Integrator behavior: 30.2 percentage points is the increase in predicted-positive rate when replacing gpt-oss with Qwen on MIMIC-IV mortality.The integrators use similar evidence but cross the positive threshold at different numbers of recorded conditions.
  • Risk distributions: Figure 3 plots mean predicted risk against recorded-condition count among patients without the event, with standard-error bars.The figure supports comparison of how integrators map comorbidity counts to risk in mortality and readmission.
  • Mortality errors: 56% of decedents missed by gpt-oss on MIMIC-IV mortality would be flagged by the Qwen integrator.Qwen crosses the condition-count threshold earlier, trading moderate-severity survivor false positives for fewer missed decedents.
  • Readmission signal: AUROC 0.77 for mortality versus 0.48 to 0.59 for readmission shows that available codes contain substantially weaker signal for the longer-horizon outcome.The comparison uses five-fold cross-validated bag-of-codes classifiers on identical input types with different labels.
  • Readmission errors: The Qwen integrator continues raising predicted risk with condition count among non-readmitted patients, whereas gpt-oss is much flatter.The reported correlations are ρ = 0.39 for Qwen and ρ = 0.12 for gpt-oss on readmission.

7 Conclusion

The paper presents a training-free, locally deployable open-weight MoA for clinical prediction and finds that role assignment, especially the integrator, is central to its high-recall behavior.

  • Contribution: The training-free pipeline uses large open-weight models as contrastive analysts and a small model as integrator.The design is evaluated for clinical prediction without task-specific fine-tuning.
  • Main conclusion: On MIMIC-IV mortality, the role-specialized pipeline matches API-based Claude-3.5 few-shot prompting on F1 while achieving higher Sensitivity than closed-model prompting and supervised pre-trained baselines.The paper identifies the gain with the integrator’s decision threshold and model-family differences in mapping recorded conditions to risk.
  • Implication: Role placement is presented as a key factor for open-weight clinical prediction, particularly in high-recall screening settings that avoid closed APIs and task-specific fine-tuning.Fine-tuned models such as KARE remain strongest overall.

8 Limitations

The evaluation is narrow in datasets, model families, role decompositions, retrieval settings, outcomes, and validation, and its advantage is limited to a risk-sensitive threshold regime rather than uniform performance gains.

  • The study uses MIMIC data without external or prospective validation and evaluates only Qwen and gpt-oss model families.
  • Only MIMIC-IV mortality shows clear discrimination for the mixed assignment; readmission lacks a principled per-task configuration winner.The authors use one fixed configuration for readmission rather than selecting different configurations per task.
  • The reported advantage is a decision-threshold difference yielding higher Sensitivity, while other baselines lead on F1, Accuracy, or AUROC.The authors state that the method matches neither the fine-tuned SOTA KARE on F1 or Accuracy nor the large model on AUROC.
  • The design evaluates one analyst-integrator decomposition, one exemplar per outcome class, and no alternative Top k retrieval setting.
  • Intermediate outputs received no clinician review, and outcomes beyond mortality and readmission, such as length of hospital stay, were not evaluated.

Ethics Statement

The research uses credentialed public MIMIC datasets and locally run open-weight models, so patient data is not transmitted to external language-model services. AI assistance supported implementation and manuscript preparation, with author review and revision of generated content.

  • MIMIC-III v1.4 and MIMIC-IV v2.0 are publicly available under PhysioNet’s Credentialed Health Data Use Agreement.The agreement requires credentialed access and human-subjects research training.
  • All pipeline models are open-weight and run locally, preventing patient data transmission to external or third-party language-model services.
  • AI assistants helped generate LaTeX table code and refine author-written text, while the authors reviewed and revised all AI-generated content.

A AUROC / AUPRC (score-based metrics)

Score-based metrics complement threshold metrics by measuring discrimination for methods that expose probabilities. Mortality retains a large-model discrimination advantage, while readmission is near chance across methods.

  • AUROC 0.763 for oss CoT, m4 is the mortality discrimination result highlighted for the large single model.Here, m4 denotes MIMIC-IV mortality.
  • AUROC ≈0.50 for every method on readmission indicates near-chance discrimination.
  • Table 5 reports AUROC and AUPRC percentages for methods exposing probabilities across MIMIC-III/IV mortality and readmission.The table labels these datasets and tasks as m3/m4 and rd3/rd4.

B Discrimination-slope (mechanism) analysis

The analysis shows that retrieval changes probability separation, while the multi-agent integrator restores mortality discrimination without reducing the finding to a threshold shift. This mechanism is strong for MIMIC-IV mortality but weaker or absent for readmission and under-powered MIMIC-III mortality.

  • Metric and pipeline: Tjur’s D measures signed separation between mean predicted risks for outcome and non-outcome groups, alongside AUROC.Positive D indicates correctly separated groups, while D≈0 indicates no spread and D<0 indicates reversed separation.
  • Mechanism: Single+RAG+Sim reduces probability-level separation, whereas multi+RAG+Sim restores the discriminative signal.The pattern indicates distributional reshaping rather than merely moving the 0.5 decision threshold.
  • MIMIC-IV mortality: 0.763→0.638→0.730 AUROC across CoT, single+RAG+Sim, and multi+RAG+Sim shows the MIMIC-IV mortality effect is ranking discrimination.For oss, Tjur’s D changes 0.170→0.071→0.121; for qwen, it changes 0.137→0.103→0.161.
  • MIMIC-III mortality: MIMIC-III mortality follows the same direction, but rare positives produce noisy estimates and wide confidence intervals.The oss single condition reaches −0.035 without statistical significance, so this dataset corroborates direction rather than resolving mechanism.
  • Readmission: Readmission shows no reliable probability spread: oss-multi is mildly negative on both datasets, while qwen·m4 multi is only +0.033.Retrieval content was matched across topologies, supporting an integration effect for mortality; the readmission null is analyzed separately.
  • Role isolation: Replacing only the integrator with the small model lifts mortality F1 from 40→47 and 38→57 and sensitivity from 26→76 and 13→65.Replacing only an analyst or retrieval model does not produce this gain; the large-analyst, small-integrator default is best on mortality cells.
Loading 2608.22176v1…