PaperlayerView exact revision on arXiv

Source-linked AI summary

Role-Specialized Mixture-of-Agents with Open-Weight LLMs for Clinical Prediction

Jun Hou, Yi Fang, Xuan Wang

arXiv:2608.22176v1cs.AIcs.LG

TL;DR

Clinical prediction needs locally deployable, training-free alternatives to proprietary LLMs because privacy and compliance concerns limit closed models. This paper evaluates a role-specialized open-weight Mixture-of-Agents with fixed retrieval, finding that a small integrator with large analysts matches closed-model mortality F1 while favoring higher sensitivity, with weaker effects for readmission.

  • Problem

    Privacy and compliance concerns create a need for open-weight, locally hosted, training-free clinical LLM systems that can approach closed-source performance.

    Source: §1 · p. 1

  • Method

    The system uses two large contrastive analysts and a final integrator, while fixing retrieval resources and exemplar pools to isolate role and model effects.

    Source: §1 · p. 1, §3 · p. 3, §4.1 · p. 4

  • Results

    The pipeline matches API-based Claude-3.5 few-shot prompting on MIMIC-IV mortality F1 (+1.4 points) while improving Sensitivity by 45.3 points; replacing only the integrator raises Sensitivity from 13.0 to 64.8.

    Source: §4.3 · p. 4, §4.2 · p. 4

  • Takeaways & Limitations

    Model placement across roles is a key factor in open-weight clinical prediction, especially for high-recall screening without closed APIs or task-specific fine-tuning.

    Source: §7 · pp. 8–9

    The evaluation uses MIMIC without external or prospective validation, and the advantage is a threshold difference expressed as higher Sensitivity rather than a uniform gain.

    Source: §8 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly applied to clinical prediction tasks such as in-hospital mortality and readmission from electronic health records (EHRs). Privacy and compliance constraints motivate systems that can be deployed locally, which has increased interest in open-weight multi-agent designs. However, most medical multi-agent systems are evaluated as a single block, leaving unclear which agent role contributes to prediction and whether retrieval drives observed gains. We study a role-specialized Mixture-of-Agents (MoA) that combines medical knowledge retrieval with contrastive similar-patient reasoning. By varying the role design while holding the retrieval setup fixed, we localize the main effect to the final integrator. Pairing large open-weight analysts with a small open-weight integrator matches closed-model prompting on F1 for mortality prediction while flagging substantially more true high-risk patients. Mechanism analysis shows the role assignment directly yields a high-recall operating point without threshold tuning. The effect is task-dependent, with smaller gains for readmission because the available records correlate weakly with this longer-horizon outcome. These results position role design as a key factor in privacy-constrained, training-free clinical LLM prediction.

1 Introduction

The paper addresses the need for locally deployable, training-free clinical LLM systems by testing whether role-specialized open-weight agents can approach closed-source performance. Its controlled design identifies the integrator as the main source of the mortality gain and links the effect to a shifted decision threshold.

  • Privacy and compliance concerns motivate open-weight, locally hosted, training-free clinical LLM systems that can approach closed-source performance.Source: §1 · p. 1
  • Existing medical multi-agent systems are often evaluated as single blocks, leaving unclear which role drives prediction and whether open-weight models can match closed-source models without training.Source: §1 · p. 1
  • The proposed role-specialized Mixture-of-Agents uses two contrastive analysts and a final integrator while keeping knowledge sources fixed on MIMIC datasets.Source: §1 · p. 1
  • The large-analyst, small-integrator configuration matches a closed-model few-shot mortality F1 while identifying substantially more high-risk patients.Source: §1 · pp. 1–2
  • Role isolation localizes the gain to the integrator, whose threshold shift improves mortality sensitivity but not readmission because pre-discharge records carry less signal for post-discharge outcomes.Source: §1 · p. 2

2 Related Work

Related work spans knowledge-augmented EHR prediction and collaborative medical multi-agent reasoning. This paper differs by studying training-free, open-weight role assignment within a fixed Mixture-of-Agents workflow.

Knowledge-augmented EHR prediction

  • Knowledge-augmented EHR methods incorporate ontologies, medical knowledge, or retrieved and generated knowledge to improve clinical prediction.Source: §2.1 · p. 2
  • KARE distills closed-source LLM reasoning into a small language model for state-of-the-art EHR prediction, whereas this paper studies training-free open-weight role design.Source: §2.1 · p. 2

Medical multi-agent reasoning

  • Medical multi-agent systems use role-playing, adaptive collaboration, debate, or independent-output aggregation to support medical reasoning and EHR modeling.Source: §2.2 · p. 2

Role assignment

  • The paper holds the Mixture-of-Agents workflow fixed while assigning contrastive exemplar readings to specialized analysts and reconciliation to the integrator.Source: §2.2 · p. 2

3 Methods

The pipeline distributes clinical prediction across two label-blind contrastive analysts and a final integrator, using medical knowledge and patient exemplars retrieved from separate databases. The integrator combines the readings and converts its probability into a binary prediction at a 0.5 threshold.

Role-specialized pipeline

  • The pipeline distributes outcome prediction across three specialized roles: a risk analyst, a protective analyst, and a final integrator.Source: §3 · pp. 2–3

Retrieval

  • Medical knowledge is retrieved for the target record or an integrator query, while patient retrieval supplies one positive-outcome and one negative-outcome exemplar.Source: §3 · pp. 2–3

Integrator

  • The integrator reconciles the two analyst readings and performs additional focused retrieval before producing the final probabilistic prediction.Source: Figure 1 · p. 3, §3 · p. 3

Contrastive analysts

  • The risk analyst compares the target record with the positive exemplar, and the protective analyst compares it with the negative exemplar without making an early prediction.Source: §3 · p. 3

Prediction rule

  • The integrator outputs positive-class probability p̂ and negative-class probability 1 − p̂, then thresholds p̂ at 0.5 to obtain the binary prediction ŷ.Source: §3 · pp. 3–4

4 Experiments & Results

Experiments use fixed retrieval resources and exemplar pools to isolate role and model effects across MIMIC mortality and readmission tasks. The mixed large-analyst, small-integrator assignment improves mortality sensitivity and F1 relative to relevant prompting baselines, while readmission remains limited by weak record signal.

Experimental setup

  • The evaluation uses KARE train/test splits across MIMIC-III and MIMIC-IV mortality and 15-day readmission tasks, with full test sets for the main results.Source: §4.1 · p. 4

Experimental controls

  • Retrieval resources, retrievers, and exemplar pools remain fixed across configurations, so the main varied factors are role topology and model assignment.Source: §4.1 · p. 4

Role isolation

  • Replacing only the integrator with the small model raises MIMIC-IV mortality Sensitivity from 13.0 to 64.8 and yields the best macro-F1 and within-cell AUROC on both mortality cells.Source: §4.2 · p. 4

Main results

  • The mixed assignment matches Claude-3.5 few-shot prompting on MIMIC-IV mortality F1 by +1.4 points while improving Sensitivity by 45.3 points.Source: §4.3 · pp. 4–5
  • Table 1 compares analyst, retrieval, and integrator assignments using F1, Sensitivity, Specificity, and within-cell AUROC across both mortality cells.Source: Table 1 · p. 5
  • Table 2 reports Accuracy, F1, and Sensitivity across four task-dataset cells and training regimes, including training-free RAG-plus-exemplar configurations.Source: Table 2 · p. 5

Ablations

  • Component ablations compare single versus multi-agent topology, MedRAG retrieval, and KARE exemplars across all four cells.Source: Table 3 · p. 6

5 Model Analysis

Model placement, rather than retrieval alone, determines the pipeline’s operating point: large analysts paired with a small integrator produce the strongest mortality results, while topology enables retrieved exemplars to help.

Model size in the pipeline

  • AUROC 0.603 on MIMIC-IV mortality is achieved by the mixed assignment with large analysts and a small integrator.This assignment avoids the conservative all-large system and the low-discrimination all-small system, while giving the best F1, Sensitivity, and within-cell AUROC on mortality.Source: §5 · p. 6
  • The mixed model assignment shifts sensitivity through the integrator rather than through analyst placement or retrieval changes.The analysis varies model placement while keeping role instructions fixed, isolating assignment effects.Source: §5 · p. 6

Topology and exemplars

  • Retrieved patient exemplars improve single-agent performance but remain below multi-agent configurations unless split across contrastive analyst roles.On MIMIC-IV mortality, large single-agent F1 rises from 37.7 with single+RAG to 52.8 with single+RAG+Sim, but the small single agent reaches Sensitivity 94.7 at Specificity 23.7.Source: §5.1 · p. 6

Topology and retrieval

  • The multi-agent topology carries the gain, while medical-knowledge retrieval is close to neutral.With all-large models, removing MedRAG changes F1 from 54.4 for multi+RAG+Sim to 51.3 for multi+Sim.Source: §5.1 · p. 6

Risk separation

  • Tjur’s D measures separation between mean predicted risk for actual positives and negatives, while AUROC and bootstrap intervals compare pipeline conditions against null values.Figure 2 uses Tjur’s D and AUROC for CoT, single+RAG+Sim, and multi+RAG+Sim on MIMIC-IV mortality.Source: §5.2 · p. 7, Figure 2 · p. 7

6 Error Analysis

The integrator’s lower threshold flags more patients, improving mortality sensitivity, but the same condition-count heuristic is poorly matched to readmission because pre-discharge records contain weak signal.

Integrator behavior

  • 30.2 percentage points is the increase in predicted-positive rate when replacing gpt-oss with Qwen on MIMIC-IV mortality.The integrators use similar evidence but cross the positive threshold at different numbers of recorded conditions.Source: §6 · p. 7

Risk distributions

  • Figure 3 plots mean predicted risk against recorded-condition count among patients without the event, with standard-error bars.The figure supports comparison of how integrators map comorbidity counts to risk in mortality and readmission.Source: §6 · p. 7, Figure 3 · p. 8

Mortality errors

  • 56% of decedents missed by gpt-oss on MIMIC-IV mortality would be flagged by the Qwen integrator.Qwen crosses the condition-count threshold earlier, trading moderate-severity survivor false positives for fewer missed decedents.Source: §6 · pp. 7–8

Readmission signal

  • AUROC 0.77 for mortality versus 0.48 to 0.59 for readmission shows that available codes contain substantially weaker signal for the longer-horizon outcome.The comparison uses five-fold cross-validated bag-of-codes classifiers on identical input types with different labels.Source: §6 · p. 8

Readmission errors

  • The Qwen integrator continues raising predicted risk with condition count among non-readmitted patients, whereas gpt-oss is much flatter.The reported correlations are ρ = 0.39 for Qwen and ρ = 0.12 for gpt-oss on readmission.Source: §6 · p. 8

7 Conclusion

The paper presents a training-free, locally deployable open-weight MoA for clinical prediction and finds that role assignment, especially the integrator, is central to its high-recall behavior.

Contribution

  • The training-free pipeline uses large open-weight models as contrastive analysts and a small model as integrator.The design is evaluated for clinical prediction without task-specific fine-tuning.Source: §7 · p. 8

Main conclusion

  • On MIMIC-IV mortality, the role-specialized pipeline matches API-based Claude-3.5 few-shot prompting on F1 while achieving higher Sensitivity than closed-model prompting and supervised pre-trained baselines.The paper identifies the gain with the integrator’s decision threshold and model-family differences in mapping recorded conditions to risk.Source: §7 · pp. 8–9

Implication

  • Role placement is presented as a key factor for open-weight clinical prediction, particularly in high-recall screening settings that avoid closed APIs and task-specific fine-tuning.Fine-tuned models such as KARE remain strongest overall.Source: §7 · p. 9

8 Limitations

The evaluation is narrow in datasets, model families, role decompositions, retrieval settings, outcomes, and validation, and its advantage is limited to a risk-sensitive threshold regime rather than uniform performance gains.

  • The study uses MIMIC data without external or prospective validation and evaluates only Qwen and gpt-oss model families.Source: §8 · p. 9
  • Only MIMIC-IV mortality shows clear discrimination for the mixed assignment; readmission lacks a principled per-task configuration winner.The authors use one fixed configuration for readmission rather than selecting different configurations per task.Source: §8 · p. 9
  • The reported advantage is a decision-threshold difference yielding higher Sensitivity, while other baselines lead on F1, Accuracy, or AUROC.The authors state that the method matches neither the fine-tuned SOTA KARE on F1 or Accuracy nor the large model on AUROC.Source: §8 · p. 9
  • The design evaluates one analyst-integrator decomposition, one exemplar per outcome class, and no alternative Top k retrieval setting.Source: §8 · p. 9
  • Intermediate outputs received no clinician review, and outcomes beyond mortality and readmission, such as length of hospital stay, were not evaluated.Source: §8 · p. 9

Ethics Statement

The research uses credentialed public MIMIC datasets and locally run open-weight models, so patient data is not transmitted to external language-model services. AI assistance supported implementation and manuscript preparation, with author review and revision of generated content.

  • MIMIC-III v1.4 and MIMIC-IV v2.0 are publicly available under PhysioNet’s Credentialed Health Data Use Agreement.The agreement requires credentialed access and human-subjects research training.Source: Ethics Statement · p. 9
  • All pipeline models are open-weight and run locally, preventing patient data transmission to external or third-party language-model services.Source: Ethics Statement · p. 9
  • AI assistants helped generate LaTeX table code and refine author-written text, while the authors reviewed and revised all AI-generated content.Source: Ethics Statement · p. 9

A AUROC / AUPRC (score-based metrics)

Score-based metrics complement threshold metrics by measuring discrimination for methods that expose probabilities. Mortality retains a large-model discrimination advantage, while readmission is near chance across methods.

  • AUROC 0.763 for oss CoT, m4 is the mortality discrimination result highlighted for the large single model.Here, m4 denotes MIMIC-IV mortality.Source: A AUROC / AUPRC (score-based metrics) · p. 11
  • AUROC ≈0.50 for every method on readmission indicates near-chance discrimination.Source: A AUROC / AUPRC (score-based metrics) · p. 11
  • Table 5 reports AUROC and AUPRC percentages for methods exposing probabilities across MIMIC-III/IV mortality and readmission.The table labels these datasets and tasks as m3/m4 and rd3/rd4.Source: Table 5 · p. 11

B Discrimination-slope (mechanism) analysis

The analysis shows that retrieval changes probability separation, while the multi-agent integrator restores mortality discrimination without reducing the finding to a threshold shift. This mechanism is strong for MIMIC-IV mortality but weaker or absent for readmission and under-powered MIMIC-III mortality.

Metric and pipeline

  • Tjur’s D measures signed separation between mean predicted risks for outcome and non-outcome groups, alongside AUROC.Positive D indicates correctly separated groups, while D≈0 indicates no spread and D<0 indicates reversed separation.Source: B Discrimination-slope (mechanism)… · p. 11

Mechanism

  • Single+RAG+Sim reduces probability-level separation, whereas multi+RAG+Sim restores the discriminative signal.The pattern indicates distributional reshaping rather than merely moving the 0.5 decision threshold.Source: B Discrimination-slope (mechanism)… · p. 11

MIMIC-IV mortality

  • 0.763→0.638→0.730 AUROC across CoT, single+RAG+Sim, and multi+RAG+Sim shows the MIMIC-IV mortality effect is ranking discrimination.For oss, Tjur’s D changes 0.170→0.071→0.121; for qwen, it changes 0.137→0.103→0.161.Source: B Discrimination-slope (mechanism)… · p. 12

MIMIC-III mortality

  • MIMIC-III mortality follows the same direction, but rare positives produce noisy estimates and wide confidence intervals.The oss single condition reaches −0.035 without statistical significance, so this dataset corroborates direction rather than resolving mechanism.Source: B Discrimination-slope (mechanism)… · p. 12

Readmission

  • Readmission shows no reliable probability spread: oss-multi is mildly negative on both datasets, while qwen·m4 multi is only +0.033.Retrieval content was matched across topologies, supporting an integration effect for mortality; the readmission null is analyzed separately.Source: B Discrimination-slope (mechanism)… · p. 12

Role isolation

  • Replacing only the integrator with the small model lifts mortality F1 from 40→47 and 38→57 and sensitivity from 26→76 and 13→65.Replacing only an analyst or retrieval model does not produce this gain; the large-analyst, small-integrator default is best on mortality cells.Source: E Full factorial (all four cells) and… · p. 13

Related papers

Loading 2608.22176v1…