PaperlayerView exact revision on arXiv

Source-linked AI summary

MAP: Evaluation and Multi-Agent Enhancement of Large Language Models for Inpatient Pathways

Zhen Chen, Zhihao Peng, Xusheng Liang, Cheng Wang, Peigan Liang, Linsheng Zeng, Minjie Ju, Yixuan Yuan

arXiv:2503.13205v1cs.AIcs.CLcs.CVcs.HCcs.MA

TL;DR

The paper addresses limited evidence and benchmarking for LLMs in complex inpatient pathways. It introduces the IPDS benchmark and MAP multi-agent framework, and reports a 25.10% diagnostic-accuracy improvement over HuatuoGPT2-13B. The authors present MAP as a foundation for inpatient pathway systems while emphasizing that it augments rather than replaces medical professionals.

Abstract

from arXiv · show

Inpatient pathways demand complex clinical decision-making based on comprehensive patient information, posing critical challenges for clinicians. Despite advancements in large language models (LLMs) in medical applications, limited research focused on artificial intelligence (AI) inpatient pathways systems, due to the lack of large-scale inpatient datasets. Moreover, existing medical benchmarks typically concentrated on medical question-answering and examinations, ignoring the multifaceted nature of clinical decision-making in inpatient settings. To address these gaps, we first developed the Inpatient Pathway Decision Support (IPDS) benchmark from the MIMIC-IV database, encompassing 51,274 cases across nine triage departments and 17 major disease categories alongside 16 standardized treatment options. Then, we proposed the Multi-Agent Inpatient Pathways (MAP) framework to accomplish inpatient pathways with three clinical agents, including a triage agent managing the patient admission, a diagnosis agent serving as the primary decision maker at the department, and a treatment agent providing treatment plans. Additionally, our MAP framework includes a chief agent overseeing the inpatient pathways to guide and promote these three clinician agents. Extensive experiments showed our MAP improved the diagnosis accuracy by 25.10% compared to the state-of-the-art LLM HuatuoGPT2-13B. It is worth noting that our MAP demonstrated significant clinical compliance, outperforming three board-certified clinicians by 10%-12%, establishing a foundation for inpatient pathways systems.

2 Introduction

The paper addresses the limited evaluation of LLMs for complex inpatient pathways by introducing the IPDS benchmark and MAP multi-agent framework. MAP coordinates specialized clinical agents and supporting modules to improve inpatient decision support.

Research gap

  • Existing benchmarks focus mainly on medical examinations and question answering, whereas inpatient pathways require analysis of diverse records and nuanced decisions.The paper motivates a benchmark that better represents dynamic clinical environments.Source: §2 · p. 2, Table 1 · p. 5

Benchmark and motivation

  • The IPDS benchmark contains 51,274 patient cases across 9 clinical departments, 17 disease categories, and 16 standardized treatment options, representing diverse inpatient scenarios.It incorporates curated demographic information, radiological reports, and medical history for comprehensive inpatient pathway decision support.Source: §2 · p. 2

Framework

  • MAP coordinates triage, diagnosis, treatment, and chief agents to simulate the inpatient pathway.Its supporting modules include record review, trainable retrieval-enhanced generation, and expert guidance.Source: §2 · p. 2, Figure 2 · p. 4

Evaluation scope

  • Experiments evaluate triage, diagnosis, and treatment tasks, disease-specific diagnostic challenges, misdiagnosis conditions, and clinical compliance.The study also examines clinician judgments and makes IPDS and MAP publicly available for further research.Source: §2 · pp. 4–5

3 Benchmark

The IPDS benchmark is built from MIMIC-IV to evaluate inpatient triage, diagnosis, and treatment using broad departmental, disease, and treatment coverage. Its multifaceted design targets realistic clinical decision-support evaluation rather than examination-style testing.

Benchmark construction

  • 51,274 IPDS cases cover 9 departments, 17 major diseases, and 16 standardized treatment options from MIMIC-IV.The benchmark incorporates demographic information, radiological reports, and medical history.Source: §3 · p. 5, Figure 3 · p. 6

Evaluation tasks

  • IPDS evaluates three clinical classification tasks: nine-category triage, seventeen-category diagnosis, and sixteen-option treatment selection.Together, these tasks assess medical information analysis, consolidation, and diagnostic decision-making.Source: §3 · p. 5

Benchmark rationale

  • The benchmark is designed for inpatient diagnostic support, contrasting with existing benchmarks that often use online consultations, textbooks, examinations, or multiple-choice questions.The paper also describes de-identification and access controls for privacy and data integrity.Source: §3 · pp. 5–6

4 Results

The study evaluates general and specialized medical LLMs alongside MAP on inpatient triage, diagnosis, and treatment tasks. Privacy restrictions prevent comparison with external API models such as ChatGPT, GPT-4, and Med-PaLM.

Evaluation setup

  • The evaluation compares MAP with general LLMs and specialized medical LLMs across inpatient pathway tasks.The tested models include InternLM2, LLaMA-3, HuatuoGPT2, Clinical-Camel, and Meditron.Source: §4 · p. 6

Evaluation scope

  • MIMIC privacy and data agreements prevent evaluating external APIs such as ChatGPT, GPT-4, and Med-PaLM.This restriction defines the comparison set used in the study.Source: §4 · p. 6

The under-performance of state-of-the-art LLMs in supporting inpatient pathways.

State-of-the-art LLMs showed limited performance on inpatient triage, diagnosis, and treatment tasks, especially complex disease cases, whereas MAP substantially improved diagnostic support.

The under-performance of state-of-the-art LLMs in supporting inpatient pathways.

  • State-of-the-art LLMs achieved limited triage, diagnosis, and treatment accuracies across inpatient pathway tasks.LLaMA-3-8B reached 60.30%, 49.30%, and 64.80% in triage, diagnosis, and treatment, while HuatuoGPT2-13B reached 58.40%, 53.00%, and 67.20%.Source: The under-performance of… · p. 6

The MAP enhances the capability of inpatient diagnostic support, particularly in complex disease cases.

  • In D5 mental and behavioral disorders, MAP achieved 79.86% accuracy versus HuatuoGPT2-13B's 38.46%.The reported enhancement was attributed to record review, trainable REG, and expert guidance modules, with ablation studies verifying their effects.Source: Figure 4 · p. 7, The MAP enhances the capability of… · p. 7

The under-performance of state-of-the-art LLMs in supporting inpatient pathways.

  • State-of-the-art LLM accuracy fell below 42.31% in challenging D5 and D9 disease categories involving complex clinical presentations.Source: The under-performance of… · p. 7

The MAP enhances the capability of inpatient diagnostic support, particularly in complex disease cases.

  • 78.10% diagnosis accuracy was achieved by MAP, a 25.10% improvement over HuatuoGPT2-13B's 53.00%.Source: The MAP enhances the capability of… · p. 7
  • MAP achieved accuracy 10%-12% higher than board-certified clinicians and showed stronger agreement with ground truth than individual clinicians.The reported ground-truth agreement was ICC = 0.81 for MAP versus ICC ∈[0.67, 0.68] for clinicians.Source: Figure 5 · p. 8

Clinical validation of MAP with significant clinical compliance.

Clinical evaluation compared MAP with three clinicians and identified disease-specific diagnostic difficulty across inpatient categories.

  • MAP achieved the best statistical performance among general LLMs, medical LLMs, and clinicians in the clinical evaluation.MAP's ground-truth agreement was ICC 0.81, compared with clinician ICC values of 0.80, 0.67, and 0.68.Source: Clinical validation of MAP with… · p. 8
  • MAP maintained strong alignment with clinician assessments, with ICC values of 0.84, 0.73, and 0.75.Source: Clinical validation of MAP with… · p. 8
  • Misdiagnosis rates across the top 10 disease categories ranged from 35.45% to 14.09%, indicating disease-specific diagnostic complexity.D17, D1, and D8 had the highest reported rates.Source: Figure 6 · p. 9

5 Discussion

The discussion identifies substantial variation in diagnostic difficulty across disease categories and reports more balanced diagnostic performance for MAP than for the best-performing clinician.

5 Discussion

  • The discussion frames MAP evaluation around both the potential and limitations of LLM-assisted inpatient pathways.It also examines disease-category challenges, clinical data quality, and areas for future development.Source: §5 · p. 9

Different diagnostic challenges across disease categories in inpatient settings.

The MAP showed consistently superior diagnosis across all disease categories.

Impact of input data on inpatient pathways.

Input ablations show that clinical data components materially affect inpatient diagnostic support, with medical history and radiological reports producing the largest gains. The study also identifies outpatient data, explainability, and continued clinician oversight as future priorities.

Impact of input data on inpatient pathways.

  • 55.02%: Including medical history increased MAP diagnostic accuracy for D5 mental and behavioral disorders from 23.08% to 78.10%.The authors attribute this gain to historical context for interpreting current symptoms.Source: Impact of input data on inpatient… · p. 10
  • 31.03%: Including radiological reports increased diagnostic accuracy for D13 diseases of the digestive system from 47.50% to 78.53%.The result emphasizes imaging data as an important input for LLM-assisted inpatient pathways.Source: Impact of input data on inpatient… · p. 10, Figure 8 · p. 11
  • 5.80%: Adding demographic information improved overall diagnostic accuracy from 46.60% to 52.40%, a smaller effect than medical history or radiological reports.The authors characterize demographic information as supportive rather than dominant.Source: Impact of input data on inpatient… · p. 10, Figure 8 · p. 11

Limitations and future directions.

  • Future work includes adding outpatient data, improving MAP explainability, and maintaining collaboration between LLMs and human clinical judgment.The framework is designed to augment rather than replace medical professionals.Source: Limitations and future directions · p. 11

8 Benchmark preparation

The IPDS benchmark integrates de-identified MIMIC-IV hospital, ICU, and note data into inpatient cases with curated demographic, diagnostic, radiological, and treatment information. Its construction combines multi-table integration, filtering, disease reclassification, and clinically prioritized labels.

8 Benchmark preparation

  • IPDS integrates MIMIC-IV-Hosp, MIMIC-IV-ICU, and MIMIC-IV-Note data while filtering missing or unrelated records and selecting relevant diagnostic features.The source modules provide hospitalization, admission, demographic, diagnosis, medical-history, and radiological information.Source: §8 · p. 16
  • 51,274 cases: Each IPDS case combines curated demographic information, radiological reports, and medical history for inpatient pathway evaluation.Cases are constructed through unique-subject multi-table searches and integration operations.Source: §8 · p. 16
  • IPDS defines triage using nine admission departments, diagnosis using 17 broader disease categories, and treatment using 16 service-based options.Diagnosis labels were reclassified from 1,298 original diseases, with three highest-priority diagnostic indicators retained.Source: §8 · p. 16
  • The benchmark applies de-identification, text filtering, regular expressions, and manual review to remove personally identifiable information before controlled PhysioNet access.The dataset is distributed as CSV files, and access requires identity verification, training, and a data use agreement.Source: §8 · pp. 16–17

9 Methods

MAP implements the triage-diagnosis-treatment pathway with three specialized clinician agents and a supervising chief agent. Retrieval, record review, structured communication, and expert guidance are used to improve the relevance, interpretability, and clinical grounding of reasoning.

9 Methods

  • Structured communication records context, agent thinking, and standardized answers with medical terminology and ICD-10 codes.This protocol is intended to make multi-agent interaction traceable and interpretable in clinical deployment.Source: §9 · p. 18
  • MAP assigns triage, diagnosis, and treatment to specialized agents, while a chief agent oversees their decisions and provides guidance.The triage agent assigns departments, the diagnosis agent analyzes clinical inputs, and the treatment agent determines plans.Source: §9 · p. 18
  • The record review module filters medical-history information that conflicts with radiological reports using ClinicalBERT correlations, cosine similarity, and a 0.1 threshold.Low-correlation history is treated as irrelevant to the radiological report and excluded from diagnostic reasoning.Source: §9 · p. 18
  • Retrieval-enhanced generation retrieves the top 10 relevant documents from patient cases and NICE guidelines, then adds structured diagnostic reasoning to the input.LlamaIndex vector search supplies similar cases, guidelines, reasoning processes, and supporting evidence.Source: §9 · pp. 18–19
  • The expert guidance module checks diagnoses against medical standards and guidelines, identifies reasoning inconsistencies, and issues corrective guidance.The chief agent can flag overlooked findings, such as a relevant lung nodule in a radiological report.Source: §9 · p. 19
  • MAP uses LLaMA-3-8B for four agents, freezes LLM parameters during training, and fine-tunes retrieval-related parameters for three epochs.During inference, clinician agents use record review and retrieval-enhanced generation, while expert guidance is used during training.Source: §9 · p. 19

10 Evaluation metrics

The evaluation combines classification, reliability, and disease-specific clinical metrics with ablation and qualitative analyses. These analyses examine both overall module effects and whether MAP integrates diverse clinical evidence in a clinically coherent way.

10 Evaluation metrics

  • The evaluation reports accuracy, macro precision, macro F1, sensitivity, specificity, Cohen’s Kappa, MCC, and AUC alongside disease-specific misdiagnosis rate.DSMR measures misdiagnosed cases relative to total actual cases for a specific disease.Source: §10 · p. 19
  • ICC(2, k) assesses rating reliability among multiple raters, with thresholds ranging from poor reliability below 0.40 to excellent reliability at least 0.75.The reported formula uses mean-square terms for rows, error, and columns, plus subject and rater counts.Source: §10 · p. 20

11 Ablation study and qualitative case study

  • MAP showed consistent improvements over ablative baselines across most scenarios and eight evaluation metrics in the ablation studies.The comparisons span the inpatient pathway tasks and are presented across Figures 9–12.Source: §11 · p. 20
  • MAP more appropriately weighted radiographic findings than baseline LLaMA-3-8B in a respiratory case involving unrelated medical history and alveolar hemorrhage.The qualitative example links structured reasoning with more accurate diagnoses for respiratory diseases.Source: §11 · p. 20

12 Data availability

The dataset is accessible through PhysioNet after credentialed-user requirements and completion of the specified training and agreement steps. The evaluation and analysis code are publicly available through GitHub and Google Drive, with specified software environments.

12 Data availability

  • MIMIC-IV access requires PhysioNet credentialing, CITI training, and signing the project’s data use agreement.The generated dataset can also be downloaded directly from PhysioNet.Source: §12 · p. 20

13 Code availability

  • The evaluation framework is available on GitHub, while analysis, plotting, and statistical-analysis code are available through Google Drive.Source: §13 · pp. 20–21
  • The released code specifies Python 3.10 and versions for PyTorch, Transformers, spaCy, LangChain, Llama-index-core, vLLM, NLTK, OpenAI, and Pandas.Dataset-creation code uses Python 3.10 and Pandas 2.1.3.Source: §13 · p. 21
Loading 2503.13205v1…