Source-linked AI summary
Standalone LLM and a Pre-specified Agentic Pipeline for Explaining ICU Mortality Predictions: a Feasibility Study on the eICU Demo Dataset
Di Zhu, Chen Xie, Haoyun Zhang, Zihan Wei, Ziwei Wang, Jiazhao Shi, Ziyu Wang, Qiyang Xie
TL;DR
Clinicians need understandable, auditable narratives for ICU mortality predictions beyond feature attributions alone. This feasibility study compares standalone and four-step agentic LLM explanations, finding that decomposition improved guideline grounding and patient-specific detail but reduced SHAP alignment and direction consistency.
Problem
Feature attributions identify influential variables but do not by themselves provide the clinical narratives needed to understand and audit ICU mortality predictions.
Method
The study compares a standalone LLM with a pre-specified agentic pipeline that decomposes data interpretation, formal-criteria application, differential construction, and explanation synthesis.
Results
Agentic explanations had higher guideline grounding (0.762 versus 0.143), value specificity (0.236 versus 0.143), and plausibility (0.700 versus 0.671), but lower SHAP alignment and direction consistency.
Takeaways & Limitations
Clinically useful risk explanations may need both attribution checks for model fidelity and guideline-grounded language for readable interpretation.
Takeaways & Limitations
The small eICU Demo dataset and only 14 overlapping SHAP-reviewed cases leave comparative explanation metrics with wide confidence intervals.
Abstract
from arXiv · showhide
Machine-learning models can predict ICU mortality accurately, but feature-attribution methods alone rarely provide the clinical narrative needed for bedside use. Large language models (LLMs) may bridge this gap, and multi-step agentic pipelines are a plausible extension because they separate data interpretation, guideline checking, and final explanation. This revised feasibility study preserves the original standalone-versus-agentic comparison while making the main clinical findings more explicit. Using the retained local eICU Demo artifact set (2,353 ICU stays; 8.1\% mortality), XGBoost achieved an AUROC of 0.855 (95\% CI 0.796--0.906) and an AUPRC of 0.332 (95\% CI 0.217--0.494). On a stratified 38-case explanation subset, the standalone LLM produced 1 explanation with explicit outcome leakage, whereas the four-step agentic pipeline produced none. Among the 14 cases that overlapped with the SHAP review subset, the standalone LLM showed higher SHAP alignment (mean Jaccard 0.171 versus 0.077) and higher direction consistency (92.9\% versus 78.6\%), while the agentic pipeline showed higher guideline grounding (0.762 versus 0.143), higher value specificity (0.236 versus 0.143), and slightly higher plausibility (0.700 versus 0.671). Clinically, the results suggest that agentic decomposition may improve safety-relevant grounding and patient-specific detail, but it should be paired with attribution-based checks before use in high-stakes risk explanation.
1 Introduction
ICU mortality models can outperform classical scores, but clinicians still need explanations they can understand and audit. This study frames LLMs and decomposed agentic pipelines as approaches for addressing the gap between feature attribution and clinical narrative.
- Motivation: ICU mortality prediction is clinically important, yet predictive performance alone does not explain why a particular patient receives high risk.Modern machine-learning models often exceed classical severity scores on structured electronic health record data, while interpretability remains a practical barrier.
- Explainability gap: SHAP identifies features that move predictions, but a ranked attribution list does not provide the pathophysiological narrative needed for clinical verification, triage, and communication.This distinction motivates using LLMs to transform structured values into clinically interpretable explanations.
- Agentic approach: Agentic systems decompose explanation into data interpretation, formal-criteria application, differential construction, and synthesis.The decomposition is intended to make multi-step reasoning more explicit in critical care, where thresholds, syndromic criteria, and interacting organ systems matter.
- Research questions: The study asks whether standard models accurately predict ICU mortality, whether standalone LLM explanations retain SHAP alignment and clinical plausibility, and whether structured agentic pipelines improve explanation quality and grounding.These questions preserve the original standalone-versus-agentic comparison while linking prediction performance to explanation evaluation.
- Study scope: The revised manuscript limits claims to the auditable local artifact set, evaluating prediction on the held-out cohort and explanation questions on an audited subset from the same versioned test snapshot.This design keeps the underlying questions and idea while restricting evidence to versioned, locally retained artifacts.
2 Results
In 2,353 ICU stays, XGBoost showed strong discrimination, while SHAP identified clinically coherent mortality correlates. Across 14 overlapping audited cases, the agentic pipeline eliminated explicit outcome leakage and improved guideline grounding, value specificity, and plausibility, but had lower SHAP alignment and direction consistency than the standalone LLM.
- Cohort and model performance: 2,353 adult ICU stays had 8.1% in-hospital mortality, with non-survivors showing higher age, heart rate, respiratory rate, lactate, and blood urea nitrogen.Non-survivors were older and had higher values for these physiologic and laboratory measures than survivors.
- Cohort and model performance: 0.855 AUROC was achieved by XGBoost, versus 0.823 for logistic regression; AUPRC was 0.332 versus 0.345, respectively.XGBoost outperformed logistic regression on AUROC, whereas logistic regression had a slightly higher AUPRC under low event prevalence.
- Feature attribution: Age, minimum SpO2, blood urea nitrogen, lactate, and respiratory rate were the most influential features in the retained XGBoost model.These features matched established ICU mortality correlates involving advanced age, hypoxemia, renal dysfunction, and respiratory compromise.
- Standalone explanation quality: 1 of 38 standalone explanations contained explicit outcome leakage and was removed, leaving 37 valid explanations, including 14 overlapping SHAP-reviewed cases.Among the 14 overlapping cases, standalone explanations had mean SHAP alignment of 0.171, direction consistency of 92.9%, plausibility of 0.671, value specificity of 0.143, and guideline grounding of 0.143.
- Agentic-versus-standalone comparison: 0 cases showed explicit outcome leakage in the agentic run, while its 14 overlapping cases had lower SHAP alignment than standalone explanations, 0.077 versus 0.171, and lower direction consistency, 78.6% versus 92.9%.The four-step pipeline comprised a clinical data interpreter, guideline consultant, differential reasoner, and final synthesizer, with prompts rewritten to consume cleaned values without outcome leakage.
- Agentic-versus-standalone comparison: 0.762 guideline grounding, 0.236 value specificity, 85.7% patient-specific data mention, and 0.700 plausibility were achieved by the agentic pipeline, versus 0.143, 0.143, 64.3%, and 0.671 for standalone explanations.These comparative results indicate stronger grounding, specificity, patient-specific detail, and slightly higher plausibility for the agentic pipeline.
3 Discussion
The discussion frames agentic decomposition as a measurable tradeoff: it improved leakage control, guideline use, and patient-specific explanation, but did not automatically improve SHAP alignment. Explanation reliability also depends on substantial clinical-data missingness and limited comparative evidence.
- Clinical findings: The mortality model was sufficiently discriminative for explanation work, with age, oxygenation, blood urea nitrogen, lactate, and respiratory rate as influential features.These features are recognizable markers of physiologic instability in critical care.
- Clinical findings: The standalone LLM usually produced concise, directionally correct risk narratives, but showed weak patient-specific values and only modest SHAP alignment.This establishes a contrast between readable narrative generation and attribution fidelity.
- Clinical findings: The agentic pipeline reduced explicit leakage and improved guideline grounding and patient-specific explanation, but its greater structure did not automatically improve SHAP alignment.The discussion therefore presents decomposition as a practical tradeoff rather than categorical superiority.
- Data quality: Missingness was substantial for mean arterial pressure (84.7%), temperature (93.4%), lactate (81.5%), and reconstructed GCS (13.2% unavailable), directly limiting explanation reliability.The passage identifies these variables as clinically important and links their missingness to explanation reliability.
- Limitations: Only 14 generated cases overlapped with the SHAP-reviewed subset, confidence intervals remained wide, and both generators used a single local base model.The comparative findings therefore reflect particular implementations rather than architecture-independent truth.
4 Methods
The study used a retained adult ICU cohort from the eICU Demo database, with first-24-hour clinical features and retained predictive and SHAP artifacts. It compared a single outcome-free LLM prompt with a four-step agentic pipeline using structured explanation-quality audits.
- Cohort and outcome: 2,353 ICU stays met the retained criteria: adults aged at least 18 years, ICU length of stay of at least 4 hours, and non-missing hospital discharge status.The primary outcome was in-hospital mortality.
- Data and features: First-24-hour features covered demographics, vital signs, laboratory results, and APACHE-related neurological variables, with added cleaning rules for explanation prompts.GCS reconstruction used valid APACHE components, and temperatures above 45 were converted from Fahrenheit to Celsius.
- Predictive modeling: The retained models were L2-regularized logistic regression and XGBoost, with bootstrap 95% confidence intervals estimated on the held-out test set by resampling 471 encounters.The confidence intervals covered AUROC and AUPRC.
- Attribution reference: The retained SHAP artifacts provided global feature importance and per-patient references, defining each reviewed patient’s SHAP top features as the three largest absolute attributions.These artifacts were reused from the original study.
- Explanation generation: The standalone baseline used one outcome-free prompt, whereas the agentic pipeline used four serial steps: data interpretation, guideline application, differential reasoning, and synthesis.Both systems received the same cleaned patient representation, and explanations were generated for a stratified subset of 38 held-out cases.
- Explanation evaluation: Explanations were audited for outcome leakage and evaluated on SHAP alignment, plausibility, direction consistency, value specificity, guideline grounding, and reasoning depth.Explanations containing explicit outcome or survival language were flagged and excluded from valid explanation-quality scoring.