PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Capabilities in Perioperative Risk Prediction and Prognostication

Philip Chung, Christine T Fong, Andrew M Walters, Nima Aghaeepour, Meliha Yetisgen, Vikas N O'Reilly-Shah

arXiv:2401.01620v1cs.AIcs.CL

TL;DR

The paper evaluates whether general-domain LLMs can use procedure descriptions and clinical notes to predict perioperative risk and postoperative outcomes. Across tasks, classification performance is promising and prompting helps on some tasks, while duration predictions remain poor.

  • Problem

    The paper asks whether general-domain LLMs can predict perioperative risk and postoperative outcomes from patient and procedure information, including in real-world EHR contexts with uncertain or incomplete information.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates general-domain LLMs on eight EHR-grounded perioperative tasks using procedure descriptions, clinical notes, and varied prompting strategies.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Classification performance is strongest for ASA-PS, ICU admission, and hospital mortality, prompting improves some tasks, and duration prediction remains poor.

    Source: §4 · pp. 8–9, §5 · p. 10

  • Takeaways & Limitations

    Current general-domain LLMs may assist perioperative risk stratification, although dedicated tabular models outperform them when standalone automated prediction is the goal.

    Source: §4 · p. 10, §5 · p. 10

    The evaluation uses outcome-balanced datasets, whereas the corresponding outcomes are rare in the real-world cases from which they were derived.

    Source: §4 · p. 10

Abstract

from arXiv · show

We investigate whether general-domain large language models such as GPT-4 Turbo can perform risk stratification and predict post-operative outcome measures using a description of the procedure and a patient's clinical notes derived from the electronic health record. We examine predictive performance on 8 different tasks: prediction of ASA Physical Status Classification, hospital admission, ICU admission, unplanned admission, hospital mortality, PACU Phase 1 duration, hospital duration, and ICU duration. Few-shot and chain-of-thought prompting improves predictive performance for several of the tasks. We achieve F1 scores of 0.50 for ASA Physical Status Classification, 0.81 for ICU admission, and 0.86 for hospital mortality. Performance on duration prediction tasks were universally poor across all prompt strategies. Current generation large language models can assist clinicians in perioperative risk stratification on classification tasks and produce high-quality natural language summaries and explanations.

1 Introduction

The introduction asks whether general-domain LLMs can use patient and procedure information to stratify perioperative risk and predict postoperative outcomes. It benchmarks eight EHR-grounded tasks and examines prompting strategies in real-world clinical records, where context may be uncertain, incomplete, or erroneous.

1 Introduction

  • Prompting can specify tasks and domains without domain-specific fine-tuning, while prior medical demonstrations include USMLE questions and EHR summarization.Instruction-tuned LLMs have also been used for knowledge retrieval, text extraction, summarization, and reasoning.Source: §1 · p. 1
  • General-domain LLMs are tested for preoperative risk stratification using patient information and a planned surgery or procedure.The motivation is to inform proceduralists, surgeons, and anesthesiologists weighing whether to proceed, cancel, or delay for medical optimization.Source: §1 · pp. 1–2
  • The study benchmarks eight postoperative tasks because no single outcome measure captures perioperative risk.Tasks include ASA-PS assignment, admission and duration predictions, unanticipated admission, and in-hospital mortality.Source: §1 · p. 2
  • The study also examines few-shot and chain-of-thought prompting as ways to guide task responses without domain-specific fine-tuning.Few-shot prompting adds representative task-and-solution examples; chain-of-thought prompts step-by-step reasoning before the final answer.Source: §1 · p. 2
  • Unlike well-defined medical Q&A, real-world EHR contexts can be uncertain, incomplete, or erroneous, and may lack a clear answer.The authors derive their dataset from this clinical context to benchmark perioperative risk prediction and prognostication.Source: §1 · p. 2

2 Methods

The methods use retrospective EHR records from three Seattle hospitals to assemble outcome-specific datasets, then evaluate GPT-4 Turbo across eight tasks and six prompting strategies. Responses are compared with EHR-derived labels using F1 for categorical or binary outcomes and MAE for duration outcomes.

2 Methods

  • The study is retrospective and uses routinely collected health records under UW IRB approval with a waiver of consent.The computational environment for PHI/PII was reviewed and approved by UW Medicine IT, and reporting followed TRIPOD.Source: §2 · p. 2

2.1 Study Cohort and Dataset Definition

  • Cases came from three Seattle hospitals between April 1, 2021 and May 5, 2023, requiring an anesthesia preoperative evaluation and another preoperative clinician note.Up to the last 10 clinician notes before surgery were used, excluding the anesthesia preoperative evaluation; notes under 100 tokens were excluded.Source: §2.1 · p. 2
  • Seven outcome-specific datasets supported eight tasks; each dataset targeted 1,250 cases, split into 1,000 inference and 250 few-shot cases.For multiple cases per patient, one case was randomly selected; sampling and balancing procedures addressed outcome imbalance and skew.Source: §2.1 · pp. 2–3

2.2 Experimental Approach

  • GPT-4 Turbo was tested on eight tasks using six strategies spanning zero-shot, few-shot, and chain-of-thought prompting with original notes or summaries.Few-shot prompting used 5, 10, 20, or 50 examples drawn from the held-out few-shot dataset; GPT-4 Turbo generated note summaries and example rationales.Source: §2.2 · pp. 3–4
  • The experimental workflow compares LLM answers with EHR-derived ground-truth labels and computes F1 or MAE according to the outcome type.Prompts vary by task, strategy, and case; zero-shot prompting uses original notes or summaries, while few-shot prompting supplies in-context examples.Source: Figure 1 · p. 3
  • Categorical and binary tasks were evaluated by F1, while duration tasks used MAE; reported metrics included 95% bootstrap confidence intervals.The study also stratified cases by note-token length and used Wilcoxon signed-rank tests for pairwise prompt-strategy comparisons.Source: §2.2 · p. 4

3 Results

Across the classification tasks, all prompt strategies beat random-guess baselines, with prompting effects varying by task. Duration predictions were generally poor; note summaries enabled more few-shot examples, while note length affected tasks differently.

3.1 Datasets

  • The task datasets were built from a large perioperative EHR cohort, but hospital mortality and unplanned-admission datasets contained only 720 and 1,186 cases, respectively, before splitting.The other datasets reached 1,250 cases each; the cohort included 90,844 patients and 137,535 procedures.Source: §3.1 · p. 4

3.2 Effect of Prompt Strategy on Perioperative Risk Prediction Tasks

  • GPT-4 Turbo answered all prompts except one unplanned-admission case, where it refused after five retries because the sole note lacked clinically relevant content.That case was excluded from evaluation metrics.Source: §3.2 · pp. 4–5
  • All prompt strategies beat random baselines on classification; CoT significantly improved ASA-PS and hospital admission, while few-shot gains were consistent for ASA-PS, hospital admission, and mortality.CoT and few-shot gains were synergistic for ASA-PS and hospital admission; CoT slightly worsened hospital mortality, and zero-shot without CoT performed best for unplanned admission.Source: §3.2 · p. 6

3.5 Numerical Prediction Tasks

  • For PACU duration, prompt strategies performed worse than the mean-predicting baseline; duration predictions often showed quantized outputs and a ceiling effect without few-shot or CoT prompting.Few-shot and CoT prompting reduced these output effects but worsened performance; similar patterns were reported for hospital and ICU duration.Source: §3.2 · p. 6, §3.5 · pp. 7–8
  • Most duration-prediction strategies underperformed the mean-predicting baseline; hospital duration reached 4.5 days MAE versus 5.4 days at baseline, a difference described as not clinically significant.Few-shot worsened hospital-duration prediction; few-shot and CoT brought ICU-duration prediction only to baseline parity.Source: §3.2 · pp. 6–7

3.3 Effect of Summary Representation of Notes

  • Summarizing notes slightly reduced performance on some tasks and improved it on others, while enabling up to 50 in-context examples that could fit the model input.The reported summary-related changes were small; summaries sometimes lost useful information but could focus relevant information.Source: §3.3 · p. 7

3.4 Effect of Note Length on Perioperative Risk Prediction Tasks

  • Longer input notes improved ASA-PS and hospital-mortality prediction but worsened ICU-admission, PACU-duration, and hospital-duration prediction.The authors attribute increased input length to longer notes or more notes being written about a patient.Source: §3.4 · p. 7

4 Discussion

The models performed well on ICU admission and hospital mortality, with prompting gains on some severity-related tasks but persistently poor duration prediction. Their explanatory outputs may assist clinical assessment, although performance depends on the task and prospective validation remains necessary.

4 Discussion

  • ICU admission sensitivity was ≥0.78 and specificity was ≥0.74 across prompt strategies; hospital mortality sensitivity was ≥0.93, with specificity of 0.81 under 20-shot prompting.The multiclass ASA-PS F1 micro score was 0.5; few-shot and chain-of-thought errors were almost always to an adjacent class.Source: §4 · p. 9
  • Few-shot and chain-of-thought prompting improved ASA-PS, hospital admission, and hospital mortality predictions, but gains did not extend to every outcome.The authors suggest severity reflected in preoperative notes may explain gains, whereas intraoperative factors not represented in the inputs may limit unplanned-admission prediction.Source: §4 · p. 9
  • PACU, hospital, and ICU duration predictions remained poor despite prompting, which reduced quantization but did not resolve the models’ continuous-value prediction weakness.The authors attribute this limitation in part to discrete tokenized outputs that do not robustly support interpolation between numerical values.Source: Figure 4 · p. 8, §4 · pp. 9–10
  • Longer input contexts did not necessarily improve performance, while generated summaries were qualitatively high-quality and the model sometimes refused irrelevant tasks.The specific effects of note length and summaries varied by task.Source: §4 · p. 10
  • LLMs underperformed dedicated tabular classification models, but their natural-language explanations may help clinicians review patient histories and assess perioperative risk.The authors describe these explanations as starting points for more comprehensive risk assessment, rather than standalone predictions.Source: §4 · p. 10
  • Prospective clinical validation is needed, particularly for rare outcomes, and comparison with existing perioperative prediction algorithms remains necessary.The source cases included postoperative ICU admission in 0.49%, unanticipated hospital admission in 0.43%, and in-hospital mortality in 0.3%.Source: §4 · p. 10

5 Conclusion

General-domain text-only LLMs can support perioperative risk prediction when outcomes are framed as classification or binary tasks, but continuous duration prediction remains weak. Few-shot prompting and chain-of-thought reasoning improve prediction performance, and hospitals should evaluate these models as risk-stratification tools.

5 Conclusion

  • General-domain text-only LLMs can predict perioperative risk in classification or binary tasks, but remain poor at predicting continuous PACU, hospital, and ICU durations.Few-shot prompting and chain-of-thought reasoning improve prediction performance; the authors recommend hospitals evaluate LLMs as tools for perioperative risk stratification.Source: §5 · p. 10

Funding.

The paper reports its funding, author contributions, competing interests, and code availability.

Funding.

  • The project received computational support from the University of Washington eScience Institute and Microsoft Azure, with financial support from the University of Washington Bonica Scholars Program, Stanford ReAP, and NIH.Funders were not involved in study design, data collection or analysis, interpretation, or manuscript writing.Source: Funding · p. 11

Contributions.

  • The authors describe shared contributions spanning research design, data preparation, experimentation, analysis, manuscript writing, and review.P.C. conducted experiments, analyzed data, and wrote the manuscript.Source: Contributions · p. 11

Competing Interests.

Code Availability.

Tables

Table 1 describes patient demographics, procedure specialty, patient class, note statistics, and outcomes across seven task datasets. The supplied PACU-duration table fragment reports 1,000 inference cases and 250 few-shot cases.

Tables

  • Table 1 covers seven datasets: ASA-PS, PACU Phase 1 duration, hospital admission and duration, ICU admission and duration, unplanned admission, and hospital mortality.The caption identifies the datasets as panels A–G.Source: Table 1 · p. 12
  • In the PACU Phase 1 Duration Dataset, the table reports case counts of 1,000 under Inference and 250 under Fewshot.Source: Table 1: Dataset Characteristics · p. 14
  • Female patients account for 494 (49.4%) and 141 (56.4%) of the two reported groups, while male patients account for 506 (50.6%) and 109 (43.6%).The supplied fragment also lists the anesthesia categories General, MAC, and Regional.Source: Table 1: Dataset Characteristics · p. 14

C. Hospital Admission Dataset Inference Fewshot

The hospital-admission dataset fragment reports 1,000 inference cases and 250 few-shot cases, alongside actual and expected patient-class and proceduralist-service counts.

C. Hospital Admission Dataset Inference Fewshot

  • The cited fragment reports case counts of 1,000 and 250, labeled “no. (%)”; it does not specify the case types, task, or dataset.Source: C. Hospital Admission Dataset… · p. 16
  • General anesthesia predominated in both reported groups, accounting for 826 (82.6%) and 210 (84.0%); MAC accounted for 106 (10.6%) and 24 (9.6%), while regional anesthesia accounted for 68 (6.8%) and 16 (6.4%).Source: C. Hospital Admission Dataset… · p. 16

D. Hospital Duration Dataset

The hospital-duration dataset fragment reports 1,000 inference cases and 250 few-shot cases, with patient-class and proceduralist-service categories.

D. Hospital Duration Dataset

  • In the cited fragment, the reported case-count figures are 1,000 and 250, appearing together under the label “Case Counts, no. (%)” in the displayed table material.Source: D. Hospital Duration Dataset · p. 18
  • General anesthesia predominated in both groups (901 [90.1%] and 229 [91.6%]); MAC and regional anesthesia were less common, at 52 (5.2%) and 47 (4.7%) versus 11 (4.4%) and 10 (4.0%), respectively.Source: D. Hospital Duration Dataset · p. 18
  • Expected patient class was more often inpatient than outpatient in both reported columns: inpatient 491 (49.1%) and 129 (51.6%), versus outpatient 44 (4.4%) and 9 (3.6%).Source: D. Hospital Duration Dataset · p. 18

E. ICU Admission & ICU Duration Dataset Inference Fewshot

The ICU admission and duration dataset fragment reports 1,000 inference cases and 250 few-shot cases, alongside patient-class and proceduralist-service counts.

E. ICU Admission & ICU Duration Dataset Inference Fewshot

  • The cited passage reports case counts of 1,000 and 250, but does not identify the associated tasks or specify what distinguishes the two counts.Source: E. ICU Admission & ICU Duration… · p. 20
  • General anesthesia predominated in both reported groups, accounting for 862 (86.2%) and 220 (88.0%); MAC accounted for 109 (10.9%) and 26 (10.4%), while regional anesthesia accounted for 29 (2.9%) and 4 (1.6%).Source: E. ICU Admission & ICU Duration… · p. 20
  • Expected patient class was most often inpatient—622 (62.2%) and 160 (64.0%); outpatient accounted for 239 (23.9%) and 63 (25.2%), while observation accounted for 32 (3.2%) and 7 (2.8%).Source: E. ICU Admission & ICU Duration… · p. 20

F. Unplanned Admit Dataset

The continued dataset table lists case counts and patient, service, and anesthesia characteristics, but the supplied entries do not identify the two numeric columns.

F. Unplanned Admit Dataset

G. Hospital Mortality Dataset Inference Fewshot

This section presents hospital-mortality evaluation metrics and supplemental zero-shot and few-shot prompt formats, including experiments with 5, 10, 20, or 50 in-context examples. It also gives an example of ASA II classification across three prompting setups and the explanations provided.

G. Hospital Mortality Dataset Inference Fewshot

  • Hospital mortality is reported using F1, MCC, and confusion-matrix metrics, with MCC and F1 tables covering note-length strata and a dummy-model baseline.The supplied passages provide table scopes but no metric values.Source: Table A18 · p. 33, Table A19, Table A20

Supplemental Figure 1: Representative Text Prompts

B.1.1 System Message

  • The system message frames the model as a physician assessing patients’ outcomes after procedures.Source: §B.1.1 · p. 36

B.1.2 Zero-shot Q&A From Original Notes

  • The zero-shot original-notes prompt provides procedure and medical-record context and requests a JSON answer with an explanation; its example assigns ASA II.Source: §B.1.2 · pp. 36, 38

B.1.3 Zero-shot Q&A From Notes Summary

  • The zero-shot notes-summary prompt uses a medical-record summary as context and its example also assigns ASA II.Source: §B.1.3 · pp. 38–39

B.1.4 Few-shot Q&A From Notes Summary

  • The few-shot notes-summary prompt supplies example task-context-answer pairs, including a hysterectomy procedure and its associated clinical context.Source: §B.1.4 · p. 39
  • Few-shot prompting on notes summaries returned ASA II, describing the patient’s atrial fibrillation as mild systemic disease.The explanation noted no recurrent symptoms, preserved left ventricular function, and stability on anticoagulation.Source: §B.1.4 · p. 42

B.1.5 Zero-shot Chain-of-Thought Q&A From Original Notes

  • ASA II was returned with zero-shot chain-of-thought prompting on the original notes, citing stable condition and normal left ventricular function despite atrial fibrillation.The explanation also noted that anticoagulation alone does not automatically increase ASA status.Source: §B.1.5 · p. 44

B.1.6 Zero-shot Chain-of-Thought Q&A From Notes Summary

  • ASA II was returned with zero-shot chain-of-thought prompting on the notes summary, citing atrial fibrillation as mild systemic disease and preserved cardiac function.The response distinguished this from ASA I, III, IV, and V.Source: §B.1.6 · p. 45

B.1.7 Few-shot Chain-of-Thought Q&A From Notes Summary

This section presents model-generated ASA classifications, patient summaries, and step-by-step rationales, along with supplemental material on dataset construction, statistical comparisons, and the effects of prompt strategies on classification errors and predicted postoperative durations. It also reports improved hospital-mortality predictions with few-shot and chain-of-thought prompting, while duration prediction remains poor.

B.1.7 Few-shot Chain-of-Thought Q&A From Notes Summary

  • The Q&A examples assign ASA 2 to patients with mild or controlled disease, ASA 3 to a patient with persistent airway stenosis, ASA 5 to emergency cases, and ASA 6 to a declared brain-dead organ donor.The explanations link classifications to the clinical conditions and the ASA definitions.Source: §B.1.7 · pp. 46, 47, 48, 50

B.1.8 Summary Generation

  • The generated patient summary recaps Homer Simpson’s atrial fibrillation, post-cardioversion stability, continued Xarelto, and planned catheter ablation.It also notes the plan to continue anticoagulation through ablation and for at least six weeks afterward.Source: §B.1.8 · p. 52

B.1.9 Chain-of-thought Rationale Generation

  • The generated rationale supports ASA 5 for Barney Gumble by citing his Type A aortic dissection, traumatic injuries, intubation, and ICU stabilization.It reasons that the dissection is life-threatening and that survival without surgery is poor.Source: §B.1.9 · pp. 52–53

Supplemental Figure 2: Dataset Creation Flow Diagram

  • The dataset flow diagram says rare outcomes were balanced, only one case per patient was retained, and clinical notes covered up to the last 10 pre-procedure notes; ICU admission and duration datasets were identical.Source: Figure B1 · p. 54

Supplemental Figure 3: Dataset Case Overlap

  • Most task-specific inference and few-shot datasets had less than 5% case overlap, while ICU admission and ICU duration datasets were identical.Source: Figure B2 · p. 55

Supplemental Figure 6: ICU Admission - Statistical Significance

  • The ICU admission significance figure compares prompt strategies using F1 score and Wilcoxon signed-rank tests with Benjamini-Hochberg false-discovery control at α = 0.05.The supplied caption does not provide p-values or identify significant comparisons.Source: Figure B5 · p. 58

Supplemental Figure 8: Hospital Mortality - Statistical Significance

  • The hospital mortality significance figure compares prompt strategies using F1 score and Wilcoxon signed-rank tests with Benjamini-Hochberg false-discovery control at α = 0.05.The supplied caption does not provide p-values or identify significant comparisons.Source: Figure B7 · p. 60

Supplemental Figure 9: Post-Anesthesia Care Unit (PACU) - Phase 1 Duration - Statistical Significance

  • The PACU Phase 1 duration significance figure compares prompt strategies using mean absolute error in minutes and Wilcoxon signed-rank tests with Benjamini-Hochberg false-discovery control at α = 0.05.The supplied caption does not provide p-values or identify significant comparisons.Source: Figure B8 · p. 61
  • The ICU-duration comparison reports mean absolute error in days and uses Wilcoxon signed-rank tests with Benjamini-Hochberg false-discovery control at α = 0.05.The caption specifies that cells with p-values below 0.05 are marked dark blue, but provides no cell values.Source: Figure B10 · p. 63
  • ICU-admission confusion matrices show high true-positive and true-negative counts relative to false positives and false negatives across all 12 prompt strategies.The caption does not report strategy-specific counts.Source: Figure B13 · p. 65
  • Few-shot and chain-of-thought prompting improved ASA predictions for classes 1 and 6 and reduced hospital-admission false positives, but worsened unplanned-admission performance by increasing false negatives.For unplanned admissions, these strategies increased true negatives but also increased false negatives; zero-shot approaches without chain-of-thought had low false negatives but high false positives.Source: Figure B11 · p. 64, Figure B12 · p. 64, Figure B14 · p. 65

Supplemental Figure 16: Hospital Mortality - Predicted vs. Actual for Each Prompt Strategy

  • Few-shot and chain-of-thought prompting reduced hospital-mortality false negatives, improved overall performance, and left false positives rare.The figure compares predicted and actual hospital mortality across all 12 prompt strategies.Source: Figure B15 · p. 66

Supplemental Figure 18: Hospital Duration - Predicted vs. Actual for Each Prompt Strategy

  • Across duration tasks, few-shot and chain-of-thought prompting removed PACU and hospital-duration ceiling effects, while few-shot prompting removed the ICU-duration ceiling effect; prediction remained poor.The figures show the original ceilings at 180 minutes for PACU, 14 days for hospital duration, and 10 days for ICU duration.Source: Figure B16 · p. 67, Figure B17 · p. 68, Figure B18 · p. 69

Related papers

Loading 2401.01620v1…