Enhancing Extubation Failure Prediction with LLM-Derived Features from Respiratory Therapy Clinical Notes

Izzy Chaiken; Aditya Khowal; Neha A. Sathe; Mark M. Wurfel; Lucy Lu Wang

Source-linked AI summary

Enhancing Extubation Failure Prediction with LLM-Derived Features from Respiratory Therapy Clinical Notes

Izzy Chaiken, Aditya Khowal, Neha A. Sathe, Mark M. Wurfel, Lucy Lu Wang

arXiv:2609.17532v1Jun 17, 2026cs.CLcs.LGphysics.soc-ph

TL;DR

Extubation failure prediction must balance the risks of prolonged ventilation against failed extubation, yet prior models largely omit information in respiratory therapy notes. This paper uses few-shot LLMs to classify EF-related note features and adds them to risk models, improving performance while showing that cohort definitions materially affect results and generalizability.

Problem
Prior EF models largely rely on structured EHR data, while inconsistent cohort inclusion criteria and EF definitions hinder comparison and generalizability.
Method
The study uses few-shot LLMs to classify 15 EF-relevant features from respiratory therapy notes and incorporates them into downstream EF prediction models.
Results
RT note features improve AUROC by 1.9 points in the best models and by 5.1 points among patients with documented RT notes.
Takeaways & Limitations
LLM-derived RT note features can add clinically relevant information to EF risk models, while standardized task definitions are needed for model comparability and clinical translation.
Takeaways & Limitations
The study uses a single healthcare system without a comparable public dataset for external validation, limiting generalizability across populations, notes, languages, and time.

Abstract

from arXiv · show

Invasive mechanical ventilation is a lifesaving therapy, but timely, safe discontinuation is essential to preventing extubation failure (EF) and related risks to health. We present a novel approach to EF prediction that leverages features classified in free-text respiratory therapy notes using a large language model and logistic regression pipeline. Applied to a patient cohort from University of Washington Medicine, our method identifies clinically meaningful EF-related features that improve EF prediction performance when included alongside structured patient data. We further highlight how differences in target populations in prior EF prediction studies, such as heterogenous inclusion criteria and EF definition, can lead to systematic differences in model performance and hinder generalizability between studies.

1. Introduction

Extubation failure prediction is clinically important because both prolonged mechanical ventilation and failed extubation carry substantial risks. The paper combines structured EHR data with LLM-classified respiratory therapy note features and examines how cohort definitions affect performance.

1. Introduction: Both prolonged IMV and EF are associated with serious adverse outcomes, including airway injury, pneumonia, longer ICU stays, and excess mortality.These risks make the timing of extubation a complex clinical decision.
1. Introduction: EF labeling depends on minimum initial IMV duration and the maximum failure window, with failure defined by whether reintubation or death occurs within that window.The study varies these criteria and uses structured features from the prior 4 hours and note features from the prior 12 hours.
1. Introduction: The study introduces an LLM pipeline that classifies 15 EF-relevant features, including cough strength and sputum quantity, from respiratory therapy notes.The notes describe patients’ airway and respiratory status during invasive mechanical ventilation.
1. Introduction: RT note features improve AUROC by 1.9 points in the best models and by 5.1 points when training only on patients with documented RT notes.The study incorporates these features into EF risk models alongside structured patient data.
1. Introduction: Increasing the minimum IMV duration generally decreases AUROC and increases F1, so models trained on different cohorts are not directly comparable.The study therefore investigates how inclusion criteria and EF definitions affect downstream model performance.

2. Related Work

Prior EF prediction work mainly uses structured EHR variables, while inconsistent cohort and outcome definitions complicate comparisons. Related work motivates extracting information from clinical notes and systematically examining selection criteria.

2. Related Work: Clinical decision-support tools can guide extubation-time assessment, but the lack of standardized EF risk assessment leads to variation in assessed risk and treatment choices.Prior work describes tools that may defer extubation or initiate therapies for patients considered high risk.
2. Related Work: Prior EF models largely use structured variables and commonly achieve AUROC between 0.83-0.85 across variable-sized cohorts.Reported approaches include gradient boosting, artificial neural networks, and some longitudinal-data models.
2. Related Work: EF research varies in failure windows and minimum initial IMV duration, producing inconsistent risk factors and outcome predictions.Shorter windows may miss delayed failure, while IMV duration is itself associated with EF risk.
2. Related Work: Table 1 summarizes prior models, cohort statistics, inclusion criteria, and EF rates to contextualize variability across experimental settings.EF rate provides an additional indicator of differences among study populations.
2. Related Work: The paper examines EF failure windows and cohort selection criteria because these choices affect model performance.This addresses a comparability problem across prior studies.

3. Data

The study assembles a University of Washington Medicine cohort of mechanically ventilated patients and defines EF as death or return to IMV within seven days. Exclusions and an expanded one-hour-intubation cohort support analyses of selection criteria.

3. Data: The source cohort includes 10,194 patients across 10,810 visits to three University of Washington Medicine hospitals between April 2021 and September 2023.Extubation is identified by a documented change from IMV to a non-IMV oxygen-delivery method.
3. Data: EF is defined as patient death or return to IMV within 7 days of initial extubation.This operational definition is applied to the cohort’s outcome labeling.
3. Data: The primary cohort excludes encounters with incomplete height or weight, no IMV session lasting at least 24 hours, a recent Do Not Intubate/Resuscitate order, or non-unique patients.These exclusions reduce the initial 10,810 encounters to the primary analysis set.
3. Data: The primary dataset contains 3,243 IMV sessions from unique patients, including 647 patients who experienced EF within 7 days.Among these EF cases, 618 resulted from reintubation and 29 from death.
3. Data: An additional 3,685 encounters use a minimum initial intubation duration of 1 hour to evaluate the impact of broader inclusion criteria.Among these additional patients, 133, or 3.61%, experienced EF.

4. RT Note Feature Classification

The paper extracts clinically relevant respiratory features from free-text RT notes using few-shot LLM classification, producing patient-level predictors for EF models.

15 EF-relevant features spanning five categories were defined from literature review and pulmonologist input.
Few-shot prompts classify entire RT notes into predefined feature values because the task targets patient-level qualities rather than note spans.
400 RT notes were manually annotated through iterative author review and critical-care pulmonologist consensus to support prompt engineering and evaluation.
Test-set Macro-F1 ranged from 0.53 for spontaneous cough to 0.99 for thick sputum, with most feature values above 0.80.
The final pipeline classified 2,869 notes from 2,509 valid encounters within 12 hours before extubation; high sputum quantity and weak cough co-occurred with EF more often than overall.

5. EF Prediction Methods

EF models combine structured EHR variables with medications and LLM-derived RT-note features, while additional experiments test model families and alternative cohort definitions.

Logistic regression and gradient boosting predict binary EF outcomes using structured EHR, medication, and RT-note feature combinations.
Baseline predictors include IMV duration, vitals, labs, ventilation data, admission diagnoses, age, and documented sex, with measurements averaged over four hours before extubation.
Supplementary variants assess temporal generalizability by training on 2021–2022 admissions and testing on 2023 patients.
Survival models estimate time-specific EF risk to test whether failure timing improves prediction beyond a single binary outcome.
The primary evaluation reports AUROC on held-out test data alongside AUPRC, recall, specificity, precision, NPV, F1, and accuracy.
Additional analyses vary minimum IMV duration from 1 to 24 hours and the EF failure window from 12 hours to 14 days.

6. EF Prediction Results

Adding RT-note features improves logistic-regression EF prediction, while performance varies with cohort inclusion criteria and does not improve systematically with longer EF windows.

0.729→0.749 AUROC after adding RT-note features to baseline logistic regression, and 0.733→0.752 when medications are also included.These gains correspond to LRB versus LRB+N and LRB+M versus LRB+M+N.
Training AUROC was 0.712 for all-variable logistic regression versus 0.782 for gradient boosting, suggesting greater vulnerability of complex models to train–test distribution shifts.
The Cox model reached 0.742 AUROC versus 0.677 for gradient-boosting survival analysis, indicating no added binary-EF performance from naive survival-model application.
RT-note sputum quantity and thickness ranked among the 20 most important predictors, alongside structured variables such as IMV duration, plateau pressure, and SpO2.IMV duration alone achieved 0.658 AUROC.
The LRB+M+N model identified 59 of 81 EF cases as high risk versus 40 identified by the checklist CDSS, while the two tools remained complementary.The model’s precision was 34% versus 21% for the CDSS, but checklist-triggered treatment may affect observed outcomes.
0.796→0.752 AUROC when minimum IMV duration increases from 1 to 24 hours, while positive-class F1 rises from 0.390 to 0.467 and negative-class F1 falls from 0.795 to 0.777.The recall and specificity trade-off differs sharply between patients with IMV under 24 hours and those with at least 24 hours.
The 1-hour and 24-hour cohorts have different predictor importance patterns and patient characteristics, supporting limited direct comparability across differently defined cohorts.Acute respiratory failure, age over 60, and FiO2 rank among top predictors only in the 24-hour minimum model; 12.7% versus 5.2% underwent intubation for their entire hospitalization.
Changing the EF window from 12 to 336 hours produces no major AUROC or class-F1 changes despite EF prevalence increasing from 8% to above 21%.IMV duration remains the most important predictor across EF-window definitions.

7. Discussion & Conclusion

The discussion finds that LLM-classified respiratory therapy note features improve EF prediction while preserving interpretability, but model performance and generalizability depend strongly on cohort definition, setting, and retrospective design.

Discussion & Conclusion: LLM-classified respiratory therapy note features improve predictive performance and identify clinically relevant EF-associated variables.The approach offers interpretability compared with dense note embeddings, although with less expressivity.
Discussion & Conclusion: At 74.2% sensitivity, adding RT note features reduces false-positive rate from 40.0% to 34.1%, equivalent to five fewer false alarms per 100 patients.The reported decision-curve analysis also indicates one additional true positive per 100 patients in a specified alarm trade-off setting.
Discussion & Conclusion: High sputum volume, thick sputum, and weak cough are associated with higher EF risk and may be valuable future structured EHR elements.Long IMV duration, low SpO2, blood urea nitrogen, and brain injury provide external validity because prior work associates them with EF.
Discussion & Conclusion: Model comparisons are sensitive to algorithm and cohort design: logistic regression may match gradient boosting, while AUROC decreases and F1 increases with higher minimum IMV duration.The authors attribute this partly to removing lower-risk patients and emphasize that clinically meaningful tasks require clinician collaboration.
Discussion & Conclusion: The retrospective analysis cannot establish causal effects because treatment counterfactuals and non-clinical extubation influences are unavailable.The authors also note that diagnosis and treatment effects are difficult to disentangle without counterfactuals.
Discussion & Conclusion: Generalizability is limited by the single-healthcare-system dataset, unavailable external validation, changing patient populations, and possible failure across notes, languages, or settings.The authors call for standardized task definitions and further evaluation across demographic and clinical groups.

Appendix D. Model Implementation Details

The implementation standardizes inputs and evaluates models with cross-validation and hyperparameter search for both medication and structured-feature components.

Appendix D. Model Implementation Details: Medication inputs include opioid, benzodiazepine, vasopressor, crystalloid, and neuromuscular-blockade variables.Dosages are represented using medication-specific equivalent measures where applicable.
Appendix D. Model Implementation Details: Features are standardized to zero mean and unit variance, while missing values are imputed using training-set means.Model performance is averaged over 8-fold cross-validation of the training set.
Appendix D. Model Implementation Details: Gradient boosting and logistic regression hyperparameters are selected through grid search over model-specific complexity and regularization settings.The search includes estimator count, learning rate, tree constraints, iteration limits, and L2 regularization strength.

Appendix E. Predictors for LR Models

The LR predictor appendix describes the combined structured, medication, and respiratory-note model and highlights stable high-impact coefficients alongside model-specific differences.

Appendix E. Predictors for LR Models: The four most important coefficients are shared between models using 24-hour and 1-hour minimum IMV thresholds, while several lower-ranked features and FiO2 coefficient direction differ.These shared features are also consistent with predictors reported in prior literature.
Appendix E. Predictors for LR Models: The LRB+M+N model combines structured variables with medication and respiratory-therapy note features.The listed structured inputs include vitals, laboratory measurements, ventilation data, and diagnoses.
Appendix E. Predictors for LR Models: The LRB+M+N precision-recall curve reaches AUROC 0.75 but does not exceed precision 0.6 on the test set.This visual summarizes the model’s discrimination alongside its precision limitation.

Appendix F. Analysis of Inclusion Criteria

The inclusion analysis shows that minimum IMV duration changes performance and subgroup behavior, whereas EF-window changes have limited overall effect; temporal shifts further constrain interpretation.

Appendix F. Analysis of Inclusion Criteria: As minimum IMV duration increases, logistic-regression negative-class F1 decreases and positive-class F1 increases, while gradient boosting shows a different negative-class pattern.The gradient-boosting pattern reflects specificity values of 0.683, 0.560, and 0.728 for 1-, 12-, and 17-hour thresholds, respectively.
Appendix F. Analysis of Inclusion Criteria: Demographic subgroup differences are not statistically significant, although white and non-male groups show lower AUROC with lower sensitivity and higher specificity.The analysis lacks adequate sample size for robust significance testing, and the white subgroup comprises more than 60% of the dataset.
Appendix F. Analysis of Inclusion Criteria: Temporal evaluation is affected by outcome shift: EF prevalence is 21% in 2021–2022 training data versus 13% in 2023 test data.The authors conclude that rapid institutional shifts may require model retraining.

Appendix I. Calibration Analysis

Calibration analysis finds similar calibration for models with and without respiratory therapy note features, with both showing slopes above 1 and Brier scores around 0.14.

Appendix I. Calibration Analysis: Both LRB+M and LRB+M+N models have calibration curve slopes greater than 1 and Brier scores of roughly 0.14.The similar Brier scores indicate that adding respiratory therapy note features does not significantly change calibration performance.

Appendix J. Decision Curve Analysis

Decision curve analysis evaluates whether extracted respiratory note features improve the clinical net benefit of EF prediction. The LRB+M+N model provides higher net benefit than LRB+M at thresholds from 0.17 to 0.22, with similar benefit outside that range.

Appendix J. Decision Curve Analysis: Decision curve analysis measures the additional true positives per false positive from including extracted respiratory note features in EF models.The analysis finds similar net benefit at higher or lower thresholds.
Appendix J. Decision Curve Analysis: Calibration curves and Brier scores are included as complementary assessments alongside decision curves for the LRB+M and LRB+M+N models.Both models have similar calibration, with Brier scores of roughly 0.14.
Appendix J. Decision Curve Analysis: Roughly 0.01 higher net benefit is achieved by LRB+M+N than LRB+M at thresholds between 0.17 and 0.22.The gain corresponds to additional true positives relative to false positives.