PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero Shot Health Trajectory Prediction Using Transformer

Pawel Renc, Yugang Jia, Anthony E. Samir, Jaroslaw Was, Quanzheng Li, David W. Bates, Arkadiusz Sitek

arXiv:2407.21124v1cs.LGcs.AIcs.CY

TL;DR

Healthcare AI development faces barriers including the expense and time required to create labeled datasets. ETHOS forecasts future patient health timelines using a zero-shot transformer approach, and the authors report performance comparable to multiple task-optimized models.

  • Problem

    Healthcare AI adoption is limited in part by the expense and time needed to develop large labeled datasets and by limited capacity to deliver recommendations to clinicians at the optimal time.

    Source: §1 · p. 1

  • Method

    ETHOS uses unsupervised learning on patient health timelines to forecast future timelines and perform zero-shot inference without task-specific training or fine-tuning.

    Source: §1 · pp. 2–3

  • Results

    ETHOS achieves performance levels comparable to multiple models optimized for different tasks, using a single model trained once with zero-shot inference.

    Source: §3 · p. 7

  • Takeaways & Limitations

    The authors conclude that ETHOS may lower the costs and complexities of AI model development and accelerate healthcare AI development and implementation.

    Source: §3 · p. 9

    ETHOS is not yet generalizable across healthcare systems because data-collection methods, quality, data types, and patient populations differ.

    Source: §3 · p. 8

Abstract

from arXiv · show

Integrating modern machine learning and clinical decision-making has great promise for mitigating healthcare's increasing cost and complexity. We introduce the Enhanced Transformer for Health Outcome Simulation (ETHOS), a novel application of the transformer deep-learning architecture for analyzing high-dimensional, heterogeneous, and episodic health data. ETHOS is trained using Patient Health Timelines (PHTs)-detailed, tokenized records of health events-to predict future health trajectories, leveraging a zero-shot learning approach. ETHOS represents a significant advancement in foundation model development for healthcare analytics, eliminating the need for labeled data and model fine-tuning. Its ability to simulate various treatment pathways and consider patient-specific factors positions ETHOS as a tool for care optimization and addressing biases in healthcare delivery. Future developments will expand ETHOS' capabilities to incorporate a wider range of data types and data sources. Our work demonstrates a pathway toward accelerated AI development and deployment in healthcare.

1 Introduction

ETHOS is presented as a transformer foundation model for patient health timelines, intended to address barriers to healthcare AI adoption including limited labeled data. It forecasts future timelines without task-specific training and is demonstrated across diverse clinical prediction and classification tasks.

1 Introduction

  • Healthcare AI adoption is constrained by costly, time-consuming labeled datasets and limited capacity to deliver recommendations to clinicians at the optimal time.The introduction also describes existing decision-support technologies as mostly rule-based, with limited effectiveness so far.Source: §1 · p. 1
  • ETHOS applies a transformer to Patient Health Timelines (PHTs), tokenized chronological records of health events, forecasting future PHTs token-by-token from patient history.PHT tokens can represent admissions, medications, or time intervals.Source: §1 · pp. 1–2
  • The authors describe adapting ETHOS to medical images, clinical and discharge notes, monitoring, wearable, and omics data as possible with appropriate modifications.They also emphasize that forecasting future PHTs supports bespoke applications and that their training and testing methodology imposes no specific data-selection criteria.Source: §1 · pp. 2–3
  • The evaluation uses MIMIC-IV, with more than 400,000 hospitalizations in over 200,000 patients, retained in its original noisy form without cleaning or targeted imputation.The authors assume ETHOS can manage noise and anomalies through sufficient data, tokenization, and training, and anticipate performance gains from expanding the dataset and data types.Source: §1 · pp. 2–3
  • Demonstrated tasks include inpatient and ICU mortality, ICU length of stay, readmission, first-day SOFA regression, and discharge DRG classification across more than 771 categories.The SOFA estimate uses information available before ICU admission.Source: §1 · p. 3
  • ETHOS is presented as an EMR foundation model distinct from clinical language models for text and other EMR models that produce machine-understandable representations.The introduction places ETHOS among foundation-model approaches while emphasizing its focus on zero-shot inference.Source: §1 · p. 3
  • ETHOS uses unsupervised learning to forecast future health events without labeled data, fine-tuning, or task-specific training.The authors contrast this with methods requiring task-specific training and describe inference across a wide array of tasks.Source: §1 · pp. 2–3

2 Results

ETHOS’s quantile and time-interval embeddings reflect the order and relationships of the values they encode. Across zero-shot clinical tasks, it reports results for mortality, readmission, ICU stay, SOFA estimation, and DRG assignment.

2.1 Tokenization of MIMIC data and training of ETHOS

  • Quantile embeddings follow Q1–Q10 order, with Q4–Q6 closer than Q9–Q10; time-interval embeddings track represented durations, grouping the shortest and longest bins.Source: §2.1 · pp. 3–4, Figure 2 · p. 4

2.2 ETHOS inferences

  • AUCs were 0.921 for hospital mortality and 0.927 for ICU mortality; sepsis-subgroup ICU mortality reached 0.889 versus 0.762 ± 0.006 in Pan et al. [2023].Comparisons were against ETHOS trained on the entire test population and task-specific algorithms developed using smaller, curated MIMIC subsets.Source: §2.2 · p. 6
  • ICU length-of-stay prediction had MAE 2.262 days (95% CI: 2.161-2.355 days).Source: §2.2 · p. 6
  • ICU readmission AUC was 0.807, hemorrhage ICU readmission AUC was 0.667, and hospital readmission AUC was 0.749; missing outcome data may limit readmission accuracy.Source: §2.2 · p. 6
  • First-day SOFA estimation achieved MAE 1.502 (95% CI: 1.475-1.534), and the authors knew of no prior literature predicting it at admission.Source: §2.2 · p. 6
  • 84.8% top-1 accuracy across 771 DRG classes exceeded the 52% reported by Wang et al. [2024].The result covers 28,932 hospitalizations.Source: §2.2 · p. 6

3 Discussion

ETHOS is presented as a single zero-shot foundation model whose performance is comparable to multiple task-optimized models. The discussion highlights its potential uses and outlines limitations in data coverage, generalizability, and clinical readiness.

3 Discussion

  • Figure 4 illustrates first-day SOFA estimation from quantile probabilities and DRG classification from a token placed after discharge-related tokens.The figure includes an actual-versus-predicted SOFA correlation plot and visualizations of DRG accuracy.Source: Figure 4 · p. 7
  • A single ETHOS model trained once with zero-shot inference achieves performance comparable to multiple models optimized for different tasks.The authors frame this as a potential route to more efficient and scalable medical-data model development.Source: §3 · p. 7
  • ETHOS predicts future patient timelines from past information, and its zero-shot approach requires no additional downstream training.Prior work has applied patient timelines and foundation models separately; ETHOS combines them with zero-shot inference [Jiang et al. 2023, Steinberg et al. 2021, Li et al. 2023, Savcisens et al. 2024, Bornet et al. 2023, Moor et al. 2023].Source: §3 · p. 7
  • ETHOS could model individualized treatment paths and projected costs at care decision points, unlike approaches based on population-averaged evidence.The authors illustrate this potential with alternative drug choices and patient-specific conditions [Zack et al. 2024, Obermeyer et al. 2019, Abid et al. 2021].Source: §3 · p. 8
  • Explainability, fairness, and transparency remain ongoing work, including planned attention visualizations and a dedicated clinical decision-making interface.The authors describe both developments as future work.Source: §3 · p. 8
  • ETHOS currently omits several data types, and it is not yet generalizable across sources with differing collection methods, data quality, and patient populations.A universal tokenization format may address some compatibility issues, but the authors say it will not resolve them all; they also propose translating PHTs between systems.Source: §3 · p. 8
  • ETHOS explicitly models quantitative values and temporal sequences in structured patient data, whereas general-purpose LLMs are not designed to model those elements.The authors cite studies describing LLM limitations in temporal information and decision support [Hager et al. 2024, Wang and Zhao 2024].Source: §3 · p. 8
  • ETHOS was evaluated on MIMIC, which may be cleaner than routine clinical datasets; the authors call for prospective testing in diverse, real-time settings.The current model uses up to 2048 PHT tokens, which may be insufficient when token density is high.Source: §3 · p. 9

4 Methods

ETHOS converts MIMIC-IV records into Patient Health Timelines and trains a transformer decoder to predict future tokens. The methods use these simulated timelines for clinical prediction tasks and comparisons with existing approaches.

4.1 Data

  • MIMIC-IV records from 12 tables supplied the patient, encounter, laboratory, medication, procedure, diagnosis, and other clinical data used to build timelines.Clinical notes were excluded because they require additional processing for tokenization.Source: §4.1 · p. 9

4.2 Patient health timelines (PHTs), tokenization

  • PHTs arrange events chronologically, encode each event with 1–7 tokens, and represent time gaps with interval tokens before removing timestamps.Intervals under 5 minutes receive no token; gaps longer than a year use repeated six-month tokens.Source: §4.2 · p. 9, Figure 5 · p. 10
  • Age and timeline-start dates use five-year bins, while six static tokens encode demographics and other fixed data at the start of the 2,048-token context.Numerical values are assigned quantile tokens; codes use hierarchical tokens, and future-obtained SOFA and DRG values are excluded during inference.Source: §4.2 · pp. 10–11

4.3 ETHOS training

  • ETHOS learns next-token prediction from concatenated patient timelines using a transformer decoder with learned position embeddings.The model size began near GPT-2 scale and was adjusted heuristically to optimize the loss function.Source: §4.3 · pp. 11–12

4.4 Evaluation of Clinical Outcomes and Tasks Using ETHOS

  • Mortality and readmission probabilities are estimated from repeated simulated timelines, while ICU length of stay is derived from generated time intervals.The mortality and readmission procedures each use 20 simulations; ICU stays ending in death are excluded from LOS calculation.Source: §4.4 · p. 12
  • SOFA scores are predicted from prior information, and DRG classes are inferred from output-token probabilities after discharge-related tokens.The dataset’s uncertain SOFA assessment timing creates a potential training-time causality issue, but true SOFA values are not used during inference.Source: §4.4 · p. 12

4.5 Statistical Analysis

  • Binary tasks were assessed by ROC/AUC with bootstrapped 95% confidence intervals, DRG classification by top-1/top-2 accuracy, and regression by MAE.ROC curves were fitted with Gaussian models allowing unequal variances.Source: §4.5 · p. 12

4.6 Comparison of ETHOS to existing methods

  • ETHOS significantly outperformed both GPT-4o temperature variants in readmission testing; traditional non-temporal baselines summarized time-varying features.The GPT-4o prompt analyzed 2,048 tokens for 2,000 test cases.Source: §4.6 · p. 13

Additional information

The supplementary material reports data and code availability, author contributions, and competing interests.

Additional information

  • MIMIC-IV is publicly available, and the paper’s code, model weights, inference results, and result-generation scripts are published online.The experiments used Python 3.10 and the listed open-source libraries.Source: Additional information · p. 13
  • AS and PR conceptualized the work and prepared the initial draft; YJ reported an additional affiliation with Verily, while other authors declared no competing interests.All authors participated in manuscript refinement and review.Source: Additional information · p. 13

Supplementary Information

The supplementary material provides dataset characteristics, model and tokenization figures, and examples of tokenized patient timelines.

Supplementary Information

  • Table S1 reports demographic characteristics at first hospital admission for characteristics used by ETHOS.Source: Table S1 · p. 16
  • Figures S1 and S2 depict code tokenization and the ETHOS transformer decoder architecture, respectively.The S1 colors distinguish code parts encoded by different tokens; S2 includes training traces and model parameter specifications.Source: Figure S1 · p. 17, Figure S2 · p. 18
  • Tables S3 and S4 provide examples of tokenized timelines for hospital and ICU admissions, including diagnoses, medications, laboratory results, and stay intervals.Source: Table S3 · p. 24, Table S4 · p. 25, Supplementary Information · pp. 25–27
Loading 2407.21124v1…