PaperlayerView exact revision on arXiv

Source-linked AI summary

NOAH: Learning the Full Patient Journey. A Longitudinal Multimodal Time-Aware Model for Representation and Forecasting

Tobias Susetzky, Raphael Rehms, Dmitrii Seletkov, Özgün Turgut, Michelle Espranita Liman, Lisa Steinhelfer, Rickmer Braren, Daniel Rueckert

arXiv:2609.09140v1cs.LGcs.AI

TL;DR

Longitudinal multimodal clinical records are difficult to model because prior systems often handle limited modalities, irregular temporal dynamics, and stochastic future trajectories incompletely. NOAH is a generative, time-aware transformer with variational patient states and bidirectional time integration, pretrained on a large MIMIC corpus. It supports multimodal representation, forecasting, zero-shot outcome estimation, and counterfactual simulation, with reported performance across diverse clinical tasks.

  • Problem

    Existing longitudinal patient-record models are often limited in modalities, temporal handling, vocabularies, or forecasting ability, despite the need to represent stochastic clinical progression.

  • Method

    NOAH is a time-aware, task-agnostic generative multimodal transformer using context-conditional variational states and bidirectional time integration.

  • Results

    NOAH demonstrates probabilistic zero-shot forecasting, time-to-event prediction, and compact patient-state representations across diverse clinical downstream tasks.

  • Takeaways & Limitations

    NOAH provides a scalable foundation for holistic patient-trajectory analysis with autoregressive rollouts, optional time control, and counterfactual simulations.

  • Takeaways & Limitations

    Autoregressive exposure bias can propagate errors over long rollout horizons, and the representations are predictive features rather than patient-history summaries.

Abstract

from arXiv · show

The digitization of healthcare has generated vast, longitudinal, and multimodal patient records over a lifetime, yet fully exploiting these data to represent and predict patient state trajectories remains a critical challenge. Current AI models often struggle to capture the complex, irregular temporal dynamics and inherent stochasticity of real-world multimodal patient data. Existing AI approaches for modeling longitudinal patient records are predominantly discriminative, limited to a few modalities, constrained by closed categorical vocabularies, treating time as a monotonic inductive bias, or they are limited in forecasting future patient states. We introduce NOAH, a time-aware, task-agnostic, generative transformer model representing and forecasting the full multimodal patient journey. NOAH features a novel bidirectional time integration and a variational latent space to capture the continuous evolution of patient states and the stochasticity of clinical trajectories. Built from over 559 million clinical events from 431,000 hospital visits of 299,000 patients across the MIMIC dataset family, NOAH natively processes medical images, time-series and numeric signals, categorical events, as well as structured and unstructured clinical records. NOAH is the first truly holistic generative model in its field, enabling autoregressive forecasting with optional time control, zero-shot classification, and counterfactual intervention simulation. It generates highly informative and predictive patient state representations that demonstrate strong performance in probing for clinical outcomes, 15 ICD chapters, and 29 comorbidities, as well as in time-to-event prediction. Seamlessly handling diverse modalities and complex temporal dynamics, NOAH provides a versatile, task-agnostic, scalable foundation for intelligent predictive systems in personalized clinical care and digital medicine.

Introduction

NOAH addresses limitations in modeling lifelong multimodal clinical records by combining generative forecasting, stochastic latent states, and horizon-aware temporal integration. It is presented as a scalable foundation for representing and predicting patient trajectories across diverse downstream tasks.

  • Motivation: Lifelong clinical records combine multimodal data with large scale, irregular timing, and stochastic disease progression that intelligent systems must represent and forecast.These records include measurements, imaging, text, signals, prescriptions, procedures, diagnoses, and administrative events.
  • Limitations of prior work: Existing approaches often cover only a few modalities, require fixed sampling or selected time points, and use closed or restricted event representations.Prior methods include multimodal combinations such as vision-language, imaging-tabular, and imaging-signals systems.
  • Temporal modeling gap: Standard temporal encodings can be insensitive to prediction horizon, although the relevance of a past event may differ for acute and chronic outcomes.An event informative for disease onset months ahead may be irrelevant to an acute event in coming days.
  • NOAH: NOAH uses a context-conditional variational approach and bidirectional time integration to model stochastic state transitions, irregular gaps, and horizon-dependent history relevance.It consumes the actual content of modalities including imaging and waveforms while varying autoregressive forecasts at inference.
  • Scope and evaluation: NOAH is pretrained on 559 million timestamped events and evaluated across probabilistic forecasting, time-to-event prediction, and patient-state representation tasks.The paper also highlights autoregressive rollouts with optional time control and counterfactual intervention simulations.

Results

NOAH is a transformer foundation model for temporal dynamics in multimodal patient trajectories. It represents events using their types, specifics, values, and temporal features across a large clinical dataset.

  • Model: NOAH is explicitly trained to learn temporal dynamics of patient states across multimodal trajectories.Its architecture is described as a transformer model with bidirectional temporally enriched attention.
  • Data: 559M timestamped events from 299k patients spanning 431k hospital visits provide the pretraining and evaluation corpus.The corpus comes from the MIMIC dataset family and supports numerous downstream setups.
  • Multimodal inputs: NOAH processes texts, X-ray and echocardiogram images, ECG waveforms, numerical data, and categorical data in arbitrary order and count.Event representations combine type category, free-text type specifics, value embeddings, and temporal features.
  • Representation: Free-text specifics and continuous values allow the model to consume unseen data such as updated drug names beyond a closed event vocabulary.Bidirectional inter-event deltas support temporally irregular sequences.

NOAH learns patient trajectories, a landscape of states and transitions.

NOAH organizes patient records as evolving latent trajectories of contextualized states and transitions. These trajectories separate differing clinical courses, care intensity, risk, and clinical novelty.

  • Latent patient states: 768-dimensional timestep representations form trajectories that evolve through embedding space, with similar starting conditions sometimes diverging according to later course and outcome.The representations are extracted from the transformer output before the variability module.
  • Latent patient states: Contextualized events, rather than isolated event types, primarily organize the learned latent space.The organization reflects care level and clinical progress.
  • Care and risk: Care intensity separates regions of the embedding space, while a high-risk zone emerges near the ICU region and lower-intensity care corresponds to stable risk.The visualizations include regions associated with time until death.
  • Transitions and novelty: Model surprise at state transitions often tracks clinical novelty, increasing before formally charted stays and peaking for emergency-department vitals.Surprise decreases with more context, stabilizing conditions, and emerging routines.
  • Transitions and novelty: Content-rich ECG and X-ray modalities remain genuinely hard to predict.This limitation is identified among the transition events and modalities analyzed.

NOAH can forecast patient trajectories with optional time control.

NOAH forecasts future multimodal events by autoregressively rolling out held-out patient trajectories across temporal horizons. Time control substantially improves short-horizon performance, while rollout errors grow with horizon.

  • Forecasting setup: 13,551 eligible test patients provide lifetime sequences for autoregressive rollout into held-out windows at different temporal horizons.The frozen model estimates event-type and modality probabilities and counts using Monte Carlo sampling.
  • Forecasting performance: AUROC 0.83-0.95 with Brier scores 0.03-0.11 outperforms a naive persistence baseline for event-type and modality occurrence forecasting.Calibration is reliable with ECE 0.08 to 0.094, despite slight overconfidence at high predicted probabilities.
  • Forecasting performance: MAEs of ≤3.1 and ≤10.7 occurrences are reported for type and modality classes, respectively, and errors naturally increase within 72 h during rollout.Longer rollouts increase divergence from ground truth as autoregressive errors propagate.
  • Optional time control: Time control improves performance by up to +0.19 AUROC within shorter, more sensitive horizons.Under time control, type-count error can increase at longer horizons because other event components remain error-prone.

Monte Carlo simulation enables zero-shot clinical decisions.

NOAH uses autoregressive rollouts and Monte Carlo sampling to estimate probabilities for clinical outcomes from patient histories. Performance varies by outcome and forecasting horizon.

  • AUROC 0.97 (Balanced Acc. 0.82) for 72h mortality prediction from events up to 48h after admission.
  • AUROC 0.74 (Balanced Acc. 0.68) for predicting length of stay ≥72 h from the same 48h cutoff.
  • AUROC 0.61 (Balanced Acc. 0.57) for 30-day readmission when prompting with events up to discharge.The authors attribute this weaker separation to error propagation and distribution drift during long-term rollouts.
  • Monte Carlo estimation draws multiple trajectories, locates target-token occurrences, and bins their time deltas to estimate outcome probabilities.

NOAH enables counterfactual simulations of clinical intervention.

NOAH simulates patient trajectories under alternative intravenous-fluid interventions and compares resulting outcomes with a clinical-trial subgroup. The simulated treatment effect favors Ringer’s, while absolute outcome calibration remains imperfect.

  • NOAH compares factual and counterfactual 0.9% saline versus lactated Ringer’s for sepsis patients using autoregressive trajectory simulation.The intervention is defined as the first intravenous fluid within six hours after a sepsis marker.
  • Ringer’s produces lower predicted mortality, shorter length of stay, and lower MAKE-30 than saline in the simulated trajectories.
  • The simulated average treatment effect favors Ringer’s and matches the sign of the SMART sepsis-subgroup effect at roughly twice its magnitude.
  • Figure 5 summarizes complementary downstream uses of NOAH embeddings, including outcome probing, NEWS2 retrieval, and time-to-event survival modeling.

NOAH’s patient state representations carry clinical information and risk over time.

NOAH’s patient-state embeddings encode information about diagnoses, comorbidities, clinical outcomes, and NEWS2 severity across a stay. Their predictive usefulness varies with when the representation is extracted.

  • AUROC scores range from 0.72 to 0.94 for 15 ICD chapters and 29 comorbidities retrieved from final patient states.
  • AUROC values are 0.87, 0.98, and > 0.99 for ICU length of stay, hospital length of stay, and mortality, respectively.
  • Peak probing performance occurs at the beginning of the stay for most tasks, while results are better for younger patients and shorter stays.
  • NEWS2 retrieval achieves MAE 1.34 across 39,952 windows from 4,414 held-out patients.
  • The embeddings achieve AUROC 0.94 for separating low versus medium-high NEWS2 risk and 0.91 for low-medium versus high risk.

NOAH’s patient state representations enable survival analysis.

NOAH’s patient-state representations support survival analysis as progressively more within-stay context becomes available. Later representations improve discrimination while preserving calibration.

  • Time-dependent C-index improves as embeddings from later phases of a patient’s stay are incorporated for time-to-death prediction.
  • D-calibration remains well calibrated across the evaluated within-stay time fractions.
  • The results indicate that progressively richer longitudinal representations provide increasingly informative signals for time-to-event prediction.

NOAH relies on time and specializes in clinical channels.

NOAH’s attention heads specialize in clinically meaningful longitudinal and observation-reporting relations, while bidirectional temporal terms shape attention according to prediction horizon. Its attributions emphasize physiological measurements and show that longer history improves prolonged-stay prediction, especially for elderly patients.

  • Head specialization: 141 of 144 attention heads specialize in clinical relations, including 76 same-type longitudinal links and 65 observation-reporting links.These patterns preserve per-modality history and encode clinical workflows and documentation logic.
  • Input attribution: Patient measurements are crucial for predicting the next token, while clinical structure dominates event-type prediction without outweighing other inputs.This attribution pattern indicates that NOAH uses physiological information rather than only record structure.
  • Temporal attention: Attention shifts most when content-content and time-content terms are removed, with time-content attention making history depend on prediction horizon.The forward time delta must be carried explicitly because causal masking prevents it from arising from history.
  • History ablation: 0.68 AUROC is reached for elderly patients with more history versus 0.63 with less history in zero-shot prolonged-stay prediction.Overall AUROC rises from 0.70 to 0.72 with more history, while younger subjects perform better across history extents.

Discussion

NOAH is presented as a universal patient-state encoder and prediction engine spanning lifetime multimodal records. Its representations support forecasting, zero-shot outcome estimation, and counterfactual intervention simulation, while autoregressive exposure bias and noncausal probabilistic counterfactuals delimit interpretation.

  • Method: NOAH learns longitudinal patient dynamics through bidirectional time integration and a context-conditional variational latent space.The latent space provides inference-time variability while preserving semantic coherence across generated-event components.
  • Representations: Linear probes and time-to-event models show that NOAH’s task-independent representations contain predictive clinical information at any point in time.Downstream systems need not directly process the complex multimodal patient records.
  • Capabilities: NOAH supports autoregressive future-event forecasting with optional time control, zero-shot clinical-outcome estimation, and counterfactual intervention simulation.These capabilities position it as both a patient-state encoder and a prediction engine across lifetime modalities and events.
  • Limitations: The model’s autoregressive rollouts face exposure bias because predictions are re-encoded iteratively rather than supplied as ground-truth predecessors.The paper states that this can propagate errors over long horizons.
  • Limitations: Counterfactual simulations are intervention-conditioned stochastic forecasts, not causal models.NOAH estimates likely developments from training data rather than physiologically grounded deterministic trajectories or logical deductions.

Online Methods

NOAH represents multimodal clinical timelines as uniform event tokens, integrates irregular time bidirectionally, and uses a context-conditional latent model for stochastic forecasting. Its evaluation spans zero-shot classification, counterfactual simulation, linear probing, and time-to-event prediction.

  • Temporal modeling: Bidirectional temporal attention adds time at token input and within attention, with queries using forward horizons and keys integrating recency.This design addresses irregular event spacing and makes temporal semantics explicit beyond ordinal sequence position.
  • Multimodal representation: NOAH converts heterogeneous timestamped clinical events into uniform tokens combining event categories, optional free-text specifics, modality value embeddings, and temporal information.Supported modalities include imaging, ECG waveforms, numeric values, and categorical data.
  • Variational autoregression: A shared continuous latent variable is sampled once per timestep and jointly consumed by decoder heads to model composite clinical events and trajectory variability.The context-conditional prior depends on history, while the posterior also observes the target event embedding during training.
  • Pretraining objective: The training objective combines event reconstruction, posterior-prior KL regularization, and auxiliary reconstruction from prior-sampled latents to improve predictive history representations.The model uses posterior-sampled and prior-sampled latents, with the latter mirroring inference-time sampling.
  • Evaluation framework: NOAH is evaluated through zero-shot Monte Carlo classification, counterfactual intervention simulation, linear probing, and time-to-event prediction across longitudinal patient contexts.The evaluation includes clinical outcomes and uses within-stay fractions from admission through the last event for context-based prediction.

Extended Data Figures

The extended-data figures illustrate NOAH’s zero-shot forecasting, longitudinal patient-state probing, NEWS2 recovery, attribution patterns, event timing, and counterfactual simulation across the MIMIC-based cohort.

  • Zero-shot forecasting: Monte Carlo rollouts estimate 72 h mortality, prolonged hospital stay, and 30-day readmission from patient history and current events without task-specific training.Rollouts continue until the target event is predicted, the class-0 boundary is reached, or the token budget is exhausted.
  • Patient-state probing: Linear probes decode ICD chapters and comorbidities from frozen patient states at successive points during each patient’s last stay.Pre-admission uses historical information only, while discharge is a clean retrieval point and earlier probes mix prediction with retrieval.
  • Clinical risk recovery: NOAH’s patient states support NEWS2 recovery, with exemplar predictions evaluated across time on held-out patients.The model maps patient-state embeddings to NEWS2 scores using windows where all required vital signs are available.
  • Attribution: Attribution analyses identify time as the most crucial input component, while value matters especially for clinically ambiguous event types such as toxicology labs, discharge summaries, and ECGs.The analysis decomposes contributions into event type, specifics, time, and value.
  • Event timing: Within hospital stays, event collection is highest near admission, declines through most of the stay, and rebounds sharply in the final bin.Events are binned by relative temporal progress from admission to discharge and summarized by stay-duration quartile.

Supplementary Information

The supplementary analyses characterize NOAH’s predictability, subgroup behavior, forecasting by horizon and stay length, counterfactual simulation, representation probing, NEWS2 scoring, and time-to-event evaluation.

  • Latent surprise: Latent surprise is lowest for routine measurements and higher for imaging, waveform events, and terminal events whose contents are less predictable from history.Per-event surprise is summarized using median, upper-tail, and mean KL(q∥p).
  • Forecasting: Forecasting analyses compare free and time-controlled variants across 24 h and multiple horizons, event types, modalities, and length-of-stay quartiles.The supplementary evaluations report macro-averaged AUROC and Brier scores, including best- and worst-predicted event types.
  • Counterfactual simulation: Counterfactual analyses simulate saline versus balanced crystalloid for mortality, length of stay, and MAKE-30 outcomes, while reporting outcome coverage and invalid-run ratios.The simulations use matched factual and counterfactual runs sharing each patient’s prompt up to the treatment time.
  • Representation probing: Linear probing evaluates clinical outcomes, ICD chapters, and Quan-Elixhauser comorbidities from patient-state embeddings at admission, intermediate timepoints, and discharge.Intermediate probes mix prediction and retrieval, whereas discharge is a clean retrieval setting; the evaluable sample shrinks for later positions.
  • Subgroups and NEWS2: Subgroup analyses assess retrieval by age and stay length, while NEWS2 analyses recover response bands and integer-score errors from held-out patient-state embeddings.The subgroup retrieval results investigate whether clinical information is retained across patient and stay characteristics.
  • Time-to-event prediction: Time-to-event evaluation reports time-dependent C-index, integrated Brier score, and D-calibration p-values for within-stay survival predictions.The supplementary material also documents the survival-analysis hyperparameter search space and dataset modality statistics.
Loading 2609.09140v1…