NOAH: Learning the Full Patient Journey. A Longitudinal Multimodal Time-Aware Model for Representation and Forecasting

• Lifelong clinical records are massive, multimodal, irregular, and stochastic, motivating intelligent systems for semantic analysis, trajectory prediction, and individualized intervention simulation.Source: Introduction · p. 2
• Existing approaches typically cover few modalities, impose fixed sampling or closed categorical vocabularies, and fail to model patient data holistically in a task-agnostic way.Source: Introduction · p. 2
• Standard temporal encodings can be insensitive to prediction horizon, while clinical trajectories require explicit stochasticity because patient states admit multiple plausible futures.Source: Introduction · pp. 2–3
• NOAH is a time-aware, task-agnostic, generative multimodal transformer that learns patient states and transitions using context-conditioned variation and bidirectional time integration.Source: Introduction · p. 3
• NOAH processes 559 million timestamped events from the MIMIC dataset family and supports zero-shot forecasting, time-to-event prediction, compact representations, time-controlled rollouts, and counterfactual simulations.Source: Introduction · p. 3

A time-aware model can understand and predict multimodal data over the entire patient journey.

• NOAH learns temporal dynamics across multimodal patient trajectories using a transformer with bidirectional temporally enriched attention and temporal control during inference.The model is pretrained and evaluated on real-world clinical data spanning the full patient journey.Source: A time-aware model can understand and… · p. 4
• 559M timestamped events from 299k patients and 431k hospital visits provide the training and evaluation foundation for downstream clinical applications.The dataset is built from the MIMIC family and represents longitudinal real-world care.Source: A time-aware model can understand and… · p. 5
• NOAH processes text, X-ray and ECHO images, ECG waveforms, numerical data, and categorical events in arbitrary order and count.Event representations combine type category, free-text specifics, value embeddings, and temporal features.Source: Figure 1 · p. 4, A time-aware model can understand and… · p. 5
• Free-text specifics and continuous values let NOAH consume unseen data, including updated drug names, beyond a closed event vocabulary.This design expands the model’s input space beyond fixed categorical event definitions.Source: A time-aware model can understand and… · p. 5
• Bidirectional timing features, including current age and inter-event deltas, enable NOAH to model temporally irregular sequences.The model incorporates timing information for each event rather than relying only on event order.Source: A time-aware model can understand and… · p. 5

Results

• As a generative transformer decoder, NOAH maintains coherent patient states across timestep outputs while allowing variability during autoregressive inference.Its latent module learns context-conditional patient states and transitions.Source: Figure 1 · p. 4, A time-aware model can understand and… · p. 5
• The pretrained model produces lifetime patient-state trajectories, forecasts future developments, and supports outcome simulation and counterfactual analysis.Figure 1 summarizes autoregressive rollout with optional time control and downstream use of patient-state embeddings.Source: Figure 1 · p. 4, A time-aware model can understand and… · p. 5

Patient-state trajectories

• NOAH’s latent trajectories separate contextualized events, care intensity, and risk, while patients with similar starts can diverge toward different courses and outcomes.The 768-dimensional transformer output before the variability module serves as the patient-state representation; projections show stable low-intensity regions and a high-risk zone near ICU states.Source: NOAH learns patient trajectories, a… · p. 5, Figure 2 · p. 6

Transition surprise

• Transition surprise increases before acute stays and peaks around ED vitals, drugs, and immediate procedures, then declines with context, stability, routines, repeated visits, and age.ED arrival is predictable from charting patterns, whereas death outside clinical settings and content-rich ECG or X-ray modalities remain difficult to predict.Source: NOAH learns patient trajectories, a… · p. 5

Forecasting and simulation

• NOAH can autoregressively roll out future patient trajectories with or without actual time deltas, evaluate prediction quality and calibration, and support zero-shot outcome classification.It also simulates multiple possible continuations and estimates clinical-outcome probabilities using Monte Carlo sampling.Source: Figure 3 · p. 7

Forecasting performance

• AUROC 0.83-0.95 and Brier scores 0.03-0.11 outperform a naive persistence baseline, while ECE 0.08 to 0.094 indicates reliable predictions with slight high-probability overconfidence.Evaluation used 13,551 eligible test patients and compared Monte Carlo forecasts with held-out ground truth across event categories and modalities.Source: NOAH can forecast patient trajectories… · pp. 7–8

Optional time control

• Time control improves performance significantly, increasing AUROC by up to +0.19 within shorter, more sensitive horizons.Time control supplies the ground-truth delta to the next token and enforces when NOAH predicts each subsequent event.Source: NOAH can forecast patient trajectories… · p. 8

Forecasting performance

• Count errors reach MAEs of ≤3.1 for event types and ≤10.7 for modalities, naturally increasing within 72 h as rollout errors propagate and forecasts diverge from ground truth.The model estimates event probabilities and counts for each category and modality within different temporal horizons.Source: NOAH can forecast patient trajectories… · pp. 7–8

Optional time control

• Under time control, modality-count error does not show the same long-horizon increase as type-count error, which worsens because enforced timing can push other sensitive predictions beyond the learned distribution.The cited explanation highlights errors in components such as the value embedding.Source: NOAH can forecast patient trajectories… · p. 8

Monte Carlo simulation

• AUROC reached 0.97 (Balanced Acc. 0.82) for 72h mortality and 0.74 (Balanced Acc. 0.68) for length-of-stay ≥72 h prediction after a 48h prompt cutoff.The model drew 25 or 50 trajectories and scored the first target-token occurrence by its time delta to the reference event.Source: Monte Carlo simulation enables… · p. 8
• AUROC was 0.61 (Balanced Acc. 0.57) for 30-day readmission when prompting with events up to discharge, exclusive.The weaker separation was attributed to error propagation and distribution drift during long-term autoregressive rollouts.Source: Monte Carlo simulation enables… · p. 8
• Scorable-outcome coverage was 65% for 30-day readmission simulations versus 80-82% for the shorter prolonged-stay and mortality simulations.Rollouts without a scorable outcome were discarded for Monte Carlo estimation.Source: Monte Carlo simulation enables… · p. 8

Counterfactual intervention simulation

• +8.92 and +8.86 points were the simulated treatment effects for MAKE-30 and death, respectively, favoring lactated Ringer’s over saline and matching SMART’s sepsis-subgroup effect sign at roughly twice its magnitude.NOAH truncated patient events after the first fluid intervention, swapped the factual and counterfactual fluids, and used autoregressive rollouts with Monte Carlo outcome estimation.Source: NOAH enables counterfactual… · pp. 8–9, Figure 4 · p. 9
• The simulations predicted lower mortality, shorter length of stay, and lower MAKE-30 under lactated Ringer’s than saline, with qualitatively diverging trajectories between arms.The model overpredicted mortality and underpredicted stay duration relative to the observed factual outcome.Source: Figure 4 · p. 9, NOAH enables counterfactual… · p. 9

Limitations

• Mortality and MAKE-30 overprediction increased with prediction horizon, attributed to autoregressive rollout drift outside the learned distribution, but did not affect within-patient treatment contrasts.Long-term survivor rollouts require avoiding significant clinical events, which is unlikely by design and contributes to the calibration error.Source: NOAH enables counterfactual… · p. 9

• AUROC reached 0.87 for ICU length of stay, 0.98 for hospital length of stay, and > 0.99 for mortality, while diagnosis and comorbidity probes achieved 0.72–0.94 across 15 ICD chapters and 29 comorbidities.These results came from logistic-regression probes of final-stay patient states under stratified five-fold cross-validation over three seeds.Source: NOAH’s patient state representations… · pp. 10–11
• NEWS2 retrieval achieved an MAE of 1.34 and AUROCs of 0.94 for low versus medium-high risk and 0.91 for lowmedium versus high risk across 39,952 windows from 4,414 held-out patients.Regression was well calibrated for low and medium severity and diverged from ground truth mainly for rarer high-risk scores.Source: NOAH’s patient state representations… · p. 11
• Probing performance peaked at the beginning of most stays, declined over time, and was better for younger patients and shorter stays, indicating embeddings prioritize predictive features over complete summarization.Initial complaints and treatment provided stronger signals, whereas routine procedures dominated mid- and long-term care; the authors characterize this as deliberate temporal forgetting.Source: NOAH’s patient state representations… · p. 11

Survival analysis

• Later within-stay embeddings improve time-to-death discrimination by time-dependent C-index while maintaining well-calibrated predictions by D-calibration.Embeddings are evaluated at within-stay fractions from hospital entry through the last event before death or discharge, using the same patients and endpoint across settings.Source: NOAH’s patient state representations… · p. 11

Clinical channel specialization

• 141 of 144 attention heads specialize in clinical relations, modeling within-type longitudinal dynamics and observation-reporting dependencies across modalities.76 heads connect same-type events across time, while 65 connect observations with their reporting, indicating internalized clinical workflows and documentation logic.Source: NOAH relies on time and specializes in… · p. 11, Figure 6 · p. 12

Input attribution

• Attributions show patient measurements drive next-value prediction, whereas clinical structure is most important for event-type prediction without outweighing other inputs.This indicates NOAH uses physiological information rather than relying only on administrative or clinical structure.Source: NOAH relies on time and specializes in… · p. 11, Figure 6 · p. 12

Bidirectional temporal attention

• Omitting content-content or time-content terms shifts attention most, showing that forward time deltas make attention depend on the prediction horizon.Content-time matters mainly in the input layer, whereas the forward time delta must be explicitly carried because causal masking prevents deriving it from history.Source: Figure 6 · p. 12, NOAH relies on time and specializes in… · p. 13

Patient-history ablation

• Adding more history raises prolonged-stay AUROC from 0.70 to 0.72 overall and from 0.63 to 0.68 for elderly patients.The model performs better on younger subjects across all history extents, while history inclusion matters significantly in the zero-shot setting.Source: Figure 6 · p. 12, NOAH relies on time and specializes in… · p. 13

• Predictive embeddings support lightweight clinical-information retrieval and time-to-event modeling, while autoregressive forecasting offers optional time control for likely future clinical events.NOAH also estimates clinical outcomes zero-shot and simulates treatment effects through counterfactual interventions.Source: Discussion · p. 14
• NOAH provides continuous, holistic patient-state representations across modalities and time, with semantically meaningful trajectories that can be extended task-agnostically into future developments.Its variational latent space supports inference-time variability while preserving semantic coherence across generated-event components.Source: Discussion · p. 14
• NOAH’s autoregressive rollout suffers exposure bias because errors from re-encoded predictions can accumulate over long forecasting horizons.The authors suggest larger reliable training data and post-training techniques may mitigate this limitation.Source: Discussion · p. 14
• NOAH learns predictive features rather than patient-history summaries, and its counterfactual outputs are probabilistic intervention-conditioned forecasts rather than causal or physiologically grounded simulations.The reported applications cover broad tasks, but experiments focus on selected feasible use cases and require deeper future evaluation.Source: Discussion · p. 14
• As a medical foundation architecture, NOAH can act as a universal patient-state encoder and prediction engine for lifelong records, supporting simulation, risk prediction, and interaction with patient history.The authors propose that this could improve personalized treatment and digital medicine by making multimodal health data more accessible.Source: Discussion · p. 14

Dataset and preprocessing

• NOAH merges six MIMIC datasets into patient-level lifetime timelines of timestamped multimodal events, using leakage-controlled splits, windowed training, and pretrained modality embeddings.Numeric vitals are aggregated by sliding-window minima and maxima, while images, text, ECG, and echocardiography use external encoders; training, validation, and test patients are disjoint under a 70/15/15 split.Source: Dataset and preprocessing · pp. 15–16

Model and pretraining

• A shared context-conditional latent variable coordinates stochastic predictions across event components, with KL-regularized posterior training and prior sampling at inference.The prior depends only on history, while the posterior also observes the current event; temperature τ controls trajectory variability, reflecting concentrated or broader samples.Source: Model and pretraining · pp. 16, 18, 19
• The model maps heterogeneous clinical events into shared tokens and combines input time features with asymmetric query-key temporal signals for irregular patient trajectories.Queries use forward time gaps, whereas keys use backward gaps, alongside ordinal position and patient age; end tokens mark the physically charted termination of ongoing events to avoid leakage.Source: Model and pretraining · pp. 17–18
• Hierarchical multimodal decoding conditions type specifics, modality, values, and timing on preceding event components, while gated time prediction and constrained modality choices preserve plausible autoregressive events.The pretraining objective combines event reconstruction, posterior-prior KL divergence, and prior-mode reconstruction, with bounded prior uncertainty preventing deterministic pass-through and variance inflation.Source: Model and pretraining · pp. 20–21

Evaluation framework

• The evaluation framework compares free and time-controlled forecasting across horizons, estimates zero-shot clinical risks from Monte Carlo rollouts, and simulates SMART-like fluid interventions using factual and substituted prompts.Forecasting uses 50 rollouts and reports macro AUROC, macro Brier, and macro count error; zero-shot tasks include prolonged stay, mortality, and readmission, while counterfactual analysis evaluates 1,085 eligible sepsis samples with 25 rollouts per arm.Source: Evaluation framework · pp. 21–23
• Counterfactual intervention simulation compares paired fluid-treatment rollouts with shared random numbers and identical latent noise using mortality, length of stay, and MAKE-30 probabilities.Average treatment effects are computed as between-arm differences averaged across patients, with chain-UMAP and NEWS2 scoring used to inspect simulated trajectories.Source: Evaluation framework · p. 23
• NOAH embeddings are linearly probed across longitudinal positions for hospital outcomes, 15 ICD chapters, and 29 Quan-Elixhauser comorbidities.Probes use ℓ2-regularized logistic regression with repeated stratified five-fold cross-validation and report AUROC with accuracy as a secondary metric.Source: Evaluation framework · pp. 23–24
• NEWS2 retrieval uses an independently trained MLP to map patient-state embeddings to scores computed from five-vital windows, including generated rollout states for simulated risk trajectories.The evaluation includes 1.83 million NEWS2 windows, patient-level 70/15/15 splits, multiple regression and classification metrics, bootstrap confidence intervals, and trivial, linear, and age-only baselines.Source: Evaluation framework · p. 24
• Survival analysis applies DeepSurv to patient-stay embeddings across six within-stay context fractions from admission through the last event before death or discharge.The cohort includes 26,476 patients meeting event and stay-duration criteria, with nested five-fold cross-validation and time-dependent C-index, D-Calibration, and IBS evaluation.Source: Evaluation framework · p. 25
• Marginal calibration is assessed by discretizing predicted survival probabilities into 20 intervals and testing uniformity, with p-values below 0.05 indicating miscalibration.Distribution Calibration is reported for each fold alongside survival discrimination and calibration metrics.Source: Evaluation framework · p. 25

Extended Data Figures and Extended Data Tables

• Extended figures illustrate zero-shot mortality, prolonged-stay, and readmission rollouts, identify time as the dominant attribution component, and describe within-stay event-rate, timing, variety, and cohort-characteristic patterns.Attribution varies by event: value matters especially when event type alone is clinically insufficient, while time and specifics usually dominate; event collection is highest at admission, declines, and rebounds near discharge.Source: Figure 1 · p. 30, Figure 4 · p. 33, Figure 5 · p. 34

Extended Data Figures

• Linear probes show that NOAH’s frozen patient states support time-indexed prediction and retrieval of ICD chapters and comorbidities through a hospital stay.Cross-validated AUROC is reported from pre-admission through discharge, with earlier probes mixing prediction and retrieval and uncertainty summarized by 2.5–97.5th percentile ranges.Source: Figure 2 · p. 31
• NOAH’s patient-state embeddings read out NEWS2 risk zones well over time, with substantial deviations mainly in the high-risk zone.An MLP is evaluated on held-out windows where all required vital signs are available.Source: Figure 3 · p. 32

Extended Data Tables

• NOAH forecasts future event-type and modality occurrences across four horizons using free and time-controlled Monte Carlo rollouts, compared with persistence and marginal baselines.The evaluation uses 50 rollouts, 13,551 patients, macro-averaged AUROC and Brier score, and 95% patient-bootstrap confidence intervals.Source: Table 1 · p. 35
• NOAH estimates clinical outcome probabilities zero-shot from target-event occurrences across Monte Carlo rollouts without task-specific training.The table reports point estimates, 95% patient-level bootstrap confidence intervals, and rollout coverage, with task configurations provided separately.Source: Table 2 · p. 35
• Counterfactual simulations compare saline and balanced crystalloid for 30-day in-hospital mortality and length of stay, using shared patient prompts and matched factual treatments.The analysis includes 1,085 patients, paired contrasts, scenario-specific estimands, bootstrap intervals, and the SMART sepsis subgroup as a reference [31].Source: Table 3 · p. 36

Latent surprise

• Imaging, waveform, and terminal events have the greatest latent surprise, whereas routine measurements are near zero, indicating lower predictability from prior patient history.Supplementary Table 1 reports median, q95, and mean KL(q∥p) by event type on the test set.Source: Table 1 · p. 38

Forecasting

• At H=24 h, forecasting performance is reported across length-of-stay quartiles using macro-averaged event-type AUROC and Brier score over 13,551 patients.The table provides 95% patient-bootstrap confidence intervals from 1,000 resamples and identifies the best model per column.Source: Table 2 · p. 39
• The forecasting supplement further ranks the 10 best- and worst-predicted event types and compares free versus time-controlled AUROC together with count MAE.These analyses use event prevalence and identify the best rollout variant at the H=24 h horizon over 13,551 patients.Source: Table 3 · p. 39

Counterfactual simulation

• Counterfactual analyses assess calibration, outcome reachability, invalid-run ratios, and simulated saline-versus-Ringer’s effects on MAKE-30 against SMART trial references.The mortality calibration analysis covers 1,085 uncensored patients and reports overprediction for this subcohort; MAKE-30 comprises death, new renal-replacement therapy, or creatinine at least twice baseline within 30 days.Source: Table 4 · p. 40, Table 5 · p. 40, Table 6 · p. 40

Linear probing

• Linear probes evaluate whether frozen patient-state embeddings encode clinical outcomes, ICD chapters, and Quan-Elixhauser comorbidities across admission, intermediate, and discharge states.Outcome probes distinguish predictive read-outs from discharge retrieval, while ICD and comorbidity probes report AUROC with cross-validation and restrict +48 h analyses to stays reaching that timepoint.Source: Table 7 · p. 41, Table 8 · p. 42, Table 9 · p. 43
• Final-state retrieval is examined across age and length-of-stay subgroups, with performance generally higher for younger patients and shorter stays.Supplementary Table 10 reports subgroup AUROCs, while Supplementary Figure 1 visualizes the same split analysis.Source: Table 10 · p. 44, Figure 1 · p. 46

NEWS2 and time-to-event prediction

• Additional evaluations recover NEWS2 from embeddings and assess within-stay time-to-event prediction using regression, clinical-band classification, td-CI, IBS, and D-calibration.NEWS2 evaluation includes 39,952 windows from 4,414 patients with mean ground-truth NEWS2 of 4.25; uncertainty conventions include patient-resampled 95% intervals.Source: Table 11 · p. 47, Supplementary Information · p. 47, Table 12 · p. 48

Implementation and dataset details

• The supplement records modality event volumes, value-embedding dimensions, survival hyperparameter search space, and NOAH’s pretraining configuration on 8× NVIDIA A100 80GB GPUs.These materials document dataset coverage and implementation settings rather than additional outcome comparisons.Source: Table 14 · p. 49, Table 15 · p. 49, Table 16 · p. 50