TRACER, a time-to-event EHR foundation model for disease detection across health systems

• EHR histories can contain disease signals years before diagnosis, and foundation models learn patterns from structured clinical events to estimate future disease risk.Source: Introduction · pp. 1–2
• Screening models must identify future cases while limiting false alerts, since frequent reminders are associated with alert fatigue and perceived alert workload with burnout dimensions.Source: Introduction · p. 2
• AUROC and C-index do not quantify cases detected or patients flagged at tolerable false-positive rates, and high specificity can still accompany low PPV for rare diseases.One cited evaluation of rare-disease algorithms reported median specificity of 99.9966% but adjusted median PPV of 3.1%.Source: Introduction · p. 2
• Evidence remains limited on detection at clinically relevant false-positive rates across multiple diseases, horizons, and health systems excluded from both pretraining and model development.Source: Introduction · p. 2
• TRACER is a decoder-based foundation model pretrained on more than 4.5 million patients from four UC systems and evaluated on 16 diseases in two systems held out from development.The evaluation examined 1-, 2-, and 5-year horizons, with cancers additionally evaluated at 6 months, and measured detection, PPV, and case yield under specified false-positive constraints.Source: Introduction · pp. 2–3

1.1 Study design and evaluation framework

• The evaluation covered 16 diseases across 1-, 2-, and 5-year horizons, with cancers also assessed at 6 months, using censoring-adjusted metrics and cohort-specific specificity thresholds.UC5 and UC6 were excluded from pretraining and model development; thresholds targeted 90%, 95%, and 99% specificity.Source: §1.1 · p. 3

1.2 Overall discrimination was largely retained across health systems

• The median reduction in C-index from in-domain to the lower external estimate was 0.041 (range 0.007–0.095), with losses varying by disease.Source: §1.2 · p. 4
• At five years, AUROC remained within 0.025 of its one-year value for seven of 12 chronic diseases and ovarian cancer, while other diseases showed larger or non-monotonic changes.For example, depression declined from 0.739 at one year to 0.688 at five years, while ovarian cancer rose at two years before declining at five years.Source: §1.2 · p. 4

1.3 Disease detection at low false-positive rates

• At 95% specificity, heart-failure sensitivity was 44.2% in-domain despite an in-domain one-year AUROC of 0.87, and sensitivity fell as the specificity requirement increased.Source: §1.3 · p. 4
• External sensitivity was lower in 59 of 104 comparisons (56.7%), with a median change of −1.9 percentage points; at one year and 95% specificity, chronic-kidney-disease sensitivity was 46.8% in-domain and 48.7% and 48.4% externally, while heart-failure sensitivity fell from 44.2% to 36.9% and 33.6%.Source: §1.3 · p. 5
• At one year and 95% specificity, rare-cancer sensitivity ranged from 12.3% to 44.6% in-domain and 14.9% to 56.5% across external evaluations.At 99% specificity, the corresponding external ranges fell to 4.0–15.7% for pancreatic cancer, 2.8–8.3% for ovarian cancer, 9.3–25.1% for glioblastoma, and 7.9–34.1% for multiple myeloma.Source: §1.3 · p. 5

1.4 PPV and case yield quantified potential review burden

• Heart failure and pancreatic cancer had similar selection rates (5.2–5.7% and 5.4–5.6%), yet pancreatic cancer’s in-domain PPV was 0.71% versus heart failure’s 13.80%.Their enrichment was comparable in two cohorts, illustrating that similar selection and relative enrichment can accompany very different absolute PPVs.Source: §1.4 · p. 6

1.5 Risk signals are captured with different model architectures

• TRACER generally had better chronic-disease AUROC than TRACER-Encoder, but AUROC and partial AUROC favored opposite models with statistical support in 15 of 144 comparisons, all external.In 14 of those comparisons, AUROC favored TRACER while partial AUROC favored TRACER-Encoder.Source: §1.5 · pp. 6–7

Discussion

• At 95% specificity, heart-failure flags selected 5.2–5.7% of patients and identified 33.6–44.2% of future cases, with 7.8-fold in-domain enrichment.For pancreatic cancer, a similar 5.4–5.6% selection rate identified 16.4–35.5% of future cases, with 6.8-fold in-domain enrichment; the in-domain PPVs were 13.8% and 0.71%, respectively.Source: Discussion · p. 7
• A median external C-index reduction of 0.041 coexisted with disease- and health-system-specific variation in sensitivity, PPV, and case yield at operating points.Global discrimination stability did not signal these operating-point changes.Source: Discussion · p. 7
• Heart-failure sensitivity at 95% specificity fell by 10.3 percentage points at UC6 despite a five-year AUROC change from 0.862 in-domain to 0.836.At one year, UC6 heart-failure PPV rose from 13.8% to 16.4% as its base rate rose from 1.78% to 2.84%; enrichment fell from 7.8-fold to 5.8-fold.Source: Discussion · pp. 7–8
• Pancreatic-cancer PPV fell from 0.71% in-domain to 0.29% at UC5 while enrichment fell from 6.8-fold to 3.3-fold, with nearly unchanged base rate.By contrast, one-year chronic-kidney-disease sensitivity and 8.0–8.5-fold enrichment were maintained at both external systems.Source: Discussion · p. 8
• Performance across prediction horizons varied by disease, so near-term results should not be extrapolated to other horizons; analyses did not identify a clear cause for external shifts.Differences in prediagnostic recording were offered as one possible explanation.Source: Discussion · p. 8
• A 5.5% selection rate under an illustrative scenario would generate about 2.1 messages weekly per physician, or 1.9% of 114 weekly system-generated messages.The estimate assumes 2,000 patients assessed annually for one disease and one initial notification per selected patient; it excludes follow-up and repeated or overlapping alerts.Source: Discussion · p. 8
• TRACER used structured EHR events without free text, imaging, pathology, genomics, or other modalities, and the UC warehouse omitted non-UC encounter data.Diagnosis-based onset reflects documented rather than biological onset, and prediction visits lacked an explicit washout period, so short-horizon performance may partly reflect diagnostic work-up.Source: Discussion · p. 8
• Matched pretraining did not isolate architecture, objective, pooling, optimization, or their interactions; prospective evaluation is needed to assess threshold transport, calibration, costs, and patient benefit.The study also did not measure clinician-level message exposure, responses, or burnout.Source: Discussion · pp. 8–9

2.1 Study Setting and Data Sources

• UC5 and UC6 were held out from both pretraining and survival-head training and used only for external validation.All six centers shared a common data structure, but had distinct patient populations, EHRs, and independent operations.Source: §2.1 · p. 9
• Four centers (UC1–UC4) supplied the in-domain data; 4,514,400 sequences were used for pretraining and 501,307 non-overlapping sequences for downstream development and evaluation.The downstream partition was divided into training, validation, and test sets.Source: §2.1 · p. 9

2.2 Data Representation

• Patient records were encoded as temporally ordered event sequences with visit boundaries, discretized inter-visit time gaps, and care-site tokens.Diagnoses, procedures, medications, and laboratory tests were mapped to standard vocabularies; laboratory values were discretized using pretraining-corpus quantiles applied unchanged to downstream cohorts.Source: §2.2 · pp. 9–10
• For downstream time-to-event modeling, frozen backbone representations were computed from final-layer hidden states and mean-pooled across eligible clinical and structural tokens.Padding and model-control tokens without patient-specific clinical or temporal information were excluded from pooling.Source: §2.2 · p. 10

2.3 TRACER Foundation Model Implementations

• The approximately 125-million-parameter models shared the pretraining corpus, tokenizer, sequence construction, model scale, and downstream survival protocol, while differing in architecture, attention, and objective.TRACER-Encoder used a bidirectional ModernBERT-style encoder with masked-token prediction; TRACER used a LLaMA-style causal decoder with next-token prediction.Source: §2.3 · p. 10

2.3.1 TRACER

• TRACER’s causal self-attention restricts each position to preceding tokens, and its next-token objective predicts subsequent clinical events from that context.Unlike the encoder objective, the decoder uses only temporally preceding sequence context to learn each conditional distribution.Source: §2.3.1 · pp. 10–11

2.3.2 TRACER-Encoder

• TRACER-Encoder reconstructs sampled masked tokens, with bidirectional attention allowing each prediction to use unmasked events before and after the masked position.The encoder was ModernBERT-style and trained with masked-token prediction.Source: §2.3.2 · p. 11

2.4 Disease endpoints and cohort construction

• The 16 endpoints comprised 12 chronic conditions and four rare cancers; ovarian cancer cohorts included female patients only, and natural outcome frequencies were retained.No endpoint used case-control sampling.Source: §2.4 · p. 11
• Disease onset was the first visit containing a concept from the endpoint’s prespecified SNOMED CT set, while patients without an endpoint code were right-censored at their final observed visit..Source: §2.4 · p. 11
• Eligible patients had at least one year of history before the event or final observed visit, and one prediction visit was sampled strictly after that first year and before onset or censoring.The sampled visit defined the landmark but was excluded from model input; each patient contributed one example per endpoint.Source: §2.4 · pp. 11–12

2.5 Downstream discrete-time survival modeling

• For each endpoint, a single-risk DeepHit head mapped frozen, pooled patient representations to a discrete event-time probability distribution.The head was a bias-free linear layer with no hidden layers, and its K interval logits were augmented by a fixed zero logit before softmax normalization.Source: §2.5 · p. 12
• Chronic-disease follow-up used boundaries at 52, 104, and 260 weeks, while cancer endpoints also included a 26-week boundary.The distribution included a terminal component for residual survival mass beyond the final evaluated interval.Source: §2.5 · p. 12
• The discrete-time negative log-likelihood assigned event cases probability mass at their observed event interval and censored cases probability of surviving beyond their discretized censoring time.Including the terminal component retained positive probability for survival beyond 260 weeks for patients censored at the final evaluated boundary.Source: §2.5 · p. 12
• A pairwise ranking term penalized predictions that failed to assign higher cumulative risk to earlier observed events, while noncomparable pairs contributed zero.Survival heads used the combined DeepHit loss and were optimized with Adam, learning rate 10^-3, batch size 256, and validation-loss early stopping; trained heads were applied unchanged to UC5 and UC6.Source: §2.5 · pp. 12–13

2.6.1 Global discrimination and probabilistic performance

• Uno’s IPCW C-index and cumulative/dynamic AUROC assess risk ordering, while IBS and IBLL assess probabilistic performance.Higher C-index values indicate better risk ordering; lower IBS and IBLL values indicate better probabilistic performance.Source: §2.6.1 · pp. 13–14

2.6.2 Low-false-positive discrimination and fixed operating points

• Partial AUROC over FPR 0–0.10 targets specificity from 90% to 100%, with standardized values of 0.5 at chance and 1.0 at perfect discrimination in that region.Global AUROC also includes false-positive rates that may be irrelevant for screening or clinical alerting.Source: §2.6.2 · p. 14
• Sensitivity is measured among horizon-specific cases at thresholds chosen to target 90%, 95%, or 99% specificity.Thresholds come from the control-score distribution at each horizon.Source: §2.6.2 · p. 14

2.6.3 Positive predictive value, case yield, and fold enrichment

• Time-dependent PPV measures the share of above-threshold records diagnosed by the horizon, while case yield estimates future cases per 10,000 scored records and fold enrichment compares PPV with baseline incidence.These measures use the same cohort-specific thresholds targeting 90%, 95%, and 99% specificity.Source: §2.6.3 · pp. 14–15

2.7 Statistical analysis

• For most analyses, 95% confidence intervals used 1,000 patient-level bootstrap replicates, with fixed-specificity thresholds re-estimated within each replicate.Censoring weights were estimated once per evaluation cell and carried through resampling as fixed patient-level weights.Source: §2.7 · p. 15
• For 52-week sensitivity and PPV, confidence intervals used bootstrap percentiles; case-yield bounds scaled sensitivity bounds while holding the weighted case pool and denominator fixed.Baseline incidence and fold enrichment were reported as point estimates without confidence intervals.Source: §2.7 · p. 15
• Head-to-head comparisons used the same resampled patients for TRACER and TRACER-Encoder, defining differences as TRACER minus TRACER-Encoder.A significant AUROC-pAUC ranking disagreement required opposite-signed point differences and paired 95% confidence intervals excluding zero for both metrics.Source: §2.7 · p. 15
• The primary comparison covered 144 endpoint-site-horizon evaluations, with four cancers adding 12 evaluations at 26 weeks.When tied risk scores equaled a fixed-specificity threshold, they crossed it jointly, so achieved specificity could differ slightly from the nominal target.Source: §2.7 · p. 15

2.8 Secondary analyses and ablations

• Supplementary analyses covered downstream labeled-data availability, local survival-head training at UC5 and UC6, and a Cox proportional-hazards survival-head ablation.These analyses are described in the Supplementary Methods rather than this section.Source: §2.8 · pp. 15–16
• Other supplementary analyses examined AUROC-pAUC ranking reversals, token-level attribution, pooling ablations, and health-system record differences.The passage lists these analyses without reporting their findings.Source: §2.8 · p. 16

Supplementary Material

• The paper is titled as a time-to-event EHR foundation model for disease detection across health systems.The supplied passage contains the title only.Source: Supplementary Material · p. 28

S1 Tables supporting the main text

• Table S1 reports observed endpoint-event frequencies across development and held-out cohorts, with rows ordered by in-domain training frequency within endpoint groups.Ovarian cancer cohorts are restricted to female patients.Source: Table S1 · p. 29
• Tables S2 and S5 report C-index discrimination across in-domain and held-out cohorts, including TRACER-versus-Encoder comparisons and TRACER’s worst-site gap.The tables describe estimates and confidence intervals, but the supplied passages do not include their values.Source: Table S2 · p. 29, Table S5 · p. 32
• Tables S3, S6, and S7 cover paired 52-week AUROC differences and AUROC across horizons and evaluation cohorts.The supplied passages describe table scope, not the corresponding estimates.Source: Table S3 · p. 30, Table S6 · p. 32, Table S7 · p. 33
• Table S4 identifies which model has higher pAUC at FPR ≤0.10 and where global AUROC favors the same or a different model.Bold cells mark significant reversals; plain and italic cells indicate agreement and nominal disagreement, respectively.Source: Table S4 · p. 31
• At 26 weeks, the additional cancer evaluations contained five nominal disagreements and no strict reversals.A strict reversal is an opposite-direction result, with TRACER-Encoder favored by AUROC and TRACER by pAUC.Source: S1 Tables supporting the main text · p. 31
• Tables S8–S11 report sensitivity and disease detection at fixed-specificity thresholds, including one-year PPV, fold enrichment, and case yield.The tables use cohort-specific thresholds; the supplied passages do not give the outcome values.Source: Table S8 · p. 34, Table S9 · p. 34, Table S10 · p. 34

S2 Additional supporting tables; S2.1 Additional comparative clinical results; S2.2 Labeled-data, health-system and attribution summaries

• Supplementary tables report pooling and survival-head comparisons, low-false-positive discrimination, case recovery, site-shift characteristics, and attribution summaries.Their captions specify the compared protocols and measures, including paired partial-AUROC differences and table-specific cohort or training-fraction scope.Source: Table S12 · p. 39, Table S13 · p. 39, Table S17 · p. 43

S3.1 Model preference at overall and low-false-positive discrimination

• 15 of 144 comparisons (10.4%) were strict reversals, and point-estimate rankings differed in 37 of 144 (25.7%).TRACER had higher estimated AUROC in 106 of 108 chronic-disease comparisons; at UC5, TRACER-Encoder had higher estimated partial AUROC in 16 of 36 disease comparisons.Source: S3.1 Model preference at overall and… · pp. 49–50

S3.2 Case recovery in the strict-reversal comparisons

• In 14 reversal comparisons, TRACER-Encoder recovered a median 6.9 additional cases per 10,000 patients at 95% specificity.Within the 15 strict reversals, point-estimate sensitivity favored TRACER-Encoder in 15, 14, and 13 comparisons at 90%, 95%, and 99% specificity, respectively.Source: S3.2 Case recovery in the… · p. 50

S3.3 Discrimination across health systems and prediction horizons

• Horizon trajectories varied by endpoint and architecture, while integrated Brier score and log-likelihood were nearly identical between models.For example, rheumatoid arthritis AUROC rose by 0.10 under TRACER-Encoder but remained nearly unchanged under TRACER; ovarian cancer AUROC declined by 0.094 versus 0.013, respectively.Source: S3.3 Discrimination across health… · p. 50

S3.4 Downstream labeled-data availability; S3.5 Differences in health-system records

• Performance generally continued to change as more in-domain labels were added, and external-cohort improvements sometimes occurred despite small in-domain gains.External records differed in density and follow-up; among reversal-associated comparisons, the largest cumulative-incidence differences from in-domain were hyperlipidemia at UC5 and heart failure at UC6.Source: S3.4 Downstream labeled-data… · p. 50, S3.5 Differences in health-system… · p. 51

S3.6 Interpretability analyses

• Probe analyses found only partial reproduction of the reversal surface, with endpoint composition contributing more uncertainty than random head initialization.The residual-only analysis opposed its prespecified hypothesis; attribution analyses localized candidate disagreement sources but did not identify why particular disease-horizon-site cells reversed.Source: S3.6 Interpretability analyses · pp. 51–52

S3.7 Pooling-mask and pooling-strategy ablations

• Across 48 comparisons, excluding care-site and time-gap tokens or applying smooth inverse-frequency weighting did not consistently improve C-index.The primary representation extraction used uniform mean pooling with care-site and time-gap tokens included.Source: S3.7 Pooling-mask and pooling-strategy… · p. 52

S3.8 Local survival-head training versus zero-shot cross-site transfer; S3.9 DeepHit versus Cox proportional-hazards survival heads

• Local survival-head training improved C-index at UC5 for all 12 chronic diseases in both models, but gains were smaller and more heterogeneous at UC6.DeepHit exceeded CoxPH in all 72 endpoint-representation-cohort comparisons; the authors caution that this does not establish general superiority over CoxPH.Source: S3.8 Local survival-head training… · pp. 52–53, S3.9 DeepHit versus Cox… · p. 53

TRACER-Encoder Pretraining

• TRACER-Encoder was trained from scratch as a ModernBERT masked-language model, using hybrid local-global attention and sequences up to 2,048 tokens.Training used a 5×10^9-token budget.Source: TRACER-Encoder Pretraining · pp. 53–54

TRACER Causal Pretraining

• TRACER was trained from scratch as a LLaMA-based causal decoder with 14 layers, 12 query heads, and sequences up to 2,048 tokens.It used grouped-query attention with four key-value heads and a 5 × 10^9-token budget.Source: TRACER Causal Pretraining · p. 54

Resources

• Both models were pretrained on eight NVIDIA A100 GPUs.Source: Resources · p. 54

S4.2 Downstream labeled-data availability

• Downstream labeled-data availability was tested by training survival heads on 10%, 30%, 50%, or 100% of the in-domain training set while keeping test and validation sets fixed.The smaller training fractions were supplementary robustness analyses, not independent tests.Source: S4.2 Downstream labeled-data… · p. 54

S4.3 Local survival-head training; S4.4 Cox proportional-hazards survival head

• Local adaptation trained frozen-backbone DeepHit heads on each external system’s local training partition, while a separate ablation fitted a linear Cox model to frozen representations.Source: S4.3 Local survival-head training · p. 55, S4.4 Cox proportional-hazards survival… · p. 55

S4.5 Pooling-mask and pooling-strategy ablations

• Pooling ablations tested excluding care-site and time-gap embeddings and using smooth inverse-frequency weighting, which reduces common-token contributions and increases rare-token contributions.Care-site and time-gap tokens remained in transformer inputs, so exclusion tested their direct contribution to pooling rather than removing their contextual influence.Source: S4.5 Pooling-mask and pooling-strategy… · p. 55

S4.6 Analyses of AUROC-pAUC ranking reversals

• Ranking-reversal analyses used equalized and residual-only probes, repeated DeepHit head initialization, and resampling to examine representation, head-fitting, and evaluation variability.The probes used frozen representations; the residual-only analysis tested components not linearly predictable from the other representation, while controlled retraining assessed recurrence across 10 × 10 head pairings.Source: S4.6 Analyses of AUROC-pAUC ranking… · pp. 55–56

Token-level Attribution; Site-shift characterization

• Integrated gradients attributed between-model predicted-risk differences, while site-shift analyses measured token-distribution divergence, record density, and follow-up characteristics.Attribution shares were normalized within sequence; site-level comparisons included Jensen-Shannon divergence and sequence-length and unique-token summaries.Source: Token-level Attribution · p. 56, Site-shift characterization · p. 57, Figure S1 · p. 58