PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology

Eugene Vorontsov, Yi Kan Wang, Alican Bozkurt, Adam Casson, Ludmila Tydlitatova, Michal Zelechowski, Ezra E. W. Cohen, Jyoti D. Patel, Max Banaszak, Caitlin McWilliams, Shane Colley, Kate Sasser, Ryan Fukushima, Eric Lefkofsky, Razik Yousfi, Siqi Liu

arXiv:2608.24688v1cs.LG

TL;DR

Precision oncology needs patient representations that capture evolving cancer states, treatments, and multimodal biology rather than relying on static snapshots. The oFM integrates longitudinal clinical and molecular episodes with H&E pathology into patient embeddings, which outperform curated baselines across prognostic and comparative-treatment benchmarks and support mechanism discovery.

  • Problem

    Static clinical systems and separate modality encoders do not adequately represent cancer evolution across time, interventions, and multimodal observations for prognostic and treatment-benefit biomarkers.

  • Method

    The oFM hierarchically encodes daily clinical and molecular episodes with pathology embeddings and integrates them longitudinally using a multimodal Transformer.

  • Results

    Across prognostic and comparative-treatment benchmarks, frozen oFM embeddings consistently outperformed expert-curated clinical and molecular baselines, including 0.774 versus 0.563 mean AUC for overall survival and benefit-ranking wins in 9 of 11 cohorts.

  • Takeaways & Limitations

    The oFM provides longitudinal multimodal representations for prognostic modeling, treatment-benefit prediction, and interpretable mechanism discovery through evidence-grounded temporal relationships.

Abstract

from arXiv · show

Precision oncology necessitates a longitudinal model of patient state that captures cancer evolution and treatment over time, integrating multimodal observations. We introduce the oFM, a foundation model developed on a real-world oncology cohort of 1.67 million cancer patients that integrates clinical trajectories with DNA, RNA, and H&E pathology. Patient-level partitions were reserved for training, validation, and testing, with over one million patients used for training. The oFM encodes daily clinical and molecular episodes and, along with pathology images, integrates them over time to produce a patient state embedding. We evaluate frozen oFM embeddings against expert-curated clinical and molecular baseline features. In prognostic benchmarks, the oFM improved AUC for treatment response, progression-free survival, and overall survival (0.774 vs. 0.563 for overall survival). Across 11 comparative-treatment cohorts, the oFM embeddings achieved a three-fold higher pooled and scale-normalized treatment-benefit AUTOC than baseline features with improved benefit ranking in 9 of 11 cohorts, and provided stronger prognostic discrimination within both treatment arms. We also evaluated a mechanism discovery framework that interprets downstream models built on oFM embeddings by linking their predicted outcomes to clinically and biologically grounded mechanisms through an evidence-grounded temporal graph, enabling evaluation in clinical and drug-development applications.

1 Introduction

Precision oncology requires models that represent changing patient states and treatment effects over time, because static clinical systems and modality-specific encoders miss longitudinal multimodal evolution. The oFM addresses this gap by integrating longitudinal records, molecular profiles, and pathology embeddings for prognostic and treatment-benefit modeling.

  • Static decision systems use clinical variables and biomarkers as snapshots, overlooking dynamic cancer behavior linked to metastasis, response, resistance, and toxicity.
  • The oFM integrates longitudinal clinical records, DNA and RNA biomarkers, and H&E pathology embeddings into patient trajectory representations.
  • The model supports prognostic prediction, treatment-benefit prediction, biomarker discovery, and mechanism discovery through attributed clinical and molecular features.

2 Related Work

Prior multimodal cancer models generally combine clinical, pathology, and molecular data at a single time point, while longitudinal EHR models focus mainly on structured records or trajectory objectives. The oFM extends this landscape by integrating multimodal observations longitudinally and using modular clinical-text and pathology encoders.

  • Single-time-point models such as MultiSurv, MCAT, and PORPOISE integrate clinical, histopathologic, and molecular data without modeling changing patient trajectories.
  • Longitudinal EHR foundation models use masked language modeling, time encoding, autoregressive objectives, time-to-event objectives, or future latent-state prediction.
  • APOLLO combines unstructured clinical text and medical image embeddings across patient trajectories, whereas the oFM uses treatment-conditioned future prediction.
  • The oFM uses GatorTron for clinical and molecular text and PRISM2 for slide-level H&E embeddings, allowing continued improvement of modality encoders.
  • THREADS, mSTAR, and OmniScreen combine pathology with molecular signals or predict DNA biomarkers from H&E, but operate at a single time point.
  • Genomic and single-cell foundation models provide reusable molecular representations, while the oFM integrates molecular inputs over patient trajectories.

3 The oFM Model Architecture

The oFM is a multimodal Transformer that converts daily clinical and molecular episodes plus pathology embeddings into a longitudinal patient state representation. Its hierarchical encoders preserve episode content, pathology information, modality identity, and relative time.

  • The oFM produces a 1,024-dimensional patient embedding h(t∗) by integrating clinical and molecular episodes with pathology embeddings over time.
  • Episode representation: Episodes are daily collections of timestamped clinical facts and molecular measurements, including diagnoses, treatments, responses, progression, DNA alterations, RNA abnormalities, and biomarkers.
  • Episode representation: A finetuned GatorTron encoder embeds all clinical and molecular events as text while Fourier Number Embedding preserves measurement magnitude.
  • Pathology representation: H&E whole-slide images are converted into an 8,192-dimensional PRISM2 biospecimen vector and projected into the shared 1,024-dimensional trajectory space.
  • Trajectory integration: A multimodal trajectory Transformer combines episode and pathology embeddings using modality encodings and rotary positional embeddings for relative time.
  • Trajectory integration: The trajectory encoder’s output CLS token is the state embedding, and evaluating it at different anchor points enables longitudinal comparison.

4 Training the oFM

The oFM uses staged representation learning and anchor-conditioned student–teacher training to learn trajectory states without leaking intervention information. Training combines reconstruction, future latent prediction, variance and covariance regularization, and survival supervision.

  • Training curriculum: Stage I pretrains the episode encoder with token and numerical reconstruction, Stage II trains the trajectory components with the episode encoder frozen, and Stage III jointly fine-tunes them.
  • Self-supervision: Masked episode reconstruction independently masks episode and pathology embeddings, then reconstructs them with modality-specific heads using mean squared error and cosine losses.
  • Anchor-conditioned prediction: Anchor episodes mark interventions or transitions, and sampled anchor pairs define the context and future state used for outcome or patient-state prediction.
  • Anchor-conditioned prediction: The student receives corrupted history strictly before an intervention anchor, while the teacher encodes a later target view whose inclusion rules prevent intervention leakage.
  • Anchor-conditioned prediction: The residual predictor maps the student’s corrupted context state to the teacher’s future patient state, with elapsed time incorporated into the prediction.
  • Supervision and regularization: The trajectory objective combines reconstruction, future prediction, variance regularization, covariance regularization weighted 0.04, and Cox survival supervision.
  • Robustness: Pathology embeddings are randomly dropped with probability 0.3 during training to improve robustness to missing pathology data.

5 Data

The study uses a large, longitudinal multimodal oncology corpus organized around patient timelines, integrating clinical events, molecular findings, and H&E pathology over time.

  • Cohort and data: The cohort contains 1,672,203 patients from a de-identified multimodal real-world oncology corpus.Training used 1,045,011 patients; later stages used treatment-recorded and information-rich subsets.
  • Cohort and data: Patient records are longitudinal sequences anchored at the earliest observed event, with subsequent events indexed by elapsed calendar days.This preserves irregular clinical intervals in the patient timeline.
  • Multimodal data: The longitudinal record combines clinical events, molecular blueprints, and digitized H&E pathology.Clinical data include encounters, vitals, labs, diagnoses, treatments, procedures, and outcomes; molecular findings are recorded per biospecimen.
  • Multimodal data: Clinical information is organized into observations, interventions, and disease-state transitions such as metastasis, progression, response, and follow-up.This organization represents both measured patient characteristics and changes in disease state or treatment.

6 Downstream Evaluation

Downstream evaluation tests frozen oFM embeddings against a curated-feature baseline across prognostic and treatment-benefit tasks. The oFM shows stronger aggregate prognostic and predictive performance, including broader treatment-benefit ranking and per-arm risk discrimination.

  • Evaluation design: The benchmarks assess the relative quality of learned patient trajectory embeddings against 7,520 curated clinical and molecular features.Evaluation uses the two-stage convex UV-Cox method, with frozen embeddings and baseline features compared under downstream probes.
  • Prognostic classification: Prognostic probes evaluate treatment response, progression-free survival, and overall survival using embeddings extracted before first-line therapy.Test performance is stratified by tumor type × therapy class to reduce prognostic confounding.
  • Prognostic classification: 0.774 mean AUC for OS, 0.688 for PFS, and 0.585 for treatment response were achieved by oFM embeddings, outperforming baseline features on all three endpoints.The oFM beat baseline on 95% of OS and PFS strata and 78% of treatment-response strata.
  • Predictive benchmarks: treatment benefit: 4.61 tAUTOC/SD for oFM versus 1.38 for baseline showed significant treatment-benefit ranking, with oFM wins in 9 of 11 cohorts.The baseline value of 1.38 did not clear its permutation null of 1.85.
  • Predictive benchmarks: treatment benefit: oFM embeddings produced better prognostic risk ordering within treatment arms, outperforming baseline on 10 of 11 control-arm tasks and 9 of 11 experimental-arm tasks.Similar C-index gains across arms were interpreted as evidence of prognostic rather than treatment-leakage signal.

7 Mechanism Discovery

The mechanism-discovery framework interprets predictions from frozen oFM embeddings by identifying sparse, clinically meaningful features, testing their influence, and organizing them into evidence-grounded temporal graphs. Applied to comparative-treatment cohorts, it connects clinical and genomic events to predicted treatment outcomes.

  • Framework overview: The framework identifies sparse directions that differentiate high- and low-scored patients, map to clinical and biological concepts, alter predictions under steering, and connect concepts to outcomes through temporal precedence.It is applied per evaluation cohort to frozen oFM embeddings and fitted predictors.
  • Framework overview: Seven stages span fact attribution, sparse autoencoder discovery, differential activation and clustering, concept labelling, causal steering, retrieval-grounded reasoning, and temporal graph assembly.All stages operate on frozen embeddings without fine-tuning.
  • Mechanism graph: Temporal edges retain concept pairs co-occurring in at least n_min patients when the first precedes the second in more than 60% of cases, with one-sided sign testing and BH-FDR q ≤ 0.05.The temporal backbone is enriched with curated biological interactions and other evidence layers.
  • Mechanism graph: The sparse autoencoder discovers and tests latent concepts, while the Mechanism Graph orders them across patient histories and links them to outcomes through established or candidate biological pathways.Narrative chains are distilled from the temporal graph and ranked by path weight and terminal-concept diversity.
  • Treatment-benefit application: In a multi-tumor T-DXd comparison, low-benefit clusters involved compromised drug delivery, reduced target antigen availability, or aggressive multi-drug resistance phenotypes.Examples included brain metastases, low CNV burden, prior triple therapy, and lineage-associated GATA4 overexpression; separating confounding from response modification requires prospective work.
  • Treatment-benefit application: Steering feature clusters shifted predicted risk scores, supporting a directed framework that maps upstream clinical and genomic events to predicted patient outcomes.The graph was illustrated for later-line T-DXd versus physician’s choice chemotherapy.

8 Conclusion

The oFM combines longitudinal clinical history, molecular findings, and H&E pathology, while its mechanism-discovery framework links predictions to evidence-grounded clinical and biological factors. Retrospective evaluations compare representation quality rather than establish validated biomarkers, treatment effects, or definitive mechanisms.

  • The oFM integrates longitudinal clinical history with patient-linked DNA and RNA findings and H&E pathology embeddings.
  • Across prognostic and comparative-treatment benchmarks, frozen oFM embeddings consistently outperformed expert-curated clinical and molecular feature baselines.The authors suggest the model captures information not readily recovered from conventional static features.
  • The mechanism-discovery framework links downstream predictions to attributed clinical and molecular events, sparse latent features, and evidence-grounded temporal hypotheses.
  • Retrospective real-world evaluations compare representation quality and do not establish clinically validated biomarkers, treatment effects, or definitive biological mechanisms.

A.1 Data Sources

Patient trajectories were built from the Tempus Data Model, a multimodal oncology platform integrating clinical and molecular data from multiple sources.

  • The Tempus Data Model integrates clinical and molecular data from multiple sources to build patient trajectories.

Clinical Data

Clinical data combine expert-curated abstractions, native EHR feeds, pathology review, external mortality data, derived variables, and harmonized duplicate records.

  • Curated records include expert abstractions of diagnoses, treatments, response assessments, progression, and other clinical endpoints.
  • Native data come directly from partner institutions and include medications, laboratory results, vital signs, encounters, performance status, and smoking status.
  • Pathology data contain diagnosis and histology information from pathologist review of biopsy specimens.
  • Third-party claims-based mortality data augment vital-status and last-known-alive dates.
  • Derived variables include BMI from height and weight measurements and therapy lines derived from curated care-plan data.
  • When curated and native records capture the same concept, they are harmonized for completeness and date precision while retaining source provenance.The source is tracked with a row_source indicator.

Molecular Data

The molecular data comprise internally generated, external, and reference-laboratory results spanning genomic, transcriptomic, immune, pathology-linked, and clinical biomarker categories. Baseline evaluation uses standardized feature selection, cohort stratification, and treatment-benefit estimation procedures designed to limit leakage and account for observational bias.

  • Molecular sources: Molecular inputs include Tempus NGS results covering DNA variants, RNA expression, MSI, TMB, HRD, HLA, immune features, neoepitopes, and pathogen detection.The assays include xT, xF, xE, and RS platforms.
  • Molecular sources: External laboratory results add NGS, PCR, IHC, and FISH measurements, including hormone receptors, MSI, MMR, and gene-level alterations.
  • Molecular representation: The molecular blueprint integrates DNA, RNA, IHC, and ISH across ten categories, including biospecimen metadata, genomic instability, immune descriptors, pathogens, pathways, SNVs/indels, and copy-number alterations.Examples include TMB-H ≥10 mutations/Mb, HLA loss of heterozygosity, TIL phenotype, and disruption of mTOR, TGF-β, RTK, HRD, and DDR pathways.
  • Evaluation design: Unweighted macro-mean AUC is computed across valid tumor-type–therapy strata to test consistency beyond high-prevalence cohorts.Strata are excluded for zero embeddings or a single outcome class, and small strata can strongly affect the macro-mean.
  • Baseline construction: 7520 candidate baseline features undergo variance filtering, univariate screening, and multivariate penalized-Cox selection using training and validation data while withholding test data.Treatment-intent labels are excluded to prevent treatment-assignment leakage, and core clinical features are retained for cross-cohort comparability.
  • Treatment-benefit evaluation: Treatment-benefit evaluation uses AUTOC, which ranks patients by estimated benefit, with scale normalization and permutation-based significance testing across observational cohorts.Benefit estimates use pseudo-outcomes and doubly robust AIPW scores because individual treatment benefit is unobserved and outcomes may be right-censored.
Loading 2608.24688v1…