Source-linked AI summary
MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology
Paul Minchella, Stéphane Chrétien, Guillaume Metzler, Loïc Verlingue, Rémi Vaucher
TL;DR
Survival models often struggle to integrate heterogeneous, longitudinal clinical data. MultiSigBERT combines narrative reports and structured variables through path signatures and sparse Cox regression, achieving a C-index around 0.743 (sd 0.029) on a real-world oncology cohort.
Problem
Survival models often struggle to combine heterogeneous clinical data while representing irregular, asynchronous, and longitudinal patient trajectories.
Method
MultiSigBERT integrates longitudinal clinical narratives and structured variables into synchronized trajectories, encodes them with path signatures, and applies sparse Cox regression.
Results
0.743 (sd 0.029) C-index was achieved on a real-world oncology cohort under a landmark design, with stable calibration.
Takeaways & Limitations
The framework offers a robust and scalable approach to longitudinal multimodal survival analysis with a favorable balance of prediction, interpretability, and computational efficiency.
Takeaways & Limitations
The study uses limited structured variables, whose irregular sampling requires interpolation within the landmark window.
Abstract
from arXiv · showhide
Machine learning has become an essential component of modern healthcare, where the integration of heterogeneous data sources offers unprecedented opportunities to improve clinical decision-making. Electronic Health Records (EHR) contain complementary information -- including narrative clinical reports, numerical measurements, and structured variables -- yet most survival models remain limited to a single modality or fail to exploit the temporal nature of patient trajectories. We propose MultiSigBERT, a unified framework for multimodal sequential survival modeling in oncology based on path signature representations. Here, narrative medical reports (free-text) are converted into sentence embeddings by extracting and averaging contextual word embeddings. These representations are then compressed via modality-specific PCA and concatenated with structured covariates to form joint temporal trajectories which are then encoded using the Signature transform, a tool from Rough Paths theory that efficiently captures higher-order temporal interactions across modalities without supervision needed. The computed Signature features are finally incorporated as high dimensional features into a LASSO-regularized Cox model to estimate individualized risk scores. The performance of our novel MultiSigBERT pipeline is illustrated on the analysis of a real-world oncology cohort from the Léon Bérard Center, comprising over 120,000 medical reports and structured records from more than 2,500 patients. The model achieves a concordance index of 0.743 (sd 0.029) on an independent test set, demonstrating the benefit of jointly modeling multimodal temporal dynamics together with patient-level geometric structure for survival prediction.
1 Introduction
The introduction identifies the difficulty of integrating heterogeneous, longitudinal EHR data for survival prediction and presents MultiSigBERT as a unified multimodal sequential framework for oncology. It extends text-focused survival modeling by combining synchronized clinical modalities, landmark-based prediction, signature features, and sparse Cox regression.
- Motivation: Survival prediction is clinically important for treatment selection, trial inclusion, and follow-up planning, but classical models often use static covariates and overlook longitudinal follow-up.EHRs include structured variables, laboratory measurements, molecular data, and narrative reports whose integration remains challenging.
- Motivation: Unimodal methods fail to capture interactions between structured variables and narrative text that shape patient trajectories and influence prognosis.The introduction frames cross-modality integration as a central unresolved challenge in survival modeling.
- Related work: SigBERT achieved a concordance index of 0.75 (sd 0.014) and a time-dependent AUC of 0.80 (sd 0.029) using narrative reports from the Léon Bérard Center.The prior framework demonstrated that encoded clinical text can improve survival risk estimation but focused exclusively on narrative reports.
- Contributions: Its landmark-based pipeline restricts feature construction to observations available before prediction time, preventing temporal information leakage in dynamic risk prediction.This design addresses leakage concerns associated with constructing temporal features without a landmark formulation.
- Contributions: MultiSigBERT synchronizes OncoBERT sentence embeddings with linearly interpolated structured variables before concatenating them into multimodal temporal trajectories for signature transformation.The framework jointly represents unstructured clinical text, structured numerical variables, and their longitudinal evolution.
- Contributions: Signature features combined with sparse Cox regression provide scalable, interpretable survival modeling, while modality-specific comparisons quantify the added value of multimodal fusion.The authors describe the framework as the first to combine multimodal temporal encoding with sparse survival modeling in one computationally efficient pipeline.
2 Method
MultiSigBERT integrates longitudinal narrative reports and structured clinical variables into multimodal temporal trajectories for survival modeling. Landmark restriction, signature encoding, and LASSO-regularized Cox regression produce individualized risk scores from these heterogeneous records.
- Clinical data representation: MultiSigBERT represents each patient’s longitudinal trajectory using time-indexed narrative reports, structured covariates, and right-censored survival outcomes.Clinical notes primarily comprise consultation reports (68%) and hospitalization reports (27%), alongside variables including Weight, Karnofsky Index, Blood Pressure, and Pulse Rate.
- Temporal encoding: The pipeline restricts observations to a landmark window, concatenates modality-specific representations, and applies the signature transform to capture higher-order temporal interactions.PCA is applied to training embeddings because signature complexity grows exponentially with input dimension; the signature also handles irregular sampling and heterogeneous modalities.
- Text embedding: OncoBERT generates 768-dimensional contextual token embeddings, which SIF pooling averages into fixed-dimensional sentence representations for oncology reports.OncoBERT is CamemBERT-based and fine-tuned on oncology notes from the Léon Bérard Center; SIF uses smoothing parameter a = 10^-3.
- Text embedding: 0.74 vs. 0.70, SIF outperformed CLS on C-index values and also produced lower Brier scores in retrospective experiments.SIF reweights and averages token embeddings, mitigating frequent or uninformative tokens and accommodating long documents beyond BERT’s 512-token limitation.
- Survival estimation: Signature-derived features enter a LASSO-regularized Cox model whose sparse coefficients select prognostic predictors and yield individualized risk scores.For q input channels and truncation level L, signature coefficients scale as O(q^L); LASSO shrinks less relevant coefficients toward zero.
3 Experiments
Experiments evaluate MultiSigBERT under a 36-month landmark design using a heterogeneous oncology cohort and repeated test-set splits. MultiSigBERT achieves strong discrimination and calibration, with only marginal discrimination gains from adding sequential structured covariates.
- Experimental design: The landmark design uses L = 36 months and a backward window w = 12 months, considering patients remaining at risk three years after diagnosis or hospitalization.Covariates are constructed from the one-year period preceding the landmark.
- Experimental cohort: The cohort includes 2,527 patients and 124,049 clinical reports, with 1,629 events and 898 right-censored cases.Patients average 49 reports each, reflecting dense longitudinal follow-up.
- Predictive performance: In Reports Only, MultiSigBERT reaches a mean test C-index of 0.741 (sd 0.027), versus CoxTime (0.646), CTVF (0.619), and DeepSurv (0.600).The C-index assesses discrimination by measuring correctly ordered comparable patient pairs.
- Predictive performance: With sequential structured covariates, MultiSigBERT reaches a mean test C-index of 0.743 (sd 0.029) and IBS@3y = 0.134 (sd 0.007).The C-index gain is marginal, while calibration slightly worsens relative to the Reports Only configuration.
- Baseline comparison: CoxTime achieves the lowest IBS@3y at around 0.091, but its C-index is approximately 0.65, indicating a discrimination–calibration trade-off.MultiSigBERT remains substantially stronger in discrimination than DeepSurv, which reaches a mean C-index of only 0.650 with structured variables.
- Limitations: The limited benefit of sequential structured variables may reflect interpolation over irregular recording times, which introduces smoothing assumptions and can propagate measurement noise.The experiments suggest that longitudinal clinical narratives already capture much of the prognostic signal in routinely collected structured measurements.
4 Conclusion
MultiSigBERT unifies longitudinal clinical narratives and structured variables through multimodal temporal representations, path signatures, and sparse Cox regression for survival prediction. In oncology experiments, it achieved a C-index around 0.743 (sd 0.029), while retaining computational efficiency and interpretability despite acknowledged data limitations.
- Framework: MultiSigBERT combines domain-specific language embeddings, sequential structured variables, dimensionality reduction, path signature encoding, and sparse Cox regression.The resulting pipeline is described as computationally efficient and interpretable for survival prediction.
- Results: 0.743 (sd 0.029) was the C-index achieved on a large real-world oncology cohort under a landmark design with stable calibration.Landmarking restricted feature construction to observations available before prediction, preventing temporal information leakage.
- Multimodal integration: Synchronized multimodal trajectories enable principled geometric encoding of temporal interactions while preserving the ordering of arbitrarily many clinical reports per patient.The architecture supports heterogeneous clinical data and remains computationally lightweight.
- Limitations and future work: The study used limited structured variables requiring interpolation within the landmark window, motivating future inclusion of static covariates and external validation on independent oncology cohorts.Suggested covariates include tumor type, sex, and age at diagnosis; future work will also explore richer multimodal representations.
- Representation: Path signatures encode temporal dependencies and higher-order interactions from heterogeneous modalities into fixed-dimensional vectors while handling irregularly sampled clinical trajectories.This representation is identified as a key contribution and as particularly suitable for real-world clinical data.
- Model properties: The landmarking, signature-based feature extraction, and Cox–LASSO combination provides a favorable trade-off among predictive performance, interpretability, computational efficiency, and robustness.The findings suggest that classical sparse survival models remain competitive when paired with expressive temporal representations.
Supplementary
The supplementary material characterizes the oncology cohort, clinical-report corpus, and longitudinal structured measurements. The dataset includes 7,885,925 observations from 2,527 unique patients and spans heterogeneous cancer types, report categories, and variable distributions.
- Cohort composition: 782 patients had breast cancer, making it the largest tumor group, followed by gynecological, intestinal, and prostate cancers.The corresponding counts were 232, 227, and 201 patients, respectively.
- Clinical corpus: 83,898 documents were consultation reports, while 33,966 were hospital stay reports, making these the two dominant report categories.Follow-up consultations during treatment, consultation reports with procedures, and discharge reports accounted for 3,778, 1,618, and 789 documents, respectively.
- Sequential measurements: Longitudinal structured variables comprised peripheral oxygen saturation, systolic blood pressure, Karnofsky performance index, and pulse rate.Empirical distributions and boxplots summarize their relative frequencies, medians, interquartile ranges, and potential outliers across available observations.
- Dataset scale: 7,885,925 observations corresponded to 2,527 unique patients in the sequential structured-variable dataset.These observations formed the descriptive-statistics basis for the longitudinal clinical variables.