PaperlayerView exact revision on arXiv

Source-linked AI summary

Saga: an in silico ontology for a multimodal patient world model

Youcheng Li, Fa Tian, Binbin Su

arXiv:2610.33054v1cs.CV

TL;DR

Longitudinal prediction must connect patient history to future events, timing and disease morphology, while organizing readouts around a shared state and advancing trajectories without real intermediate observations. Saga learns that shared state, predicts clinical events and multimodal images, and re-encodes generated events and times; at eight steps, it outperformed a longitudinal Transformer on event F1 and timing error.

  • Problem

    Longitudinal prediction must connect patient history to future events, clinical timing and disease morphology, while organizing future readouts around a common state and advancing trajectories without real intermediate observations.

    Source: Introduction · p. 1

  • Method

    Saga learns a shared predictive representation of multimodal patient histories, predicts clinical events and images, and appends generated events and times to re-encode the history.

    Source: Introduction · p. 3

  • Results

    At eight steps, Saga's event F1 was 0.672 versus 0.553 for the longitudinal Transformer, and its timing error was lower by 8.7 days.

    Source: Event feedback supports continuing… · p. 13

  • Takeaways & Limitations

    Saga connects future events, timing and disease morphology through an updatable patient state, with generated-event feedback advancing trajectories and comparative advantages maintained through eight steps.

    Source: Discussion and outlook · p. 19

    Progression-risk shifts under altered conditioning describe model readouts and do not estimate causal treatment effects.

    Source: Comparators, ablations and conditional… · p. 8

Abstract

from arXiv · show

Simulating an individual's disease course requires connecting patient history to future events, clinical timing and disease morphology. Longitudinal prediction and multimodal representations exploit this information, but organizing future readouts around a common patient state and advancing trajectories without real intermediate observations remain central challenges. Here we introduce Saga, a multimodal patient world model that learns a shared state from patient histories to predict clinical events and generate multimodal images. Generated events and times are appended to the history and re-encoded to advance prediction. We studied 461,370 patients with 12 cancer types across 112 centres. Modality-specific paired evaluations linked future images to observed follow-up. With historical images held fixed, removing patient conditioning reduced event F1 from 0.716 to 0.591 and increased lesion-burden error from 11.8% to 21.3%. In free rollouts, eight-step event F1 was 0.672, compared with 0.553 for a longitudinal Transformer evaluated from the same origins, and timing error was lower by 8.7 days. In the internal test set, one-year progression AUROC was 0.882, with values of 0.871 and 0.856 in temporal and geographical external evaluations. These results show that shared patient states connect future phenotypes across observational scales with event-driven trajectory progression, sustaining comparative predictive advantages during rollout and discrimination in external clinical evaluation. Saga makes patient-level disease simulation a multimodal prediction task that can be tested against observed follow-up.

Introduction

Saga addresses the challenge of organizing fragmented patient histories into an updatable representation that links future events, timing and disease phenotypes. It predicts clinical events and multimodal images, then re-encodes generated events and times to advance prediction.

Introduction

  • Patient histories support longitudinal clinical prediction and trajectory generation, but multimodal disease modelling also needs future events and phenotypes organized around a common, updatable representation.Prior work includes patient-record prediction and trajectory models, as well as multimodal oncology and pathology models.Source: Introduction · p. 2
  • Saga evaluates patient conditioning by holding historical images fixed and evaluates rollout by appending generated events and times without real intermediate records.The study uses 461,370 patients with 12 cancer types from 112 centres.Source: Introduction · pp. 2–3
  • Saga learns a shared predictive representation of multimodal patient histories to predict clinical events and generate multimodal images.Generated events and their times are appended to the history and re-encoded to advance prediction.Source: Introduction · p. 3

Methods

The study links clinical, imaging and pathology records into patient-level histories, while defining partitions and prediction origins to preserve temporal and geographic separation. Inputs are limited to information available at each prediction origin.

Cohort, partitions and available history

  • Available history includes only information available at the prediction origin, excluding results obtained afterward even when the underlying event occurred earlier.Events are defined by content, occurrence time and information-availability time.Source: Cohort, partitions and available history · p. 3
  • Patients with tumour diagnoses, linkable identifiers and chronologically orderable records are included, with eligibility defined separately for each evaluation.Records with unreliable links, conflicting key dates or missing task reference outcomes are excluded from that evaluation; all four imaging modalities are not required for cohort inclusion.Source: Cohort, partitions and available history · p. 3
  • Patients are assigned to partitions before prediction origins and observation pairs are generated, and all records from each patient remain in one partition.Geographical evaluation holds out centres; temporal holdout uses subsequently eligible patients after a prespecified cutoff.Source: Cohort, partitions and available history · p. 3
  • Diagnoses, treatments and laboratory measurements are mapped to common vocabularies and units, with missingness represented by indicators and observation masks.Missing modalities are not interpreted as normal findings.Source: Cohort, partitions and available history · p. 4

Tasks, clinical endpoints and evaluation metrics

Saga evaluates clinical outcomes using task-specific endpoints and metrics, and encodes longitudinal histories into a shared patient state. Transitions use treatment and clinical interval information to produce future states and readouts.

Tasks, clinical endpoints and evaluation metrics

  • The main clinical tasks are next event, 90-day events, one-year progression and overall survival, evaluated with NDCG@5, AUPRC, AUROC and C-index, respectively.Progression is recorded during clinical follow-up, whereas overall survival uses all-cause death as its endpoint.Source: Tasks, clinical endpoints and… · p. 4
  • Additional tasks include PFS, DFS/RFS, first response, next treatment and treatment-line switching, each using task-appropriate classification, ranking or timing metrics.Next-treatment Recall@5 measures reference-label coverage among the five highest-ranked candidates; macro-F1 equally weights class-specific F1 scores.Source: Tasks, clinical endpoints and… · p. 4
  • Cancer-specific survival evaluation reports time-dependent AUC and integrated Brier score to describe discrimination at a specified time and prediction error integrated over time.Source: Tasks, clinical endpoints and… · p. 4

Shared patient state, transitions and clinical readouts

  • The shared patient state is encoded from longitudinal inputs and supports clinical readouts from current or predicted states, plus image generation from predicted states.The input includes content, modality, age and time-gap embeddings; grouped temporal causal attention prevents earlier origins from accessing later time blocks.Source: Shared patient state, transitions and… · pp. 4–5
  • The transition is conditioned on the current state, treatment and clinical interval, with next-event timing predicted from the current state and treatment.Fixed-horizon tasks instead use the specified interval.Source: Shared patient state, transitions and… · p. 4

Modality conditioning mappings and image generation

Saga uses separate patient-state and historical-image pathways to condition multimodal image generation. The shared generation procedure covers four modalities, while reported quantitative imaging evaluation covers three.

Modality conditioning mappings and image generation

  • Conditional DDIM sampling and decoding generate images from initial noise, guided by patient and historical-image conditions.Ultrasound, MRI, CT and pathology use modality-specific encoding and decoding branches; pathology outputs are local patches.Source: Modality conditioning mappings and… · p. 6
  • Quantitative imaging evaluation covers ultrasound, MRI and pathology, although the generation procedure also includes CT.Source: Modality conditioning mappings and… · p. 6
  • Modality mappings transform the predicted patient state into patient-condition tokens, while historical-image features form separate image-condition tokens.Both conditions enter the denoising network through cross-attention.Source: Modality conditioning mappings and… · p. 6

Training and sampling

Clinical training combines event, timing and survival objectives, with image generation trained on observed follow-up images. Checkpoint and hyperparameter selection use development data only.

Training and sampling

  • Clinical training combines event-classification, timing-prediction and survival objectives.The image-generation branch uses observed follow-up images encoded into latent variables; its denoising network predicts added Gaussian noise with a mean-squared-error loss.Source: Training and sampling · p. 6
  • Records missing a label type or target image contribute only to objectives for which targets are available.Source: Training and sampling · p. 6
  • Checkpoint and hyperparameter selection are restricted to development data.Source: Training and sampling · p. 6

Longitudinal imaging pairs and generation evaluation

Longitudinal image evaluation pairs generated images with observed follow-up from the same patient and modality at specified intervals. Modality-specific and aggregate metrics assess changes, representations, phenotype classification and image similarity.

Longitudinal imaging pairs and generation evaluation

  • At 6, 12 and 24 months, ultrasound and MRI use lesion-change and burden-change MAE, while pathology uses representation cosine similarity and phenotype macro-F1.Cosine similarity measures representation-vector alignment; macro-F1 is the equally weighted mean of category-specific F1 scores.Source: Longitudinal imaging pairs and… · p. 6
  • Aggregate image evaluation reports SSIM, PSNR and LPIPS, alongside major-lesion-error and image-evaluability proportions.SSIM and PSNR are evaluated in spatially corresponding image regions; PSNR is defined by image dynamic range and mean squared error, while LPIPS measures perceptual differences using pretrained network features and learned distance weights.Source: Longitudinal imaging pairs and… · pp. 6–7
  • Images are paired within the same patient, modality and anatomical site, using consistent evaluation regions and preprocessing.Source: Longitudinal imaging pairs and… · p. 6
  • MAE is the mean absolute difference between predicted and reference measurements, in the measurement’s unit.Figure 3b presents ultrasound lesion-change and MRI burden-change errors separately; Figure 3f presents lesion-burden error in the conditioning experiment, all as percentages.Source: Longitudinal imaging pairs and… · pp. 6–7

Event feedback and free multistep rollout

Free rollout begins from observed history and advances by feeding generated events and occurrence times back into that history. The updated history is re-encoded for the next transition, while historical images remain fixed and generated images do not update clinical state.

Event feedback and free multistep rollout

  • Free rollout starts from observed history and updates it with generated events and their occurrence times.Comparisons use the same origin histories and reference follow-up, with rollout inputs limited to origin information and model-generated events and times.Source: Event feedback and free multistep… · p. 7
  • After event write-back, the history is re-encoded to support the next transition; historical images remain fixed at the origin.Generated images are downstream readouts and neither enter history nor update clinical state.Source: Event feedback and free multistep… · p. 7
  • At 1, 2, 4 and 8 evaluation depths, event F1 compares Saga, a longitudinal Transformer, direct multihorizon prediction and a frozen-latent-state model.Timing MAE compares Saga and the longitudinal Transformer; depth counts recursive rollout steps, while elapsed clinical time is determined by generated intervals.Source: Event feedback and free multistep… · p. 7

Comparators, ablations and conditional responses

The comparisons assess clinical models, state-transition alternatives, component ablations and changes in patient or treatment conditioning. Progression-risk shifts describe changes in readouts, not causal treatment effects.

Comparators, ablations and conditional responses

  • Clinical model comparisons include Saga, a clinical baseline, text-only and image-only models, a static multimodal model and a longitudinal Transformer.Text-only and image-only models restrict input modality; the static model fuses origin features without event order, while the Transformer encodes history over time.Source: Comparators, ablations and conditional… · pp. 7–8
  • State-model comparisons test non-recurrent direct prediction, a frozen latent state, and removal of longitudinal history, treatment/action tokens or explicit time encoding.Non-recurrent direct prediction removes subsequent state recursion, while the frozen-latent-state comparator retains a fixed latent representation.Source: Comparators, ablations and conditional… · pp. 7–8
  • Component ablations compare linear mapping, no mapping, and removal of imaging, text, laboratory measurements, time encoding or pretraining against the corresponding full model.Source: Comparators, ablations and conditional… · p. 7
  • Patient-condition exchanges and treatment-condition changes assess conditional responses in event and imaging readouts.Progression-risk shifts describe altered model readouts and do not estimate causal treatment effects.Source: Comparators, ablations and conditional… · pp. 7–8

Preliminary clinician assessment

The section describes physician assessment of trajectory plausibility and cross-output consistency, evaluation across hospital and population strata, and the report’s descriptive statistical summaries.

Preliminary clinician assessment

  • Two physicians with more than ten years of experience assessed trajectory plausibility, image–text–time consistency, major contradictions and uncertainty or unevaluability, with agreement summarized by κ.The assessment covered observed follow-up, Saga-generated trajectories and contradictory controls.Source: Preliminary clinician assessment · p. 8

External evaluation and population strata

  • Hospital evaluation spans internal, temporal-holdout, geographical-holdout and leave-one-centre-out settings, with cancer-specific results and age, sex and stage strata also reported.Leave-one-centre-out results use the mean and between-centre standard deviation.Source: External evaluation and population… · p. 8

Statistical summaries

  • Results are described using point estimates and within-experiment model differences, with between-centre standard deviations and 95% confidence intervals for secondary tasks.Comparisons are descriptive and use the direction and magnitude of differences.Source: Statistical summaries · p. 8

Results

Saga predicts clinical outcomes, future imaging and recursively advancing trajectories from patient histories. Evaluations report advantages over comparators, while also documenting performance across follow-up horizons and external settings.

Future imaging prediction and responses to patient conditioning

  • Across paired follow-up, pathology cosine similarity declined from 0.883 at 6 months to 0.818 at 24 months, while ultrasound lesion-change MAE rose from 10.8% to 20.1%.MRI burden-change MAE rose from 7.6% to 13.9%; evaluations included 1,284, 1,116 and 860 total pairs at 6, 12 and 24 months.Source: Future imaging prediction and… · p. 11, Figure 3 · p. 12
  • At 24 months, summary imaging SSIM was 0.691, LPIPS was 0.154, major-lesion errors were 9.7%, and evaluability was 86.9%, compared with 0.781, 0.098, 4.8% and 94.6% at 6 months.The reported fidelity, lesion-error and evaluability measures indicate greater difficulty at longer horizons.Source: Future imaging prediction and… · p. 11, Figure 3 · p. 12
  • With historical images held fixed, removing patient conditioning reduced event F1 from 0.716 to 0.591 and increased lesion-burden MAE from 11.8% to 21.3%.Swapping patient conditions yielded event F1 of 0.623 and burden MAE of 18.5%; removing treatment conditions yielded 0.681 and 15.6%, respectively.Source: Future imaging prediction and… · pp. 11–12, Figure 3 · p. 12

Event feedback supports continuing trajectories with clinical coherence

  • At eight rollout steps, Saga event F1 was 0.672 versus 0.553 for the longitudinal Transformer, and timing MAE was 8.7 days lower.Saga appends generated events and times to history and re-encodes it; its event F1 advantage over the Transformer grew from 0.030 at one step to 0.119 at eight steps.Source: Event feedback supports continuing… · p. 13, Figure 4 · p. 14
  • Physicians rated Saga trajectories 94.8% plausible and 93.1% consistent across image, text and time, with a 5.4% major-contradiction rate versus 97.2%, 96.5% and 2.8% for real trajectories.Agreement for model trajectories was κ = 0.72; contradictory controls had a 31.6% major-contradiction rate.Source: Event feedback supports continuing… · pp. 13–14

Multitask clinical prediction and patient-state component comparisons

  • Relative to non-recurrent prediction, the full model improved one-year progression AUROC, event F1 and timing MAE by 0.021, 0.029 and 3.3 days; removing longitudinal history lowered progression AUROC to 0.842.Removing explicit time encoding lowered progression AUROC to 0.858 and increased timing MAE to 22.7 days.Source: Multitask clinical prediction and… · pp. 15–16
  • Saga increased next-event NDCG@5 from 0.744 to 0.782, 90-day event AUPRC from 0.704 to 0.734, one-year progression AUROC from 0.854 to 0.882, and OS C-index from 0.779 to 0.807 versus the longitudinal Transformer.These are the four main clinical tasks compared with the strongest listed comparator.Source: Multitask clinical prediction and… · p. 15, Figure 5 · p. 16

Performance across hospitals, cancers and prediction horizons

Discussion and outlook

Saga frames multimodal future readouts and recursive event prediction as one updatable patient-state problem. The discussion positions its findings as descriptive and identifies external and prospective validation as next steps.

Discussion and outlook

  • Saga connects future clinical events, timing, and disease morphology through an updatable patient state, organizing multimodal generation with continuing trajectory prediction.The discussion groups evidence across paired image follow-up, conditioning perturbations, free rollouts, and clinical evaluations.Source: Discussion and outlook · p. 19
  • With historical images fixed, swapping or removing patient conditions reduces event and morphology performance, while treatment-output differences remain descriptive without causal identification.Ultrasound and MRI characterize lesion-level change; pathology patches characterize local tissue phenotypes.Source: Discussion and outlook · p. 19
  • Saga maintained event and timing advantages over matched-origin Transformer comparisons through eight rollout steps, while sequence distances and feature drift increased with depth.The event-F1 difference from the Transformer grew from 0.030 to 0.119.Source: Discussion and outlook · p. 19
  • Routine-care examination frequency, modality availability, and follow-up completeness vary with disease status and care access, so imaging results require acquisition-context interpretation.The discussion notes that imaging error and evaluability should be interpreted alongside acquisition protocols and tissue-sampling scales.Source: Discussion and outlook · p. 19

Outlook

  • The outlook calls for broader external validation, larger blinded plausibility assessments, joint event-image evaluation, and studies of model information in clinical workflows.Workflow studies will assess effects on physician decisions and patient outcomes.Source: Outlook · pp. 19–20
  • Complete histories may support trial eligibility assessment, risk stratification, and prognostic enrichment, but treatment-effect-based patient selection requires randomized trials or explicit causal identification.The proposed uses are to be tested in independent trial data.Source: Outlook · pp. 19–20

Related papers

Loading 2610.33054v1…