Cross-Modal Translation and Alignment for Survival Analysis

1. Introduction

• Survival prediction supports assessment of disease progression and treatment efficiency, while traditional clinical indicators and follow-up reports can be time-consuming and impractical.The introduction also describes pathological images and genomic profiles as sources of prognostic information.Source: §1 · p. 1
• Straightforward feature fusion can overlook cross-modal correlations, while genomic-guided methods can discard pathology information irrelevant to gene expression.The authors identify both as shortcomings of existing multimodal survival-analysis approaches.Source: §1 · p. 2
• CMTA uses parallel encoder-decoder structures to extract intra-modal representations and generate cross-modal representations for pathological images and genomic profiles.The framework uses cross-modal representations to enhance and recalibrate intra-modal representations before producing survival predictions.Source: §1 · p. 2
• A cross-modal attention module explores intrinsic correlations and transfers potential complementary information between modalities.It is embedded in the encoder-decoder structure.Source: §1 · p. 2
• CMTA consistently achieves superior performance to state-of-the-art methods on five public TCGA datasets.The introduction reports this as the outcome of the experiments.Source: §1 · p. 2

2.1. Survival Analysis using Single-modality

• Pathology-based methods use multi-instance learning to identify image regions contributing to survival, and pathology can outperform radiology, though performance remains short of clinical-application requirements.Genomic profiles are described as a further route to disease modeling and prognosis.Source: §2.1 · p. 2

2.2. Survival Analysis using Multi-modality

• CMTA’s illustration shows parallel encoders extracting intra-modal representations, decoders generating cross-modal representations, and attention bridging the modalities before prediction fusion.The figure labels the representations p and g, and ˆp and ˆg, respectively.Source: Figure 1 · p. 3
• Multimodal approaches combine complementary patient information, but straightforward fusion can overlook cross-modal correlations and genomic guidance can discard pathology unrelated to gene expression.The related-work discussion presents these limitations as conflicting with the aim of integrating complementary information.Source: §2.2 · p. 2, …Cross-Modal Attention Module · p. 3

3. Method

• The method outlines multimodal survival formulation, processing of both modalities, and the framework's components.Source: §3 · p. 3

3.1. Problem Formulation

• Each patient is represented by WSI and genomic sets, censoring status, and survival time; the model estimates hazard from both modalities.The record includes right-uncensoring status and overall survival time in months.Source: §3.1 · p. 3

3.2. Data processing and Feature Extraction

• WSIs become 512×512 patches at 40× with ResNet-50 1024-dimensional features; RNA-seq, CNV, and SNV are grouped into six categories and embedded.The genomic categories include tumor suppression, oncogenesis, protein kinases, cellular differentiation, transcription, and cytokines and growth.Source: §3.2 · p. 4

3.3. Pathology Encoder and Genomics Encoder

• Modality-specific self-attention encoders integrate patch and gene-set features; pathology adds PPEG, while Nystrom attention approximates global attention for large sets.The pathology encoder uses a learnable class token to gather information from patches.Source: §3.3 · p. 4

3.4. Cross-Modal Attention Module

• Bidirectional attention maps relate pathology and genomics tokens, extract cross-modal information, and transfer complementary information between modalities.The module takes pathology and genomics instance tokens as input; its maps represent associations in opposite directions.Source: §3.4 · pp. 4–5

3.5. Genomics Decoder and Pathology Decoder

• Parallel decoders translate modality-related information into cross-modal representations using learnable class tokens and self-attention; the genomics decoder also uses PPEG.The pathology decoder applies two self-attention layers, while the genomics decoder applies two self-attention layers with PPEG.Source: §3.5 · p. 5

3.6. Feature Alignment and Fusion

• Cross-modal representations are L1-aligned to intra-modal features, which they enhance and recalibrate before fusion into survival predictions.The intra-modal representations are detached during alignment, making the optimization unidirectional.Source: §3.6 · p. 5

4.1. Datasets and Evaluation Metrics

• Experiments used five TCGA cancer datasets, 5-fold cross-validation, and c-index to evaluate survival-prediction methods.The datasets were BLCA (n = 373), BRCA (n = 956), GBMLGG (n = 569), LUAD (n = 453), and UCEC (n = 480); c-index measures pairwise ordering of predicted survival times.Source: §4.1 · p. 6, §4 · p. 6

4.2. Comparisons with State-of-the-Art

• CMTA improved over the best single-modal methods by 4.54% on BLCA, 2.01% on BRCA, 1.61% on GBMLGG, 5.29% on LUAD, and 0.75% on UCEC.The authors report that CMTA consistently achieved superior performance across all five datasets.Source: §4.2 · p. 7
• Against MCAT, CMTA's performance increased by 1.83% on BLCA, 0.89% on BRCA, 1.81% on GBMLGG, 2.67% on LUAD, and 6.39% on UCEC.The authors also report that CMTA outperformed other listed state-of-the-art multimodal methods.Source: §4.2 · p. 7
• Compared with DualTrans, CMTA improved performance by 3.03% on BLCA, 0.42% on BRCA, 1.38% on GBMLGG, 1.58% on LUAD, and 2.51% on UCEC.The authors attribute these improvements to translating related information into specific cross-modal representations.Source: §4.2 · p. 7

4.3. Ablation Studies

• L1 loss outperformed the other tested similarity metrics on all datasets except GBMLGG, where results showed no obvious differences.The alternatives were MSE loss, KL divergence, and cosine similarity; Figure 3 presents performance under different similarity metrics.Source: §4.3 · pp. 7–8, Figure 3 · p. 7
• Removing cross-modal attention, alignment constraints, or the p/g detaching step lowered performance across datasets; detaching-step removal caused losses up to 9.08% on BLCA.The authors associate detaching with avoiding redundant shared information; Table 2 summarizes the three component removals.Source: Table 2 · p. 7, §4.3 · p. 8
• Removing the PPEG module resulted in performance drops of 2.81% on BLCA, 0.52% on BRCA, 0.49% on LUAD, and 3.16% on UCEC.Source: §4.3 · p. 8

4.4. Survival Analysis

• Kaplan-Meier analyses separated patients by predicted risk scores, and Logrank-test p-values were below 0.05 for every dataset.Patients were split into low- and high-risk groups using the mid-value of predicted risk scores.Source: §4.4 · p. 8, Figure 4 · p. 8

• CMTA outperformed state-of-the-art methods in experiments on five public TCGA datasets.Source: §5 · p. 8
• A cross-modal attention module bridges pathological and genomic modalities to explore their potential correlations and interactions.Source: §5 · p. 8
• Parallel encoder-decoder structures integrate intra-modal information and generate cross-modal representations for pathological and genomic features.The generated cross-modal representations enhance and recalibrate intra-modal representations for survival prediction.Source: §5 · p. 8