Source-linked AI summary
Case-Specific Priors-Guided Multimodal Fusion for Prognosis Prediction in Head and Neck Cancer
Yue Lin, Shuchang Ye, Shenghai Wu, Dashun Zheng, Mingyuan Meng
TL;DR
HNC prognosis prediction must integrate heterogeneous clinical and pathological evidence, while existing fusion approaches rely on case-generic rationales rather than case-specific cross-modal correspondence. CPM uses LLM-derived priors to guide multimodal fusion through calibration, pairwise interaction, and aggregation. On HANCOCK, it achieved a mean AUC of 83.98% for 5-year overall survival and 2-year recurrence-free survival, improving 7.58% over baseline methods not guided by case-specific priors.
Problem
Existing multimodal fusion approaches use case-generic clinical rationales and do not explicitly exploit LLM-derived case-specific cross-modal correspondence for prognosis prediction.
Source: §1 · p. 2Method
CPM derives case-specific intra-modal contexts and inter-modal relations, together with case-generic multimodal rationale, to guide three-stage fusion.
Source: §1 · pp. 2–3Results
83.98% mean AUC for 5-year overall survival and 2-year recurrence-free survival, improving 7.58% over baseline methods not guided by case-specific priors.
Source: §1 · p. 3Takeaways & Limitations
CPM's reported HANCOCK results demonstrate the effectiveness of explicitly modeling case-specific cross-modal correspondence for multimodal HNC prognosis prediction.
Source: §5 · p. 9InterRI uses pairwise exchanges as a tractable simplification because characterizing full case-specific multimodal correspondence becomes increasingly difficult to model and reliably derive with more modalities.
Source: §2.3 · p. 5
Abstract
from arXiv · showhide
Accurate prognosis prediction for head and neck cancer typically necessitates integrating highly heterogeneous clinical and pathological evidence by sophisticated multimodal fusion. However, existing fusion techniques are usually grounded in case-generic clinical rationales about complementary synergies across modalities, underestimating case-specific variance. In this study, we propose a prognosis prediction framework that explicitly incorporates Case-specific Priors to guide Multimodal fusion (named CPM). It leverages broad medical knowledge of LLMs to characterize each case's (case-specific) intra-modal contexts and inter-modal relations, as well as case-generic multimodal rationale, as priors, enabling three-stage priors-guided multimodal fusion: (i) Intra-modal Context-guided Calibration (IntraCC) that leverages case-specific intra-modal priors to guide feature calibration within each modality; (ii) Inter-modal Relation-guided Interaction (InterRI) that leverages case-specific inter-modal priors to guide pair-wise feature interaction in modality pairs; and (iii) Multimodal Rationale-guided Aggregation (MultiRA) that leverages case-generic multimodal priors to guide final feature aggregation over all modalities. On the well-established HANCOCK challenge dataset involving four distinct medical modalities, our CPM achieved a mean AUC of 83.98% for predicting 5-year overall survival and 2-year recurrence-free survival, improving upon the baseline by 7.58%, pioneeringly demonstrating the effectiveness of LLM-derived medical priors in guiding multimodal fusion for prognosis prediction. The code is available at https://github.com/yuelin421/CPM.
1 Introduction
HNC prognosis prediction requires integrating heterogeneous clinical and pathological evidence, but existing fusion approaches do not adequately capture case-specific cross-modal correspondence. CPM uses LLM-derived priors to guide fusion and reports improved performance on HANCOCK.
1 Introduction
- HNC prognosis prediction depends on multimodal fusion because risk stratification uses clinical and pathological evidence, which is heterogeneous across modalities.The passage identifies ambiguous cross-modal correspondence and mismatched representation granularity as challenges.Source: §1 · p. 2
- Prior clinically grounded fusion methods are tailored to specific modality scenarios and rely on case-generic clinical priors, limiting adaptability to case-specific variance.Generic deep fusion methods also rely on learned correspondence without explicit clinical rationale.Source: §1 · p. 2
- Existing LLM-assisted survival methods use generated text as single-modality evidence, while explicit LLM-derived case-specific cross-modal guidance remains unexplored.Case-by-case expert annotation of cross-modal priors is difficult to obtain at scale.Source: §1 · p. 2
- CPM uses LLM-derived intra-modal contexts, inter-modal relations, and multimodal rationale to guide three-stage fusion.IntraCC calibrates each modality, InterRI guides pairwise interactions, and MultiRA aggregates the resulting features.Source: §1 · pp. 2–3
- 83.98% mean AUC across 5-year overall survival and 2-year recurrence-free survival, improving 7.58% over baseline methods not guided by case-specific priors.The evaluation used the four-modality HANCOCK dataset.Source: §1 · p. 3
2 Methods
CPM derives clinical priors and uses them to guide calibration, pairwise interaction, and aggregation of multimodal evidence. Its training objective combines endpoint prediction with contrastive alignment of modality representations and their intra-modal priors.
2 Methods
- CPM predicts overall-survival and recurrence-free-survival endpoint probabilities and uses masked multi-task binary cross-entropy when corresponding labels are available.The framework takes multimodal evidence for each patient as input.Source: §2 · p. 4
2.1 LLM-derived Clinical Priors
- The LLM-derived priors comprise case-specific intra-modal contexts and inter-modal agreements or conflicts, plus a case-generic rationale for how modalities complement each other in prognosis prediction.The LLM assessments summarize intra-modal findings before characterizing case-level relations.Source: §2.1 · p. 5
2.2 Intra-modal Context-guided Calibration
- IntraCC uses context tokens and residual cross-attention to calibrate each modality, then applies an InfoNCE objective to align modality representations with their corresponding priors.The contrastive loss treats same-patient pairs as positives and cross-patient pairs as negatives, and is computed only for available modalities.Source: §2.2 · p. 5
2.3 Inter-modal Relation-guided Interaction
- InterRI conditions pairwise exchanges on each case's relation evidence, using symmetric cross-attention, a learnable pair gate, and residual connections.Pairwise decomposition simplifies full multimodal interaction, which becomes harder to model and characterize reliably with more modalities.Source: §2.3 · p. 5
2.4 Multimodal Rationale-guided Aggregation
- MultiRA scores available modality vectors conditioned on a shared clinical rationale, scales each vector, and concatenates them for prediction.A soft mixture with the unscaled concatenation reduces sensitivity to noisy predicted weights while retaining a dedicated subspace for each modality.Source: §2.4 · p. 6
3 Experimental Setup
The experiments evaluate CPM on HANCOCK using its official patient split and four medical modalities. Comparisons include unimodal, multimodal, and LLM-assisted methods, with stage-wise ablations and DeLong testing for pairwise AUC differences.
3.1 Dataset
- HANCOCK contains 764 HNC patients with four modalities, and experiments use the official split of 611 training and 152 test patients.The modalities include whole-slide images, tumor-center tissue microarrays, clinical narratives, and structured clinicopathologic and laboratory measurements.Source: §3.1 · p. 6
3.2 Implementation Details
- CPM training combines masked multi-task BCE with IntraCC's InfoNCE loss, and model selection uses five-fold cross-validation on the training split.The implementation uses AdamW and 50 epochs; modality features are extracted using UNI embeddings for WSIs, ResNet-50 for TMA cores, ClinicalBERT for narratives, and encoded structured features.Source: §3.2 · pp. 6–7
3.3 Experiment Designs
- Comparisons include unimodal results, feature concatenation, Random Forest, recent LLM-assisted methods, and stage-wise CPM ablations; DeLong's test assesses pairwise AUC differences.Top HANCOCK challenge methods were excluded because they were not publicly released and the hidden test set was inaccessible.Source: §3.3 · p. 7
4 Results and Discussion
CPM achieved the best overall performance among the reported methods on HANCOCK, while tabular-only prediction outperformed several multimodal methods. Removing any one CPM stage reduced overall AUC, though the ablations remained competitive with representative methods.
4 Results and Discussion
- Tabular-only prediction outperformed several multimodal methods, indicating that effective fusion across HANCOCK's four modalities is challenging.The discussion attributes limitations in existing methods to lack of clinical priors, scenario-specific designs, or lack of explicit case-specific cross-modal correspondence.Source: §4 · pp. 7–8
- CPM achieved the best overall performance, surpassing unimodal settings and existing multimodal methods on HANCOCK.The comparison is reported in Table 1.Source: Table 1 · p. 8, §4 · p. 8
- Removing IntraCC, InterRI, or MultiRA consistently reduced overall AUC, while each single-stage ablation remained competitive with representative methods.Table 2 reports the CPM ablations.Source: Table 2 · p. 8, §4 · p. 8
5 Conclusion
CPM uses LLM-derived case-specific priors to guide multimodal prognosis prediction for head and neck cancer. On HANCOCK, it consistently outperformed representative baselines and competing methods.
5 Conclusion
- On HANCOCK, CPM consistently outperformed representative baselines and competing methods.The authors highlight explicit modeling of case-specific cross-modal correspondence as effective for multimodal prognosis prediction.Source: §5 · p. 9
- CPM guides multimodal fusion by modeling intra-modal contexts, inter-modal relations, and multimodal rationales through IntraCC, InterRI, and MultiRA.Source: §5 · p. 9
- The authors identify LLM-derived case-specific priors as the basis for guiding multimodal prognosis prediction in head and neck cancer.Source: §5 · p. 9