Source-linked AI summary
SYNPRED: A Synergistic Approach to Multimodal Learning for Clinical Prediction
Ivana Janíčková, Yen Y. Tan, Thomas H. Helbich, Philipp Seeböck, Julieta Di Marco, Zsuzsanna Bago-Horvath, Ulrike Heber, Thomas Spiegel, Georg Langs
TL;DR
Multimodal methods that emphasize shared information can underuse complementary, task-relevant signals across medical modalities. SYNPRED disentangles shared, modality-specific, and instance-specific representations in a multimodal VAE; it improves multimodal prediction over the described baselines and also benefits single-modality prediction, particularly imaging.
Problem
Alignment-based approaches primarily capture shared information, while medical modalities can contain complementary, modality-specific signals relevant to prediction and share limited semantic overlap.
Source: §1 · p. 2Method
SYNPRED uses a multimodal VAE to disentangle shared and modality-specific task-relevant representations from instance-specific variation, with reconstruction and KL-based regularization.
Source: §1 · p. 2, Overall Loss · p. 5Results
SYNPRED achieved 0.693 AUROC versus 0.549 for the simple unimodal approach on imaging, and outperformed alignment-based and multimodal VAE baselines in multimodal prediction while improving single-modality prediction, particularly imaging.
Source: Comparison with state-of-the-art… · p. 6, §4 · p. 9Takeaways & Limitations
The results provide empirical evidence that multimodal training can uncover predictive structure in the weaker imaging modality that is difficult to capture from imaging alone.
Source: §5 · p. 9The formulation assumes two modalities, each generated from instance-specific variation, modality-specific task-relevant information, and shared task-relevant information.
Source: Problem Formulation · p. 3
Abstract
from arXiv · showhide
Multimodal medical data captures complementary aspects of disease. It provides a comprehensive basis for diagnosis and the identification of biological associations linked to progression or treatment response. Established contrastive learning approaches exploit shared information across modalities, but fall short in exploiting complementary, modality-specific signals relevant for a task. Here, we propose a synergistic multimodal variational autoencoder (VAE) framework that separates task-relevant unimodal and multimodal information from instance-specific variation task-irrelevant, allowing complementary signals across modalities to be leveraged for prediction. The method yields benefits even if modalities are missing at test time. We evaluate the method on a treatment response prediction task using medical imaging and RNA data. Empirical comparison with alignment-based and multimodal VAE state-of-the-art approaches demonstrates improved multimodal prediction performance, and consistent benefits for prediction from a single modality after multimodal training. The full code can be found here: https://github.com/cirmuw/SYNPRED.
1 Introduction
Medical modalities can describe complementary aspects of disease while sharing limited semantic overlap, making shared-information-focused approaches an incomplete fit. SYNPRED disentangles shared and modality-specific task-relevant signals from instance-specific variation and improves multimodal and single-modality prediction after multimodal training.
1 Introduction
- Multimodal medical data can improve predictive performance and clinical decision support.It can combine complementary aspects of disease rather than only overlapping semantic information.Source: §1 · pp. 1–2
- Alignment-based approaches primarily capture shared information and may miss complementary, modality-specific signals relevant to prediction when modalities have limited semantic overlap.Imaging may reflect tumour structure, while molecular data characterises biological processes such as gene expression.Source: §1 · p. 2
- SYNPRED separates shared and modality-specific task-relevant features from residual task-independent variation, targeting synergistic cross-modal information unavailable from one modality alone.The framework is proposed for clinical prediction using multimodal generative representations.Source: §1 · p. 2
- Experiments on paired breast MRI and gene-expression data from the BreastDCEDL I-SPY2 cohort show improved multimodal performance and consistent single-modality gains, particularly for imaging.The authors present these results as empirical evidence for synergistic cross-modal interactions improving representation learning.Source: §1 · p. 3
2 Method
SYNPRED models two modalities through shared task-relevant, modality-specific task-relevant, and instance-specific latent components. It trains these representations with reconstruction and KL-based regularization, then fuses task-relevant embeddings for prediction, including when modalities are missing.
2 Method
- The formulation represents each modality as generated from instance-specific variation, modality-specific task-relevant information, and shared task-relevant information, aiming to preserve complementary and shared signals.It is presented for a two-modality setting with inputs X1 and X2 and task label Y.Source: Problem Formulation · p. 3
2.1 SYNPRED: Synergistic Multimodal VAE
- Each modality encoder produces Gaussian posteriors for shared task-relevant (TR-fusion), modality-specific task-relevant (TR-uni), and instance-specific (IS) latent components.Sampled latent components are concatenated for modality reconstruction, with an ℓ1 reconstruction loss and KL regularization of IS latents toward a standard Normal prior.Source: §2.1 · p. 3, Figure 1 · p. 4, Latent Decomposition and Reconstruction · p. 4
- TR-fusion posteriors are combined across available modalities with a mixture-of-experts, while task-relevant posterior parameters are fused by a product-of-experts for prediction.Evaluation uses linear probing or fine-tuning on multimodal embeddings and tests unimodal or multimodal inputs without retraining.Source: Multimodal Fusion · p. 4, Prediction · p. 5
- Task-conditioned KL terms shape TR-fusion and TR-uni posteriors toward label-conditioned priors to capture shared and complementary task-relevant information.The overall training objective weights reconstruction and the two KL-based terms, with β controlling latent regularization strength.Source: Multimodal Fusion · p. 5, Overall Loss · p. 5
3 Experiments and Results
SYNPRED is evaluated for breast-cancer treatment-response prediction using paired MRI and RNA data, with multimodal comparisons, ablations, and explainability analysis. It outperforms state-of-the-art approaches in multimodal prediction, improves imaging-only prediction after multimodal training, and produces more coherent task-relevant localization.
3 Experiments and Results
- The evaluation uses 1,452 patients with pretreatment breast-cancer labels, while paired MRI and gene-expression data are available for 982 I-SPY2 patients.MRI inputs are derived from early and late subtraction images; RNA profiles contain measurements for over 19,000 genes.Source: §3 · p. 5
- The model uses two modality-specific VAE branches and a fusion module, with performance assessed by linear probing and MLP fine-tuning.Missing RNA is handled using mixture- and product-of-experts fusion.Source: Implementation Details · pp. 5–6
3.1 Results
- SYNPRED outperforms state-of-the-art methods for multimodal treatment-response prediction, including alignment-based and multimodal VAE baselines.Table 1 covers image-only, RNA-only, and multimodal AUROC evaluations; its caption does not provide cell values.Source: Comparison with state-of-the-art… · p. 6, Table 1 · p. 6
- 0.693 AUROC for SYNPRED versus 0.549 for the simple unimodal approach in imaging-only prediction after multimodal training.The authors report that the imaging benefit is particularly pronounced.Source: Comparison with state-of-the-art… · p. 6
- Removing latent disentanglement lowers multimodal AUROC from 0.725 to 0.646, while omitting unimodal or fused task-relevant components yields 0.658–0.680.The ablation results indicate that both modality-specific and multimodal task-relevant information matter for prediction.Source: Ablation Study · p. 7, Table 2 · p. 7
- SYNPRED reaches a DICE score of 0.167 against lesion segmentations and improves localization compared with state-of-the-art approaches.The full disentangled model also outperforms the non-disentangled variant and all ablated configurations in representation-based explainability evaluation.Source: Analysis of Explainability · p. 7, Figure 2 · p. 8
4 Discussion
The discussion argues that explicitly modeling shared and complementary task-relevant information is useful when modalities have limited semantic overlap and different predictive strength. The authors report improved multimodal prediction and imaging localization, while noting that generalizability remains untested beyond one paired breast MRI–RNA benchmark.
4 Discussion
- For modalities with limited semantic overlap, the framework models shared, modality-specific, and instance-specific representations rather than relying on alignment alone.This setting includes radiology-genomics applications in which imaging may be the weaker predictive modality.Source: §4 · p. 9
- The authors report stronger multimodal prediction than alignment-based and multimodal VAE baselines, plus improved overlap with ground-truth segmentations.Ablations indicate that both unimodal and fusion components contribute to performance.Source: §4 · p. 9
- Generalizability is not established beyond a single paired breast MRI–RNA benchmark, and the framework was not evaluated for more than two modalities or tasks beyond classification.The authors call for validation on additional datasets.Source: §4 · p. 9
5 Conclusion
The paper presents a multimodal generative framework for clinical prediction with modalities that have limited semantic overlap. It reports consistent single-modality gains after multimodal training, particularly for imaging, alongside competitive multimodal performance.
5 Conclusion
- The framework separates modality-specific and shared task-relevant representations from instance-specific variation to learn from complementary multimodal signals.The conclusion reports consistent single-modality gains after multimodal training, particularly for imaging with limited standalone predictive power.Source: §5 · p. 9