Source-linked AI summary
Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review
Can Cui, Haichun Yang, Yaohong Wang, Shilin Zhao, Zuhayr Asad, Lori A. Coburn, Keith T. Wilson, Bennett A. Landman, Yuankai Huo
TL;DR
Medical diagnosis and prognosis increasingly require integration of heterogeneous image and non-image data, but effective fusion remains methodologically difficult. This survey reviews 34 recent deep-learning studies and categorizes their fusion approaches. Multi-modal models typically outperform uni-modal models, although adding modalities is not consistently beneficial and generalizability remains open.
Problem
Integrating heterogeneous, complementary medical modalities for diagnosis and prognosis is difficult because they differ in information content and data format.
Method
The survey reviews 34 recent deep-learning studies using image and non-image data, organizing fusion methods into operation-based, subspace-based, tensor-based, and graph-based families.
Results
Multi-modal data typically yielded superior performance compared with uni-modal data, but models using more modalities did not always perform better.
Takeaways & Limitations
Appropriate fusion and modality selection remain important for developing more generalizable and explainable models from limited and incomplete medical data.
Takeaways & Limitations
Over 50% of reviewed studies used multi-modal datasets containing fewer than 1,000 patients.
Abstract
from arXiv · showhide
The rapid development of diagnostic technologies in healthcare is leading to higher requirements for physicians to handle and integrate the heterogeneous, yet complementary data that are produced during routine practice. For instance, the personalized diagnosis and treatment planning for a single cancer patient relies on the various images (e.g., radiological, pathological, and camera images) and non-image data (e.g., clinical data and genomic data). However, such decision-making procedures can be subjective, qualitative, and have large inter-subject variabilities. With the recent advances in multi-modal deep learning technologies, an increasingly large number of efforts have been devoted to a key question: how do we extract and aggregate multi-modal information to ultimately provide more objective, quantitative computer-aided clinical decision making? This paper reviews the recent studies on dealing with such a question. Briefly, this review will include the (1) overview of current multi-modal learning workflows, (2) summarization of multi-modal fusion methods, (3) discussion of the performance, (4) applications in disease diagnosis and prognosis, and (5) challenges and future directions.
Keywords: Multi-modal learning, fusion method, medical data, diagnosis and prognosis
This review takes a technical-centered approach to deep multi-modal fusion of image and non-image medical data for disease diagnosis and prognosis.
- The review focuses on deep multi-modal fusion methods combining image and non-image medical data for diagnosis and prognosis.
1. INTRODUCTION
Routine clinical care generates heterogeneous image and non-image data that offer complementary views for diagnosis and prognosis, but integrating them remains challenging. This review addresses that challenge by organizing recent fusion techniques from an operation-based perspective.
- Routine care produces pathology, radiology, camera, laboratory, genomic, and clinical data that provide different views of the same patient.
- Clinical decision-making can be subjective, qualitative, and affected by substantial inter-subject variability.
- Effective fusion is difficult because modalities contain complementary information while differing in information content and data format.
- Unlike prior clinical-centered surveys, this review categorizes fusion techniques as operation-based, subspace-based, tensor-based, and graph-based methods.
- The survey reviews 34 deep-learning studies from the preceding five years covering diagnosis, prognosis, and treatment prediction.
2. OVERVIEW
The reviewed multi-modal workflow prepares each heterogeneous modality separately, extracts uni-modal representations, fuses them, and produces diagnosis or prognosis predictions. Fusion may occur at feature or decision level, with the review emphasizing feature-level methods.
- 2.2 Workflow: The generalized workflow contains data preprocessing, uni-modal feature extraction, multi-modal fusion, and prediction stages.
- 2.2 Workflow: Separate preprocessing and feature extraction are typically required because image and non-image modalities are heterogeneous.
- 2.3 Multi-modal fusion: Fusion strategies are divided into feature-level fusion and decision-level fusion, with feature-level fusion including early and intermediate fusion.
- 2.3 Multi-modal fusion: Decision-level fusion combines uni-modal predictions through operations such as majority vote, weighted sum, or averaging without retraining uni-modal models.
- 2.4 Applications: The reviewed applications include disease-severity, benign-versus-malignant, and clinical-score tasks, plus survival and treatment-response prediction.
- 2.5 Predictor: Most studies use multi-layer perceptrons to generate diagnosis or prognosis outputs, with cross-entropy common for classification and MSE for regression.
3. UNI-MODAL DATA PREPROCESSING AND FEATURE EXTRACTION
Because medical modalities are heterogeneous, reviewed studies use modality-specific preprocessing and feature extraction before fusion. These steps include image region preparation, structured-data encoding and imputation, text embedding, and representation alignment.
- Pathology, radiology, camera, laboratory, genomic, and clinical modalities require distinct preprocessing and feature-extraction methods before fusion.
- 3.1 Image data: Pathology whole-slide images are commonly cropped into informative patches or regions of interest to fit computational memory.
- 3.1 Image data: Radiology preprocessing may define regions of interest through skull-stripping, registration, foreground extraction, or lesion segmentation before resizing or cropping.
- 3.2 Non-image data: Structured clinical and laboratory data commonly use one-hot encoding, numerical standardization, genomic feature selection, and missing-value imputation.
- 3.2 Non-image data: Free-text clinical reports are processed with natural-language processing, including tokenization and BERT-based embedding.
- 3.3 Uni-modal representation: Feature vectors may be transformed or aggregated with PCA, denoising autoencoders, replicated scaling, deconvolution, biLSTM, or attention-based multi-instance learning.
- 3.3 Uni-modal representation: Uni-modal feature extraction may be unsupervised or supervised and may be trained independently or jointly with fusion.
4. MULTI-MODAL FUSION METHODS
The review organizes multi-modal fusion into feature-level and decision-level strategies, emphasizing operation-, subspace-, tensor-, attention-, and graph-based approaches for combining image and non-image medical data. Reported studies show that fusion performance varies by task and design, with several multi-modal methods outperforming unimodal or simpler fusion baselines.
- Decision-level fusion: Decision-level fusion combines unimodal probabilities or categorical predictions through averaging, voting, or trainable meta-classifiers, offering flexibility and tolerance to missing modalities.Unimodal models can be trained independently, allowing combinations of modalities without testing-phase adjustment.
- Operation-based fusion: Operation-based feature fusion uses parameter-free concatenation, element-wise summation, or multiplication, although summation and multiplication require matched feature shapes.These simple operations were used in early studies comparing multi-modal and unimodal models.
- Subspace-based fusion: Subspace-based methods learn shared representations by increasing cross-modal correlation, enforcing subject-level feature similarity, or reconstructing complementary information.Examples include CCA-based losses, margin-based similarity losses, and encoder-decoder reconstruction objectives.
- Attention-based fusion: Attention-based fusion weights or transforms features using information from another modality, including channel-wise attention and dynamic affine transformations.The dynamic affine transform conditioned scale and offset on image and clinical data and outperformed simple concatenation and channel-wise attention.
- Tensor- and graph-based fusion: Tensor-based fusion models higher-order cross-modal interactions through outer products, while graph-based fusion aggregates information between related patients represented as connected nodes.Tensor fusion was applied to pathology images, cell graphs, and genomic data; population graphs incorporated image and non-image features.
5. DISCUSSION AND FUTURE WORK
The review identifies performance, data, feature-extraction, modality-selection, and explainability challenges that constrain multimodal medical fusion. It argues that fusion strategy and preprocessing must be matched to task, data quality, model capacity, and clinical interpretability.
- Fusion strategy selection: Selecting an optimal fusion strategy remains unresolved because studies use different datasets and settings, rarely include external testing, and may require task-dependent choices.The review calls for fairer comparisons, benchmark datasets, and broader evaluation of fusion levels and metrics.
- Performance and modality selection: Multimodal models typically outperform unimodal counterparts, but adding modalities can have neutral or negative effects when information is redundant, noisy, or biased.One study reported a 4.6% accuracy decrease after adding biopsy-site clinical information; redundant or noisy graph features also reduced performance.
- Data availability: Multimodal datasets remain scarce, with over 50% of reviewed studies containing fewer than 1,000 patients.Pretraining, multitask learning, feature reduction, and data augmentation were commonly used to improve robustness or reduce overfitting under limited data.
- Feature extraction: Unimodal preprocessing and feature extraction substantially influence fusion results and therefore require careful design and evaluation.Reported differences included better fusion with ViT than CNN image encoders and weaker multimodal performance from ClinicalBERT despite stronger unimodal prediction than BERT.
- Explainability: Explainability remains a deployment challenge, although reviewed studies used heatmaps, attention weights, permutation measures, dropout, saliency, and modality-importance analyses.The clinical usefulness of these explanations still requires validation, while multimodal explanations can visualize interactions such as gene–pathology relationships.
6. CONCLUSION
The review synthesizes deep fusion of image and non-image medical data for diagnosis, prognosis, and treatment prediction. It reports typical multimodal gains while emphasizing the need for appropriate methods that support generalizable and explainable models despite limited and incomplete data.
- The survey reviews deep multimodal fusion methods using image and non-image data for medical diagnosis, prognosis, and treatment prediction.
- Multimodal data typically outperform unimodal data, and appropriate fusion methods can further improve performance.
- Open challenges include developing more generalizable and explainable models from limited and incomplete multimodal medical data.
- The review expects multimodal learning to contribute increasingly to precision medicine as a quantitative and trustworthy clinical decision-support methodology.