Source-linked AI summary
A review of deep learning-based information fusion techniques for multimodal medical image classification
Yihao Li, Mostafa El Habib Daho, Pierre-Henri Conze, Rachid Zeghlache, Hugo Le Boité, Ramin Tadayoni, Béatrice Cochener, Mathieu Lamard, Gwenolé Quellec
TL;DR
Multimodal medical image classification needs comprehensive evidence on how different imaging and clinical modalities should be fused, beyond task- or field-specific surveys. This review synthesizes the literature into five fusion architectures, evaluates their suitability across scenarios, and discusses incomplete data and future Transformer-based approaches. Its conclusion is that fusion-method suitability depends on modality structure, dataset availability, and classification complexity, while clinical alignment remains essential.
Problem
Medical image classification lacks a comprehensive survey of multimodal fusion across clinical applications, despite the complementary information offered by multiple modalities.
Method
The review examines multimodal classification literature, clinical-modality complementarity, five fusion architectures, incomplete-data strategies, and emerging Transformer-based methods.
Results
Fusion-method suitability varies: single-level fusion is more robust than input fusion, hierarchical fusion suits four-class classification, and output fusion performs well with many unimodal datasets.
Takeaways & Limitations
The taxonomy helps match fusion architectures to multimodal classification scenarios and application domains, while Transformer-based fusion offers promising future directions.
Takeaways & Limitations
Multimodal fusion studies may not improve clinically relevant multi-level dementia classification when evaluations focus mainly on normal-control versus Alzheimer’s disease classification.
Abstract
from arXiv · showhide
Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.
1. Introduction
This review examines deep learning-based multimodal medical image classification, motivated by the complementary information provided by multiple modalities and gaps in comprehensive classification-focused surveys. It proposes a five-part fusion taxonomy, surveys datasets and architectures, and discusses challenges and future trends.
- Context: Multimodal imaging combines complementary information because a single modality often does not provide all information needed for accurate diagnosis.The review connects this motivation to potential improvements in diagnostic accuracy, reduced human error, and personalized treatment planning.
- Research gap: Existing medical imaging surveys addressed tasks such as fusion, synthesis, segmentation, and registration, but not multimodal medical image classification comprehensively.Some surveys focused on specific fields, including neurology and oncology, rather than broader medical applications.
- Review methodology: The literature review searched publications associated with 14 public multimodal image datasets and added 19 articles using private datasets for underrepresented organs, yielding 114 publications.Articles were selected from concatenated PubMed searches and handpicked based on abstracts addressing deep learning-based multimodal information fusion.
- Taxonomy: The review analyzes five deep learning fusion architectures: input, single-level, hierarchical, attention-based, and output fusion.These are organized within the broader input, intermediate, and output fusion stages; intermediate fusion is further subdivided into three types.
- Challenges and future trends: The paper discusses challenges and future trends, including architecture selection, incomplete multimodal data, representation learning, dedicated multimodal frameworks, and Transformer-based fusion.It highlights Transformer architectures because medical images contain sequence relationships that may be efficiently modeled through long-range dependencies.
2. Multimodal medical images
Medical multimodal imaging combines complementary information from different modalities, sequences, views, and clinical data to support diagnosis and classification. The review highlights common modality pairings, their distinct diagnostic information, and the availability and challenges of multimodal datasets.
- Complementary modalities: Different imaging modalities provide distinct structural, functional, metabolic, vascular, stiffness, or subsurface information for diagnosis.Examples include MRI/CT structural and functional information, PET metabolic information, Color Doppler vascular distribution, and elastography tissue stiffness.
- Common multimodal combinations: Frequently studied combinations include multiparametric MRI, MRI with PET, PET with CT, multimodal ultrasound, CFP with OCT, dermoscopic with clinical images, and image data with clinical data.These combinations target diseases across neurological, oncological, ophthalmic, dermatological, and other clinical applications.
- MRI sequences: Multiple MRI sequences capture complementary tumor characteristics, with T1 and T1c suited to tumors without edema and T2 and FLAIR suited to tumors with peritumoral edema.DWI detects acute strokes, while ADC and DWI provide information about water diffusion.
- Multimodal datasets: Multimodal datasets support fusion-method development, but privacy and imaging costs make comprehensive multimodal data difficult to obtain.Freely available resources such as ADNI, BraTS, and TCIA provide examples of datasets containing multiple imaging modalities and associated clinical or annotation data.
3. Multimodal classification pipeline
The multimodal classification pipeline standardizes preprocessing, fusion, feature extraction, classification, and evaluation despite varied terminology across publications.
- Pipeline overview: The review adopts a five-stage pipeline to clarify and standardize multimodal medical classification tasks.The pipeline is intended to encompass medical multimodal classification tasks despite differing terminology across publications.
- Pre-processing: Preprocessing prepares multimodal images through registration, cropping, denoising, resampling, normalization, ROI extraction, feature selection, and augmentation.Augmentation can help prevent overfitting and improve generalization, while feature selection reduces dimensionality and retains pertinent information.
- Pre-processing: Image registration aligns modalities such as MRI, CT, and PET in a common coordinate system, which is particularly important for input-level fusion.Challenges include limited supervised training data, modality-specific similarity measures, and anatomical changes across patients or time.
- Information fusion: Fusion is categorized by level as input, intermediate, or output fusion, and implemented through concatenation or merge operations.Concatenation forms a single tensor for the next step, whereas merge operations combine modalities through calculations and produce less data.
- Feature extraction and classification: Backbones extract modality features and final classifiers generate predictions from fused features or independent classification results.CNNs are widely preferred for medical-image feature extraction; fully connected layers and, in some cases, SVMs serve as final classifiers.
- Evaluation metrics: Evaluation uses TP, TN, FP, and FN to calculate metrics including sensitivity, specificity, accuracy, precision, F1 score, AUC, and Kappa.Kappa uses observed accuracy p0 and expected agreement pe, with pe based on actual and predicted category totals.
4. Multimodal classification networks
Multimodal classification networks use input, intermediate, and output fusion, with intermediate fusion comprising single-level, hierarchical, and attention-based architectures. Single-level fusion is most common, while hierarchical and Transformer-based attention fusion offer richer multimodal interaction but introduce registration or data-scale constraints.
- Network taxonomy: Five architectures are identified: input fusion, single-level fusion, hierarchical fusion, attention-based fusion, and output fusion.Single-level, hierarchical, and attention-based fusion are subcategories of intermediate fusion.
- Input fusion: Input fusion combines modalities before the deep-learning backbone through channel concatenation or pixel- or voxel-level merging.The fused data then enters a single-branch feature extractor, reducing parameters and deployment difficulty.
- Intermediate fusion: Single-level fusion extracts modality-specific features and fuses them once before classification, using either classic or network fusion structures.Its branch-based feature extraction supports unregistered or different-dimensional data.
- Attention-based fusion: Attention-based fusion uses Transformer backbones to extract and fuse modality features according to cross-modal attention relationships.Examples include AHM-Fusion, MMIF with cross-attention, 3MT with query-key-value cross-attention, and SBM with token exchange and adaptive intermediate-feature fusion.
- Intermediate fusion: Hierarchical fusion combines multimodal features at multiple dimensions while preserving modality-specific features, enabling fuller exploration of complementary information.Registration of multimodal data may affect classification because hierarchical fusion includes low-dimensional feature fusion.
- Output fusion and future directions: Output fusion combines unimodal results without feature fusion, making implementation relatively easy but limiting complementary-information use and potentially failing when modality performance differs substantially.Transformer-based multimodal fusion is promising, but medical datasets may be too small for the pretraining typically supporting Transformer performance.
5. Discussion
The review finds that fusion architecture suitability depends on modality structure, task complexity, available unimodal data, and clinical objectives, while incomplete modalities and redundant information remain challenges.
- Fusion-method comparisons: Single-level fusion improves overall accuracy over input fusion, while hierarchical fusion benefits four-class classification more than dichotomous classification.The review relates greater model complexity to more complex classification problems.
- Fusion-method suitability: Input, single-level, and hierarchical fusion are worth investigating for modalities with similar structures because registration is easier.Single-level and hierarchical methods fuse deeper features, which the review associates with improved classification performance.
- Fusion-method suitability: Single-level and attention-based fusion are preferable when modalities have widely varying structures or dimensions, whereas output fusion suits settings with many unimodal datasets.Output fusion can leverage pre-training of separate modality branches before multimodal fine-tuning.
- Combined fusion: Combining fusion methods can integrate registered and unregistered modalities or fuse data at input, feature, and decision levels.The review presents combined fusion as a promising strategy for multimodal medical classification.
- Incomplete multimodal data: Modality incompleteness is pressing because scanning costs and potential harm can lead patients to refuse multiple imaging examinations.In ADNI, all subjects had MRI data, but only about half had PET scans.
- Incomplete multimodal data: GAN-based methods generate missing modalities for classification, increasing available subjects and improving classification performance.The review describes GAN-based solutions as currently the most promising approach to multimodal incompleteness.
- Fusion limitations: Multimodal fusion may fail to improve classification when modalities are redundant or a modality poorly defines the target class.The review also notes that multimodal networks did not improve multi-level dementia classification in one cited evaluation.
- Future trends: The review identifies Transformers as promising because they efficiently model long-range sequence relationships, which are prevalent in medical-image representations.Few multimodal medical image classification papers are associated with public code.
6. Conclusion
The review establishes a taxonomy and surveys multimodal medical classification architectures across application scenarios. It highlights Transformer-based fusion and identifies interpretability, data limitations, and learning strategies as priorities for future work.
- Conclusion: The review examines five architectures: input fusion, single-level fusion, hierarchical fusion, attention-based fusion, and output fusion.It covers multimodal fusion scenarios and the application domains suited to different architectures.
- Conclusion: The review highlights Transformer-based multimodal fusion because sequence relationships are more prevalent in medical imaging applications.It also discusses representation learning and dedicated frameworks such as TorchMultimodal.
- Future research: Future research should investigate novel fusion and optimization methods, interpretable models, data imbalance and scarcity, and unsupervised or semi-supervised learning.The review also recommends applying multimodal fusion to emerging areas.