Source-linked AI summary
Review of multimodal machine learning approaches in healthcare
Felix Krones, Umar Marikkar, Guy Parsons, Adam Szmul, Adam Mahdi
TL;DR
Single-modality healthcare machine learning does not capture clinicians’ integration of diverse information sources. This review surveys multimodal data-fusion approaches, modalities, datasets and training strategies, concluding that fusion has shown potential to improve prediction across healthcare applications.
Problem
Single-modality healthcare machine learning is limited in replicating clinicians’ integration of diverse information sources for decision-making.
Method
The paper reviews healthcare multimodal data-fusion techniques, clinical modalities, datasets, applications and common training strategies.
Results
The review identifies modality-level fusion, multimodal datasets and multimodal deep learning applications across diagnosis, treatment planning and survival prediction.
Takeaways & Limitations
Multimodal deep learning has shown potential for improving predictive accuracy across healthcare areas, especially disease diagnosis and prognosis.
Takeaways & Limitations
Multimodal healthcare datasets remain limited because privacy, ethical, regulatory, collaboration, resource and modality-alignment challenges constrain their creation.
Abstract
from arXiv · showhide
Machine learning methods in healthcare have traditionally focused on using data from a single modality, limiting their ability to effectively replicate the clinical practice of integrating multiple sources of information for improved decision making. Clinicians typically rely on a variety of data sources including patients' demographic information, laboratory data, vital signs and various imaging data modalities to make informed decisions and contextualise their findings. Recent advances in machine learning have facilitated the more efficient incorporation of multimodal data, resulting in applications that better represent the clinician's approach. Here, we provide a review of multimodal machine learning approaches in healthcare, offering a comprehensive overview of recent literature. We discuss the various data modalities used in clinical diagnosis, with a particular emphasis on imaging data. We evaluate fusion techniques, explore existing multimodal datasets and examine common training strategies.
1 Introduction
Healthcare machine learning has been slower to adopt despite demand for personalised care, while multimodal methods aim to better reflect clinicians’ use of diverse patient information. This review synthesises multimodal fusion techniques, datasets and applications across medical conditions.
- Healthcare machine learning adoption has lagged despite increasing pressure on healthcare systems and demand for high-quality, personalised care.
- Clinicians integrate demographic, medical-history, laboratory, vital-sign and imaging information to contextualise assessments and treatment decisions.A study of radiologists found that 85% considered clinical context crucial for interpreting radiological examinations.
- Prior reviews have covered general healthcare applications, image-focused methods, fusion, architectures, optimisation and multimodal pipelines.
- Clinical prediction tasks use imaging, text, time-series and tabular modalities to support diagnosis, prognosis and treatment.
- This review examines healthcare data-fusion techniques, key modalities, development steps, multimodal datasets and studies spanning diverse medical conditions.
2 Data modalities
Healthcare modelling uses imaging, text, time-series and tabular data, each offering distinct clinical information and processing requirements. Imaging modalities provide varied structural, functional and tissue information, while NLP supports extraction from medical records.
- Clinical data modalities include imaging, text, time-series and tabular data used in healthcare modelling.Examples include X-rays, radiology reports, vital signs and patient metadata.
- Imaging data: DICOM files may be converted to JPEG, PNG or NIfTI formats for machine-learning preparation.NIfTI preserves essential metadata in the image header.
- Imaging data: X-rays are widely available, inexpensive 2D images, but limited spatial depth often makes them more suitable for screening.In England in 2022, almost half of 43.3 million imaging scans were X-rays.
- Imaging data: CT produces 3D volumes from reconstructed slices and offers high-resolution, fast, cost-effective imaging, but uses ionising radiation and distinguishes soft tissue less effectively.
- Imaging data: MRI is non-ionising and provides detailed soft-tissue views, although 2D acquisitions may produce orthogonal-direction aliasing artifacts requiring preprocessing.
- Imaging data: PET and SPECT reveal metabolic activity, blood flow or function and are often paired with CT or MRI for complementary clinical insights.They can support diagnosis, treatment monitoring, staging, prognosis and treatment planning.
- Imaging data: Ultrasound commonly produces 2D frames, with newer 3D and 4D imaging; frequency trades off resolution against penetration depth.Doppler overlays can depict blood flow and velocity.
- Text data: NLP extracts relevant information from medical records into structured formats, while active learning, transfer learning, augmentation and synthetic notes address labelling demands.
3 Model development
Multimodal model development follows standard machine-learning stages, with pre-training learning representations and fine-tuning adapting models to downstream tasks. The review also highlights preprocessing, supervised or self-supervised pre-training, and evaluation for robustness and generalisability.
- Model development: Multimodal healthcare development includes problem formulation, preprocessing, training and evaluation, with pre-training and fine-tuning forming two additional deep-learning stages.Pre-training learns salient, usually task-agnostic concepts; fine-tuning adapts them into task-specific information.
- Data preprocessing: Data preprocessing cleans, integrates, transforms and reduces raw data to address missing values, inconsistencies, unsuitable formats and excessive dimensionality.Examples include imputation, source integration, text embeddings, image resizing, scaling, normalisation and principal component analysis.
- Stage 1: Model pre-training: Pre-training learns modality-specific or joint representations through pretext tasks on large datasets, and may include multiple stages before downstream fine-tuning.The review distinguishes pre-training as providing the fine-tuning initialisation point, while fine-tuning updates the model for the required downstream task.
- Stage 1: Model pre-training: Supervised pre-training uses labelled classification or regression tasks, whereas self-supervised learning exploits inherent data properties without requiring specific labels.Self-supervised learning is grouped into discriminative and restorative pretext tasks, including contrastive learning among discriminative approaches.
- Stage 1: Model pre-training: Supervised pre-training is strong on very large datasets, while self-supervised models can outperform it on smaller datasets and avoid large-scale labelling costs.The cited comparison contrasts ImageNet-21k with ImageNet-1k and attributes the supervised-learning disadvantage to label-dependent information and labour costs.
- Model evaluation: Evaluation should test robustness, interpretability and generalisability across patient subgroups, conditions and geographical regions because deployment can involve out-of-distribution data.This is especially important across multiple healthcare sites, where data may vary substantially between facilities and populations.
4 Fusion approaches
Fusion approaches differ in when modalities are combined and how their features are merged. Early, intermediate, late and mixed strategies offer different trade-offs involving joint learning, missing data, modality interactions and architectural complexity.
- Early fusion: Early fusion combines raw or extracted features from multiple modalities before training a single model, with feature extraction ranging from simple aggregation to separate models.Examples include aggregating time-series data for XGBoost and stacking multiple images as CNN channels.
- Intermediate fusion: Intermediate fusion processes modalities with individual models, combines their extracted features, and back-propagates the loss through feature extraction models.A common strategy pre-trains modality-specific models, then concatenates outputs and trains a final model while unfreezing at least some weights during training.
- Late fusion: Late fusion runs distinct models on separate modalities and aggregates their predictions or features through an aggregation function or auxiliary model.It has been applied to combinations such as MRI with demographics, biomarkers or cognitive scores, and CT with demographic and clinical measurements.
- Late fusion: Late fusion handles missing patient data but cannot model cross-modal interactions, potentially losing information and complicating the choice of aggregation method.The review gives CLIP as an example that can classify images without text during inference through zero-shot learning.
- Mixed fusion: Mixed fusion combines strategies so modalities can be processed in tailored ways while modelling inter-modality dependencies that late fusion cannot capture.Designing mixed architectures is challenging because the point at which modalities should be combined requires careful consideration.
- Feature-level fusion: Feature-level fusion includes concatenation, operation-based combination and learning-based reconstruction into a shared informative space.Operation-based methods require same-shaped vectors, whereas concatenation appends vectors end-to-end.
- Mixed fusion: In healthcare, mixed fusion commonly uses images alongside other modalities, including histopathology with genomic tabular data and MRI with demographic, genetic and clinical data.These examples illustrate modality-specific processing before combining image-derived representations with non-image information.
5 Multimodal applications
The review surveys multimodal healthcare datasets and applications, finding that studies most often combine imaging with tabular data and use established fusion and learning strategies.
- Datasets: Open-source multimodal healthcare datasets commonly combine imaging with demographic information and electronic health records, sometimes including reports and vital-sign time series.
- Modalities: Most reviewed studies combined imaging data with tabular features such as demographics, clinical measurements, or genomic variables.
- Modalities: MRI was the most commonly used imaging modality, followed by PET, CT, and general X-ray imaging; dermoscopy predominated in skin-disease studies.
- Modalities: Free-text reports and raw time-series data were used relatively infrequently, with time-series studies typically relying on extracted features rather than raw sequences.
- Analysis methods: CNNs predominated for imaging, aggregation-based feature extraction for time series, and RNNs for clinical text.
- Training and fusion: Early and intermediate fusion were most frequent, while transfer learning addressed limited data and supervised learning remained dominant amid growing interest in semi-supervised and unsupervised methods.
6 Future research and conclusions
Future multimodal healthcare research is constrained by scarce, difficult-to-align datasets and by deployment, regulatory, interpretability, and ethical challenges. The review nevertheless identifies fusion methods, transfer learning, and emerging generalist models as important directions.
- Challenges: Limited multimodal datasets containing clinical notes, images, and time-series data for the same individual remain a major development challenge.
- Challenges: Privacy, ethical and regulatory complexity, limited collaboration, resource constraints, and modality alignment difficulties hinder multimodal dataset creation.
- Future models: Foundation models may support generalist medical AI capable of handling images, text, and structured information across medical tasks.
- Future models: Realising generalist medical AI requires improving interpretability and ensuring ethical and responsible deployment.
- Deployment: Multimodal AI must adapt to differing patient-management systems, clinical practices, and digital infrastructures while meeting stringent privacy requirements for varied data sources.
- Conclusions: The review highlights modality-level fusion, multimodal datasets, transfer learning, and growing semi-supervised and unsupervised learning as key insights.