PaperlayerView exact revision on arXiv

Source-linked AI summary

Artificial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data

Farida Mohsen, Hazrat Ali, Nady El Hajj, Zubair Shah

arXiv:2210.13462v1cs.LGcs.AIcs.CV

TL;DR

Prior reviews covered other forms of multimodal medical data fusion, while clinical data can support image interpretation. This scoping review synthesizes AI studies combining EHR and imaging, finding that these models generally outperformed single-modality models for the same diagnostic or prediction task.

  • Problem

    Prior reviews covered other multimodal scopes, while clinical data can support image interpretation; this review focuses on AI fusion of EHR and medical imaging.

    Source: Introduction · pp. 1–2

  • Method

    The authors conducted a PRISMA-ScR scoping review of AI studies fusing EHR and medical imaging, examining fusion strategies and clinical applications.

    Source: Introduction · p. 2, Preliminaries · p. 2, Methods · p. 4

  • Results

    Multimodal EHR and imaging models generally outperformed single-modality models for the same disease diagnosis or prediction task.

    Source: Principal findings · p. 13

  • Takeaways & Limitations

    The review concludes that simple fusion strategies can be effective but may not fully exploit the information embedded in EHR and imaging data.

    Source: Conclusion · p. 16

    Publication bias may have led to an overestimation of the benefits associated with multimodal data analysis.

    Source: Limitations · p. 16

Abstract

from arXiv · show

Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different clinical applications. More specifically, we focus on studies that only fused EHR with medical imaging data to develop various AI methods for clinical applications. We present a comprehensive analysis of the various fusion strategies, the diseases and clinical outcomes for which multimodal fusion was used, the ML algorithms used to perform multimodal fusion for each clinical application, and the available multimodal medical datasets. We followed the PRISMA-ScR guidelines. We searched Embase, PubMed, Scopus, and Google Scholar to retrieve relevant studies. We extracted data from 34 studies that fulfilled the inclusion criteria. In our analysis, a typical workflow was observed: feeding raw data, fusing different data modalities by applying conventional machine learning (ML) or deep learning (DL) algorithms, and finally, evaluating the multimodal fusion through clinical outcome predictions. Specifically, early fusion was the most used technique in most applications for multimodal learning (22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same task. Disease diagnosis and prediction were the most common clinical outcomes (reported in 20 and 10 studies, respectively) from a clinical outcome perspective.

Introduction

The introduction motivates combining EHR and imaging data because clinical context supports accurate image interpretation, and presents a scoping review of AI methods for fusing these modalities. It distinguishes the review by its scope and coverage and sets out questions on fusion, diseases, outcomes, algorithms, and datasets.

Motivation

  • Clinical context supports accurate image interpretation [6–9], and prior AI fusion studies reported better performance than single-source models across several diagnosis and prediction tasks [12–19].Source: Introduction · pp. 1–2

Review scope

  • The review covers 34 recent studies using conventional ML or DL to fuse EHR and imaging, extending beyond Huang et al.26’s DL-focused review of 17 studies through 2019.Other reviews addressed imaging modalities, omics, IoMT data, or multimodal EHR rather than the same scope.Source: Introduction · p. 2

Review questions

  • It asks which fusion strategies are used and most common, which diseases are studied, which clinical outcomes and ML algorithms accompany those strategies, and what public multimodal datasets exist.Source: Introduction · p. 2

Review aims

  • The review aims to clarify how ML models can align modalities for clinical tasks and identify gaps in multimodal EHR–imaging data resources.Source: Introduction · p. 2

Preliminaries

The review defines its EHR-and-imaging scope and uses fusion strategies to classify how studies combine the modalities’ features or predictions.

Preliminaries

  • The preliminaries identify EHR and imaging modalities, then introduce fusion strategies as the framework for examining multimodal studies.The review matches studies to a fusion taxonomy based on the stage at which features are fused in the ML model [26].Source: Preliminaries · p. 2, Fusion strategies · p. 4

Data modalities

  • The review treats EHR and clinical imaging as its two modalities, spanning structured or free-text records and imaging such as X-ray, MRI, PET, CT, or ultrasound.Multiple EHR forms or imaging types are each treated as a single modality when combined within that category.Source: Data modalities · p. 2

Fusion strategies

  • Early fusion joins modality features before one ML model, whereas late fusion trains separate modality-specific models and combines their predictions [26].Early fusion may join original or extracted features; late fusion can use voting or a meta-classifier.Source: Fusion strategies · p. 4
  • Joint fusion combines intermediate-layer neural-network features with other modalities in a final model, propagating its loss back to update feature extractors [26].Type I uses neural networks to extract features from all modalities; type II does not use neural networks to extract every modality’s features.Source: Fusion strategies · p. 4

Methods

The scoping review followed PRISMA-ScR and used a structured search, eligibility screening, verified data extraction, and narrative synthesis. The authors did not assess study quality.

Methods

Search strategy

  • The review searched four databases for English-language articles published from January 1, 2015, to January 6, 2022, using AI, multimodality-fusion, and imaging/EHR terms.Searches were conducted January 3–6, 2022, using titles and abstracts; only the first 110 relevant Google Scholar results were selected.Source: Search strategy · p. 4

Inclusion and exclusion criteria

  • Eligible studies used AI to fuse EHR with clinical imaging for any clinical application; single-modality, multi-omics, nonmedical, non-AI, and specified non-peer-reviewed or non-English works were excluded.Imaging included modalities such as MRI, PET, CT, and ultrasound; EHR could be structured or free text, and classical statistical regression models were not considered AI models in this review.Source: Inclusion and exclusion criteria · p. 4

Study selection

  • Two authors screened titles, abstracts, and eligible full texts in Rayyan29, resolving conflicts by discussion or consultation with a third author.Duplicates were removed before title-and-abstract screening.Source: Study selection · p. 5

Data extraction

  • One author extracted study, disease, outcome, modality, fusion, data-source, model, and evaluation details; two others verified the data, resolving disagreements by consensus.The extraction form was piloted on four studies.Source: Data extraction · p. 5

Data synthesis

  • The authors narratively synthesized studies across fusion strategies, diseases, clinical outcomes and ML algorithms, data sources and types, and evaluation mechanisms.They also recorded model implementation, feature extraction, single-modality evaluation, and data availability.Source: Data synthesis · p. 5

Study quality assessment

Results · Search results · Demographics of the studies

The review retained 34 studies for synthesis; most were journal articles, published in 2020–2022, and conducted in the USA or China.

Search results

  • 34 studies met the inclusion criteria and were selected for data extraction and synthesis.The search retrieved 1,158 studies; 971 remained after duplicates were removed, 44 reached full-text review, and 10 were excluded at that stage.Source: Search results · p. 5

Demographics of the studies

Data fusion strategies

The review mapped included studies to a fusion-strategy taxonomy, with early fusion the most common approach, followed by joint fusion and late fusion. Across strategies, studies combined modality-specific features or predictions and often reported better performance than single-modality models.

Early fusion

  • Early fusion was used in 22 of 34 studies (∼65%), usually after imaging features were converted to 1D to match EHR data.Nineteen studies used manual or software-based imaging features, while three used neural network-based feature extraction; the latter used CNNs and concatenated CNN-extracted and EHR features.Source: Early fusion · pp. 5, 7
  • Thirteen of 14 early-fusion studies comparing fusion with single-modality models reported better performance for fusion.Six studies reduced feature dimensions before concatenating modalities, using methods such as recursive feature elimination, Pearson correlation, Gini importance, Relief, backward elimination, and Gini ranking; three used PCA.Source: Early fusion · p. 7

Joint fusion

  • Joint fusion was the second most common strategy, used in 10 studies; seven of the studies evaluating it against a single modality reported improved performance.These studies used neural networks to process imaging and EHR data, commonly concatenating learned features before further classification or processing.Source: Joint fusion · pp. 7–8

Late fusion

  • Late fusion was used in only two studies; one study’s late-fusion models outperformed its early-fusion, joint-fusion, and single-modality models.The approaches aggregated modality-specific predictions by averaging probabilities or using a neural-network classifier; another study combined imaging-model predictions by voting before fusion with clinical models.Source: Late fusion · p. 8

Diseases

The included studies covered seven disease categories, with neurological disorders the most common focus.

Diseases

  • Neurological disorders were the most common focus (n = 16) among studies spanning seven categories: neurological, cancer, cardiovascular, Covid-19, psychiatric, eye, and other diseases.Table 3 presents the distribution of studies across these disease and disorder categories.Source: Diseases · p. 8

Clinical outcomes and machine learning models

The review groups EHR-imaging applications into diagnosis and early prediction, describing their fusion strategies and machine-learning models. Diagnosis was the most common outcome, while prediction covered disease, treatment, mortality, and survival outcomes.

Diagnosis

  • 20 (∼59%) studies addressed diagnosis, the most common clinical outcome in the review.Neurological disorders accounted for nine diagnostic studies, including four each focused on AD and MCI.Source: Diagnosis · p. 9
  • For diagnosis, 13 studies used early fusion, five used joint fusion, and two used late fusion; the late-fusion studies reported better performance than unimodal models.Methods included concatenating features for conventional classifiers or neural networks, learning joint multimodal representations, and combining model outputs by majority voting.Source: Diagnosis · pp. 9–10

Early Prediction

  • 14 (∼41.2%) studies addressed prediction: 10 disease prediction, two treatment-outcome prediction, and one each mortality and overall-survival prediction.Disease prediction meant estimating whether an individual might develop a disease in the future.Source: Early Prediction · pp. 10–11
  • For prediction, early fusion covered six disease-prediction studies and both mortality and survival studies, while four disease-prediction studies and one treatment-outcome study used joint fusion.The review reports multimodal models outperforming single-modality models in examples of disease, treatment, mortality, and survival prediction.Source: Early Prediction · p. 11

Datasets

The included studies used diverse imaging modalities alongside EHR data, most often structured EHRs, and relied more on private than public data sources; ADNI was the most-used public dataset.

Patient Data Types

  • Across the included studies, MRI and PET were the most utilized imaging modalities; PET images were used in 8 studies, mostly for AD diagnosis and prediction, while structured EHR data was most common (n = 32).Source: Patient Data Types · pp. 11, 13

Patient data Resources

  • 21 studies (~59%) used private data sources, while 13 used public datasets; ADNI was used in 7 of those 13 and was most popular for AD and MCI diagnosis and prediction.Other public datasets included NACC, MIMIC-IV, NCI, ADNI TADPOLE, and MR CLEAN Trial.Source: Patient data Resources · p. 13

Evaluation metrics

Evaluation metrics depend on the clinical task, with accuracy, AUC, sensitivity, specificity, F1-measure, and precision typically used for diagnosis and prediction.

Evaluation metrics

  • Diagnosis and prediction are typically evaluated with accuracy, AUC, sensitivity, specificity, F1-measure, and precision; Table 6 shows the measures used across included studies.The evaluation metrics are mainly dependent on the clinical task.Source: Evaluation metrics · p. 13

Discussion

The review found that combining EHR and medical imaging generally outperformed single-modality models for the same diagnostic or predictive task. It recommends trying fusion when multimodal data are available, while identifying limited data and varied study methods as important constraints.

Principal findings

  • EHR–imaging models generally outperformed single-modality models for the same diagnostic or predictive task.The review recommends attempting fusion when multimodal data are obtainable.Source: Principal findings · p. 13
  • Multimodal imaging–EHR resources remain limited, and studies often used only basic EHR features such as age and gender.ADNI was the most prominent dataset, with MRI and PET images from about 1,700 individuals plus clinical and genetic information; the review recommends developing similar datasets for other medical sources and exploring richer EHR features [12,15,44,51].Source: Principal findings · pp. 13, 16
  • Early fusion was the most commonly used approach, combining extracted 1D imaging features with EHR features in a single model.CNN-derived imaging features often performed better than manually or software-derived features; early fusion can learn cross-modal interactions with one model, though CNN feature extraction requires training multiple models.Source: Principal findings · p. 15
  • Joint fusion was the second most common approach and is preferred for large datasets and complementary modalities.Neural networks iteratively update feature representations by propagating loss across feature extractors to learn cross-modal correlations, but may be limiting with small datasets; the review cites AD diagnosis as a complementary-modality example [13–15,49].Source: Principal findings · p. 15
  • Late fusion is suited to independent modalities, small datasets, missing inputs, or high-dimensional concatenated features.Separate modality-specific models limit each model’s input size, and predictions can be aggregated even when one modality is unavailable; brain MRI pixels and MMSE scores for MCI diagnosis are an example [37].Source: Principal findings · p. 15
  • AD diagnosis and prediction [12–15,44,48,49] were the most common applications addressed in a multimodal setting; ML fusion consistently improved AD diagnosis, while clinicians struggled to diagnose accurately and reliably even with multimodal data [26].Source: Principal findings · pp. 15–16

Future directions

  • Future research could combine EHR and imaging with multi-omics and environmental data, while sharing data across institutions to broaden population coverage.Federated learning can support secure, multicenter collection and model training without directly pooling data [67,68].Source: Future directions · p. 16

Limitations

  • The review was limited by its 2015–2022, English-language, EHR–imaging scope, possible publication bias, and heterogeneous tasks and metrics that hinder direct statistical comparison.Not all included studies reported confidence bounds, further limiting statistical comparisons.Source: Limitations · p. 16

Conclusion

This review surveyed AI methods that fuse EHR and medical imaging data, finding growing interest in multimodal medical ML but continued reliance on relatively simple fusion strategies. The authors suggest that further development could support more comprehensive analysis and clinical decision-making.

Conclusion

  • The review surveyed EHR–medical imaging fusion strategies, clinical tasks, models, diseases, publicly accessible datasets, and directions for future research.It focuses on multimodal medical machine learning combining EHR with medical imaging data.Source: Conclusion · p. 16
  • Growing interest in multimodal medical data coexists with reliance on relatively simple fusion strategies that can be effective but may not fully exploit the modalities’ rich information.The authors note that fast-changing AI methods or studies using combinations of strategies may fall outside the review’s definition of fusion strategies.Source: Conclusion · p. 16
  • Further development of multimodal medical data analysis could provide greater support for clinical decision-making.The review presents this as a prospective benefit of the field’s development.Source: Conclusion · p. 16

Author contributions statement

The authors shared conceptualization and supervision, while F.M. led data curation, synthesis, and original drafting; all authors approved the final manuscript.

  • F.M., H.A., and Z.S. contributed to conceptualization; F.M. and H.A. administered the project, while F.M. curated data, synthesized findings, and drafted the original manuscript.Source: Author contributions statement · p. 20
  • H.A. and N.E. reviewed and edited the manuscript, Z.S. and H.A. supervised the study, and all authors read and approved the final manuscript.Source: Author contributions statement · p. 20

Related papers

Loading 2210.13462v1…