PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications

Valerio Guarrasi, Fatih Aksu, Camillo Maria Caruso, Francesco Di Feola, Aurora Rofena, Filippo Ruffini, Paolo Soda

arXiv:2408.02686v1cs.LGcs.AI

TL;DR

Biomedical intermediate-fusion research is assessed through a synthesis of its methods and training strategies. The review reports that many surveyed models lack robustness to missing modalities, and identifies reporting gaps that constrain methodological transparency.

  • Problem

    Missing modalities are a major concern in multimodal learning, particularly in medical settings where their combination with data scarcity limits available data.

    Source: §3.7.3 · p. 60

  • Method

    The review synthesizes biomedical intermediate-fusion studies, including their fusion mechanisms and training strategies.

    Source: §3.4.6 · p. 48, §3.7.1 · p. 59

  • Results

    45 of 54 surveyed articles showcase models that lack robustness to missing modalities.

    Source: §3.7.3 · p. 60

  • Takeaways & Limitations

    A thorough MDL experimental setup should include resampling, unimodal and multimodal comparisons, comparisons of fusion strategies, statistical tests, and external validation.

    Source: §3.7.5 · p. 63

    Gaps in reporting input and marginal representation sizes, explainability, and fusion-strategy comparisons reduce methodological transparency.

    Source: §4.1.1 · pp. 65–66

Abstract

from arXiv · show

Deep learning has revolutionized biomedical research by providing sophisticated methods to handle complex, high-dimensional data. Multimodal deep learning (MDL) further enhances this capability by integrating diverse data types such as imaging, textual data, and genetic information, leading to more robust and accurate predictive models. In MDL, differently from early and late fusion methods, intermediate fusion stands out for its ability to effectively combine modality-specific features during the learning process. This systematic review aims to comprehensively analyze and formalize current intermediate fusion methods in biomedical applications. We investigate the techniques employed, the challenges faced, and potential future directions for advancing intermediate fusion methods. Additionally, we introduce a structured notation to enhance the understanding and application of these methods beyond the biomedical domain. Our findings are intended to support researchers, healthcare professionals, and the broader deep learning community in developing more sophisticated and insightful multimodal models. Through this review, we aim to provide a foundational framework for future research and practical applications in the dynamic field of MDL.

1. Introduction

Intermediate fusion combines modality-specific features during learning, while the review addresses biomedical data heterogeneity and high dimensionality. It formalizes current methods and their challenges, and distinguishes its focus from prior surveys through a structured notation intended to extend beyond biomedicine.

1. Introduction

  • Intermediate fusion combines modality-specific features at the feature-extraction stage, where a shared representation can preserve and use each data type’s distinct qualities.The review describes this as especially relevant when biomedical modalities such as imaging and genomic information need to interact.Source: §1 · pp. 3–4
  • Biomedical MDL faces heterogeneous, high-dimensional data, alongside interpretability, data-availability, modality-balancing, and computational-cost challenges.The introduction notes that data scarcity or privacy concerns can limit data availability, while sparse or lower-quality modalities complicate balancing contributions.Source: §1 · pp. 3–4

1.1. Objective of the Review

  • The review analyzes and formalizes biomedical intermediate-fusion methods, addressing techniques, limitations, future directions, and notation intended to extend beyond biomedicine.Its research questions include how the notation and categorization might apply to other deep-learning domains.Source: §1.1 · pp. 4–5

1.2. Comparison with Previous Surveys

  • Prior surveys have covered multimodal learning broadly or biomedical integration without focusing on intermediate fusion; this review adds structured notation with potential cross-domain applicability.Some biomedical surveys emphasized application categories or only a limited set of modalities.Source: §1.2 · p. 7

2. Methodology

The review targets peer-reviewed English-language studies of biomedical MDL using end-to-end intermediate fusion. It searched multiple databases and screened records in stages, ultimately including 54 manuscripts.

2.1. Inclusion/exclusion criteria

  • The review included English-language original studies of biomedical applications using multiple modalities and end-to-end deep-learning intermediate fusion, excluding reviews and early- or late-fusion studies.It excluded non-peer-reviewed sources and studies without multimodal fusion.Source: §2.1 · pp. 7–8

2.2. Search Strategy

  • Only 2% of publications were categorized as biomedical MDL via intermediate fusion, which the review describes as a relatively underdeveloped area.The search covered PubMed, IEEE Xplore, Scopus, and Google Scholar, using synonyms across MDL, biomedical, and fusion categories.Source: §2.2 · pp. 8–9, Figure 1 · p. 10

2.3. Study Selection

  • The staged selection process resulted in 54 manuscripts on biomedical MDL via intermediate fusion, after title-and-abstract screening and full-text assessment by two independent reviewers.Disagreements were resolved through discussion, and exclusions at full-text assessment were recorded.Source: §2.3 · pp. 9–11

3. Intermediate Fusion in Biomedical Applications

The review organizes intermediate-fusion models as a pipeline from modality inputs to a target output, with learning strategies also examined.

3. Intermediate Fusion in Biomedical Applications

  • The model-component overview follows the pipeline from modality inputs through unimodal feature extraction, fusion, multimodal processing, and target output.The review also analyzes the learning strategies used to train the model.Source: §3 · pp. 11–12

3.1. Definitions

Intermediate fusion extracts modality-specific features, combines them into a shared representation, and processes that representation to produce an output. Backpropagated loss trains the unimodal and multimodal components.

3.1. Definitions

  • Each modality is processed by a specialized unimodal network to extract features, which the fusion module combines into a multimodal representation for a subsequent network.The figure depicts modality-specific features retaining their colors before blending at fusion.Source: §3.1 · p. 12, Figure 3 · p. 13
  • Backpropagation of the loss trains both unimodal and multimodal modules to capture intra- and inter-relationships in multimodal data.The target guides loss-function choice and iterative improvement across the unimodal, fusion, and multimodal modules.Source: §3.1 · pp. 12–13
  • The target y is the network output, such as a diagnosis or prognosis in biomedical applications.The framework comprises modalities, unimodal modules, a fusion module, a multimodal module, and a target.Source: §3.1 · pp. 12–13

3.2. Modalities

The review considers six macro-modalities in intermediate-fusion studies, spanning medical images, tabular data, text, time series, video, and audio. Imaging and tabular data dominate the analyzed articles, while text, audio, and video account for smaller shares.

3.2. Modalities

  • The review describes these modalities as complementary perspectives that can be combined for a more comprehensive view of medical conditions.Examples include integrating medical images with report text or laboratory data.Source: §3.2 · p. 14, §3.2.1 · p. 16

3.2.1. Type of Modalities

  • The six macro-modalities include imaging, tabular data, text, time series, video, and audio, with imaging covering diagnostic and monitoring images and tabular data including clinical and genomics information.Time series include electrical, clinical, and tactile signals; video includes temporal imaging and RGB recordings.Source: §3.2.1 · pp. 14–15
  • Imaging and tabular data represent 37% and 35% of modalities, respectively; MRI is the most prevalent imaging type at 33%, followed by CT at 26% and X-rays at 21%.Among tabular modalities, clinical data account for 65%, genomics for 29%, and chemical features for 6%.Source: §3.2.1 · pp. 15–16
  • Time series make up 15% of modalities, while text, audio, and video account for 5%, 4%, and 4%, respectively.Free-text reports represent 83% of text, speech 60% of audio, and RGB video 60% of video.Source: §3.2.1 · pp. 15–16

3.2.2. Data Origin

Across the reviewed articles, real data predominate and public datasets are widely used. However, no dataset appears in most studies, limiting cross-study assessment of overall model effectiveness.

3.2.2. Data Origin

  • Only 2 of 54 articles incorporated GAN-generated synthetic data alongside real data.The review identifies synthetic-data generation as an open technical challenge in biomedical research.Source: §3.2.2 · p. 17

3.2.3. Data Source

  • About 30% of articles used private datasets, while the remaining real-data sources mostly came from public datasets.Public datasets can avoid the costs, time, and privacy constraints associated with acquiring and managing private data.Source: §3.2.3 · p. 17
  • The articles used 52 distinct public datasets, and the most common, COBRE and MIMIC-CXR, appeared in only 4 articles each.Table 1 covers publicly available datasets used in at least two articles.Source: §3.2.3 · p. 17, Table 1 · p. 18
  • No benchmark dataset was used by a majority of studies, making it difficult to establish the overall effectiveness of the models.Source: §3.2.3 · p. 17

3.2.4. Sample Size

The review finds a mix of dataset sizes, with many moderate-sized datasets and a substantial group exceeding 1,501 samples. Imbalanced and highly imbalanced datasets together outnumber balanced ones, while only a minority of affected articles report mitigation.

3.2.4. Sample Size

  • 27 datasets had more than 1,501 samples, while 7 had fewer than 100; the review notes that intermediate fusion preferably requires large datasets.The surveyed sample-size ranges also included 24 datasets with 101–500 samples, 14 with 501–1,000, and 6 with 1,001–1,500.Source: §3.2.4 · pp. 18–19
  • The review notes that small datasets may lead to overfitting and suggests augmentation or transfer learning as possible mitigations.It also reports that large datasets may support robust models capable of capturing diverse variations and potentially yielding more generalizable outcomes.Source: §3.2.4 · p. 19

3.2.5. Class Distribution

  • For classification tasks, the review assessed dataset balance using the Gini-index [70]; its binary-classification example linked balanced, imbalanced, and highly imbalanced datasets to corresponding class-prior ranges.The binary class-prior ranges were 50-50 to 41-59 for balanced, 40-60 to 31-69 for imbalanced, and 30-70 to 0-100 for highly imbalanced datasets.Source: §3.2.5 · p. 19
  • 24 datasets were imbalanced and 13 highly imbalanced, together exceeding the 20 balanced datasets.Only 37% of articles using imbalanced or highly imbalanced datasets reported addressing the issue with specific techniques.Source: §3.2.5 · p. 20

3.2.6. Missing Modalities

Missing modalities are common in biomedical multimodal datasets, but few reviewed studies specifically design for them; others impute missing values or leave the issue unaddressed.

3.2.6. Missing Modalities

  • Only 4 articles developed models specifically for missing modalities, while 6 used imputation and the rest used complete-data datasets or removed affected samples.Imputation used neural networks or machine-learning algorithms, or mean or predefined values.Source: §3.2.6 · p. 21

3.2.7. Modalities Combination

The review finds that most studies use two modalities, with imaging paired frequently with tabular data; the common combinations differ across bimodal and trimodal studies.

3.2.7. Modalities Combination

  • 74% of articles used bimodal datasets, 20% trimodal datasets, and 6% high-modal datasets with more than three modalities.Imaging and tabular data were the most prevalent bimodal pairing, appearing in 16 articles (40% of bimodal articles).Source: §3.2.7 · p. 21
  • Two imaging modalities appeared in around 13% of bimodal articles (5 studies), including different MRI variants and CT with PET.The authors speculate that these modalities may provide complementary anatomical and functional tissue information.Source: §3.2.7 · p. 22
  • Imaging and text appeared together in 10% of bimodal articles (4 studies), most often as X-rays paired with free-text reports.The text reports provide medical interpretations alongside X-ray images.Source: §3.2.7 · p. 22
  • Two imaging modalities plus one tabular modality was the most recurring trimodal combination, accounting for 55% of trimodal articles.Examples predominantly paired MRI variants with genomics or clinical data.Source: §3.2.7 · p. 22
  • No discernible combination trend emerged among high-modal articles because few articles fell into this category.The review directs readers to Supplementary Material A for article-level combinations.Source: §3.2.7 · p. 22

3.2.8. Feature Extraction

Feature extraction prepares model inputs before unimodal processing and can change their modality category; raw data remains prevalent, especially for imaging and tabular data.

3.2.8. Feature Extraction

  • Feature extraction transforms raw data into representative features used as unimodal-model inputs, which may belong to a different modality category than the original data.The review distinguishes this preparation step from model training and from the unimodal module itself.Source: §3.2.8 · p. 22
  • After feature extraction, tabular inputs increased from 2 articles with two original tabular modalities to 8, while graph inputs emerged in 2 articles.Text became tabular in 3 of 5 articles; some trimodal and high-modal inputs also became entirely tabular.Source: §3.2.8 · pp. 23–24, Figure 6 · p. 23
  • Input features are raw when no extraction is applied, handcrafted when experts engineer them, or learned when a model extracts them from raw data.Learned extraction trains a model through forward and backward passes.Source: Figure 7 · p. 24, §3.2.8 · p. 24
  • Raw data was used in 80% of imaging cases and 87% of tabular cases; imaging also had 16% handcrafted and 4% learned extraction, while tabular had 13% handcrafted.The tabular handcrafted cases concerned chemical features.Source: §3.2.8 · pp. 25–26
  • Raw data predominated across the other macro-modalities too: time series 50%, text 57%, video 40%, and audio 60%.Time series also used handcrafted extraction in 45% and learned extraction in 5%; text used learned features in 43%, video used learned and handcrafted features in 40% and 20%, and audio used each in 20%.Source: §3.2.8 · p. 26
  • Overall, raw data was more prevalent than extracted features, particularly for imaging (80%) and tabular data (87%).The review notes less skewed raw-use percentages for time series, text, audio, and video.Source: §3.2.8 · p. 26

3.3. Unimodal Module

The review characterizes unimodal modules by architecture, modality, and representation size, finding that architecture choices track input types and that modules can condense or expand features before fusion.

3.3. Unimodal Module

  • Separate unimodal branches extract modality-specific marginal representations, which a global loss derived from fused features can adapt toward the overall learning objective.The representations are passed to a fusion module to create a common multimodal representation.Source: §3.3 · pp. 26–27, §3.3.4 · p. 34

3.3.1. Architecture Types

  • Among 54 articles, 24 used homogeneous architectures and 30 heterogeneous ones; CNNs appeared in 39 articles, FCNNs in 24, and RNNs in 10.Transformers, graph neural networks, and autoencoders appeared in 4, 3, and 2 articles, respectively.Source: §3.3.1 · p. 28

3.3.2. Modality Distributions

  • CNNs served 63 modules, including 38 imaging modules; FCNNs served 41, including 32 tabular modules, while five modules without an architecture used raw tabular data.RNNs served 11 modules, five of them time-series modules.Source: §3.3.2 · pp. 29–30, §3.3.1 · p. 29, Figure 10 · p. 30

3.3.3. Architectural Combinations

  • CNN–CNN (19 articles) and CNN–FCNN (13) were the most prevalent architecture pairings; GNNs appeared only in combinations with other GNNs.The review attributes GNN pairing to the specialized nature of graph-structured data and maintaining consistent graph representations.Source: §3.3.3 · p. 31, Figure 11 · p. 32

3.3.4. Dimensionality Transformation

  • Marginal representations can condense inputs when smaller, expand them when larger, or retain their size; only 39% of surveyed articles reported input and representation sizes for at least one modality.The majority of reported modules condensed information by a factor of one to two.Source: §3.3.4 · pp. 31–32, Figure 12 · p. 33
  • CNNs, particularly for imaging, showed greater condensation, while FCNNs with tabular data showed smaller condensation ratios or expansions.In one example, equal 512-dimensional representations yielded an imaging ratio of 0.01 and a tabular expansion ratio of 28.44.Source: §3.3.4 · pp. 32–33, Figure 13 · p. 34
  • Among 18 articles reporting every modality’s representation size, only five used unequal sizes; equal sizes were intended to avoid bias, though their suitability across modalities is uncertain.The review concludes that condensation or expansion depends on both input modality and architecture.Source: §3.3.4 · pp. 33–34, Figure 14 · p. 35

3.4. Fusion Module

This section formalizes intermediate fusion as the integration of modality representations into a unified representation and introduces notation and taxonomies for describing fusion choices. It reviews fusion inputs, repetition, timing, and operations, including reported patterns in fusion methods and feature dimensions, while noting uncertainty about whether equal dimensions suit different modalities.

3.4. Fusion Module

  • The fusion module combines hidden representations from unimodal modules into a unified multimodal representation, enabling intermediate interaction between modalities.Unlike early fusion, it does not integrate modalities at the raw-data level; unlike late fusion, it does not rely on independently learned disjoint spaces.Source: §3.4 · p. 35

3.4.1. Systematic Notation for Multimodal Fusion

  • The proposed notation describes a fusion’s nature, localization, and timing, and represents the fusion structure as a directed bipartite graph of modality and fusion nodes.Its components include fusion ordering, input modalities or previous fusion outputs, input layer depth, operation type, and whether the fusion produces the final representation.Source: §3.4.1 · pp. 36–37

3.4.2. Fusion Module Taxonomy

  • The taxonomy classifies fusion by what is combined, how many fusions occur, when they occur, and how they are performed, while remaining extensible beyond the reviewed methods.The authors note that the taxonomies may not cover every possible intermediate-fusion methodology, although the notation is intended to simplify analysis and extension.Source: §3.4.2 · p. 39

3.4.3. What?

  • The “what” classification identifies modality types and makes visible fusions between modalities and earlier multimodal representations before the model’s final output.The listed modality categories include imaging, tabular, text, time-series, video, and audio.Source: §3.4.3 · pp. 40–41

3.4.4. How Many?

  • 35 of 54 reviewed studies used a single fusion, compared with 19 of 54 using multiple fusions.The authors relate the preference for simpler integration to diverse biomedical data and the central role of data curation and processing.Source: §3.4.4 · p. 41

3.4.5. When?

  • 11 multiple-fusion studies used sudden timing, versus 5 gradual and 3 multi-flow; single-sudden models were asynchronous in 24/35 cases, while multiple-fusion models split 10 asynchronous and 9 synchronous.Single-fusion approaches are all sudden; sudden fusion combines modalities at once, gradual fusion adds modalities across successive fusions, and multi-flow combines distinct fusion sequences into one representation.Source: §3.4.5 · pp. 41, 43, 44, Table 3 · p. 44

3.4.6. How?

  • Fusion operations comprise concatenation, attention, tensor-operation, calibration, and knowledge-sharing; concatenation predominates, used by 29 of 35 single-fusion studies.Concatenation joins features into a larger representation and can raise dimensionality concerns; tensor operations support varied output dimensions but may lack dynamic mechanisms for rich modality interaction. Attention weights features according to inter-modality relevance and complementarity.Source: §3.4.6 · pp. 45–46
  • 82% of single-fusion approaches used concatenation, versus 9% tensor operations, 6% knowledge-sharing, 3% attention, and 0% calibration.Figure 17 shows the distribution of fusion operations across single and multiple approaches.Source: §3.4.6 · pp. 48–49, Figure 17 · p. 49
  • In multiple-fusion approaches, concatenation accounted for 46%, tensor operations 32%, and attention 7%, compared with 3% attention in single approaches.The review attributes the greater use of tensor operations to diverse operations across the fusion module, and describes attention as effective across stages while retaining some explainability through attention maps.Source: §3.4.6 · p. 49, Figure 17 · p. 49
  • Calibration appeared only in multiple architectures, accounting for 13% of their fusion methods.It generates a reduced-dimensional shared representation as a calibration signal, which is then fused with the inputs; the signal may be generated once or repeatedly to build increasingly abstract representations.Source: §3.4.6 · pp. 47, 50
  • Knowledge-sharing accounted for 6% of single and 2% of multiple approaches, remaining under-explored and more often supplementary in multiple-fusion methods.These methods learn fused representations through optimization rather than direct interactions between unimodal components.Source: §3.4.6 · pp. 48, 50

3.4.7. Unimodal Module Influence on the Fusion Module

  • Only 18 of 54 articles reported feature sizes for all modalities, and 13 of those used equal dimensions, split between 7 homogeneous and 6 heterogeneous unimodal modules.Equal feature sizes may help avoid modality bias, but the review notes that their suitability for modalities with different characteristics remains unclear.Source: §3.4.7 · pp. 50–51
  • Among equal-dimension approaches, 12 of 13 used single sudden fusion, while 19 of 24 homogeneous approaches were synchronous and 28 of 30 heterogeneous approaches were asynchronous.The review links these patterns to uniform, balanced representation spaces in homogeneous synchronous approaches and varied architectures, data types, and operations in heterogeneous asynchronous approaches.Source: §3.4.7 · p. 51

3.5. Multimodal Module

The multimodal module processes the fused representation toward the target, with its role and size varying across architectures. Across the reviewed studies, unimodal modules are usually larger, while the fusion module generally performs the main integration work.

3.5. Multimodal Module

  • FCNNs predominate in both single-fusion methods (29 of 35) and multiple-fusion methods (17 of 19).The review describes FCNNs as mapping the feature space to the target space for the final loss.Source: §3.5 · pp. 52–53
  • CNN multimodal modules appear in 3 works, while recurrent layers capture dependencies and 2 approaches use channel or spatial attention.The CNN implementations include two with convolutional and final fully connected layers and one with convolutional layers only.Source: §3.5 · p. 53
  • 45 of 54 studies use larger unimodal modules, compared with 7 using comparable sizes and 2 using larger multimodal modules.The size comparison is based on the module dimensions reported in the review.Source: Figure 19 · p. 54, §3.5 · p. 55
  • Among M<U approaches, 37 use FCNN multimodal modules and 4 use attention; the fusion module carries most representation learning.The multimodal module typically fine-tunes fused representations for the downstream task.Source: §3.5 · p. 55
  • When the multimodal module is equal to or larger than unimodal modules, it can further transform fused data; overall, fusion is the main integration point.The multimodal module is often used to make the architecture compatible with downstream tasks.Source: §3.5 · pp. 52, 55

3.6. Target

The reviewed studies use supervised tasks across several generic and medical task categories, with classification predominating. Oncology, mental health, and pneumology form the largest disease-area clusters, and their modality patterns differ.

3.6. Target

  • All selected articles perform supervised tasks on annotated datasets; 47 of 54 use classification, including 26 diagnostic and 12 prognostic studies.The review also identifies regression, segmentation, and object detection among the generic task types.Source: §3.6 · pp. 55–56
  • Three articles use regression for prognosis, while one classification study addresses drug discovery and another question answering.Two works use multitask approaches: one combines segmentation with pixel-level cancer-likelihood prediction, and one combines classification with bounding-box regression.Source: §3.6 · p. 56
  • The largest disease-area clusters are oncology (22 of 54), mental health (14 of 54), and pneumology (9 of 54), with modality patterns varying by area.Oncology relies mostly on genomics, mental health on MRI and electrical data, and pneumology on clinical data and X-ray imaging.Source: §3.6 · p. 56

3.7. Learning

The section surveys practical training and deployment choices for intermediate-fusion multimodal models. It reports limited uptake of transfer learning and missing-modality robustness, sparse explainability and rigorous evaluation, alongside comparisons favoring multimodal and intermediate-fusion approaches.

3.7.1. Transfer Learning

  • 46 of 54 studies trained from scratch, while 11 of 54 used pretraining, including 9 that pretrained unimodal modules and 2 that pretrained the entire model.Pretraining in a related domain is favored; the review says generic ImageNet pretraining does not guarantee superior performance or more meaningful representations.Source: §3.7.1 · pp. 57–58
  • All transfer-learning articles used at least one pretrained CNN; two also used pretrained RNNs, and one each used an FCNN and a transformer.The CNNs included ResNet-based models, InceptionV3, and EfficientNet.Source: §3.7.1 · p. 59

3.7.2. Multimodal optimization

  • 51 articles used indirect losses, while direct optimization strategies targeted intra-modal or inter-modal effects on multimodal representations.Examples include triplet loss to separate shared and modality-specific features, confidence weighting before fusion, and functional-entropy regularization.Source: §3.7.2 · pp. 59–60

3.7.3. Missing Modalities

  • 45 of 54 models lacked missing-modality robustness, while only 5 of 54 explicitly claimed to handle missing modalities.Approaches included permutation-invariant attention, recurrent aggregation, modality plug-in/out, random zero imputation, and transferring graph-manifold knowledge to incomplete samples.Source: §3.7.3 · pp. 60–61

3.7.4. Explainability

  • 34 of 54 articles did not address explainable AI or mentioned it only as future work.The review links explainability in medical applications to understanding model decisions and uncovering patterns or associations.Source: §3.7.4 · pp. 61–62
  • The review distinguishes unimodal explanations of salient within-modality features from multimodal explanations of modality importance and interactions.CNNs were the most explained architecture type; only three publications explained RNNs, two transformers, one GNN, and one autoencoders.Source: §3.7.4 · p. 62, Table 4 · p. 63

3.7.5. Experimental Configuration

  • The recommended experimental setup combines robust resampling, unimodal and fusion-strategy comparisons, statistical testing, and external validation.Only 19 articles used robust resampling; average scores should be reported with standard deviations because higher means alone may not indicate a better or more stable approach.Source: §3.7.5 · pp. 63–64
  • 40 works reported multimodal superiority over unimodal models, while 24 reported intermediate fusion outperforming early and late fusion.Among the 40 multimodal-superiority works, 29 omitted standard deviations; 16 of 24 intermediate-fusion wins did likewise. Only 14 articles used statistical tests, 5 external validation, 4 all five recommended criteria, and 11 released code.Source: §3.7.5 · p. 64

4. Discussion and Conclusion

The review closes by identifying reporting and methodological gaps in biomedical intermediate-fusion research, then outlining research directions and practical implications. Its discussion emphasizes transparency, broader modality and scale coverage, and stronger evaluation and deployment practices.

4.1.1. Limitations

  • Missing input and representation dimensions, explainability details, fusion-strategy comparisons, validation, statistical testing, and code links constrain transparency, reproducibility, and assessment of generalizability.The review calls for more rigorous reporting standards to address these omissions.Source: §4.1.1 · pp. 65–67

4.1.2. Potential Research Gaps

  • Current work is concentrated on bimodal imaging–tabular combinations, with multi-scale integration and standardized public benchmark datasets also identified as gaps.The review describes imaging plus tabular data as the typical combination; it highlights the lack of datasets that enable comparison across biomedical domains.Source: §4.1.2 · pp. 67–68
  • Marginal representation dimensions lack a systematic selection approach, while fusion operations and multimodal-module architectures remain underexplored and often lack theoretical grounding.The review notes that equal-dimensional representations across modalities may be unjustified and that studies predominantly use FCNNs in the multimodal module.Source: §4.1.2 · pp. 68–69
  • The review identifies gaps in interpretability, transfer across biomedical domains, robustness to missing modalities, resource-constrained deployment, and ethical safeguards.It notes that concatenation-based fusion does not make models robust to missing modalities and calls for transfer-learning and domain-adaptation strategies.Source: §4.1.2 · pp. 70–71

4.1.3. Future Research

  • Future work should also develop richer fusion and task-mapping architectures, optimize modality-specific representation dimensions, and expand data and evaluation coverage.The review specifically recommends diverse modalities, multi-scale data, and standardized benchmark datasets.Source: §4.1.3 · pp. 72–73
  • Proposed research directions include transfer learning and domain adaptation, privacy-preserving and federated learning, real-time clinical decision support, and multimodal foundation models.The review also calls for trimodal and higher-modal combinations, multi-scale integration, benchmark datasets, and improved interpretability.Source: §4.1.3 · pp. 73–75

4.2. Implications for Practice

  • The review presents intermediate fusion as having practical potential for diagnosis, personalized care, data use, biomedical research, and resource allocation, while stressing ethical and privacy considerations.These are stated implications rather than results from a clinical evaluation in the supplied passages.Source: §4.2 · pp. 75–76

Related papers

Loading 2408.02686v1…