PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Fusion on Low-quality Data: A Comprehensive Survey

Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen, Di Hu, Changqing Zhang

arXiv:2404.18947v3cs.LGcs.AI

TL;DR

Reliability of multimodal fusion under low-quality data remains insufficiently explored, although conventional fusion may fail in such settings. This survey synthesizes challenges and advances through a data-centric taxonomy, identifying open problems in noisy and incomplete multimodal learning.

  • Problem

    The reliability of multimodal fusion under low-quality data remains largely unexplored, and conventional fusion may fail on noisy, imbalanced, or corrupted inputs.

    Source: §1 · p. 1

  • Method

    The paper surveys challenges and recent advances in low-quality multimodal fusion and organizes them into noisy, incomplete, imbalanced, and quality-varying data categories.

    Source: §1 · pp. 1–2, §6 · p. 16

  • Results

    The survey finds that noisy-data methods often target specific scenarios, while general noise patterns and learning paradigms remain relatively unexplored.

    Source: §2.3 · p. 4

  • Takeaways & Limitations

    The discussion identifies noise correlations, noisy-clean modality complementarity, high-level semantic noise, imputed-instance quality, and tasks beyond fusion as open directions.

    Source: §2.3 · p. 4, §3.3 · p. 8, §6 · p. 16

    Noisy-data methods typically focus on specific scenarios, and quality assessment of imputed instances is commonly overlooked.

    Source: §2.3 · p. 4, §3.3 · p. 8

Abstract

from arXiv · show

Multimodal fusion focuses on integrating information from multiple modalities with the goal of more accurate prediction, which has achieved remarkable progress in a wide range of scenarios, including autonomous driving and medical diagnosis. However, the reliability of multimodal fusion remains largely unexplored especially under low-quality data settings. This paper surveys the common challenges and recent advances of multimodal fusion in the wild and presents them in a comprehensive taxonomy. From a data-centric view, we identify four main challenges that are faced by multimodal fusion on low-quality data, namely (1) noisy multimodal data that are contaminated with heterogeneous noises, (2) incomplete multimodal data that some modalities are missing, (3) imbalanced multimodal data that the qualities or properties of different modalities are significantly different and (4) quality-varying multimodal data that the quality of each modality dynamically changes with respect to different samples. This new taxonomy will enable researchers to understand the state of the field and identify several potential directions. We also provide discussion for the open problems in this field together with interesting future research directions.

1 INTRODUCTION

Conventional multimodal fusion may fail on low-quality data in the wild. The survey organizes the problem around four challenges and reviews recent work addressing them.

Noisy multimodal data

  • Noisy multimodal learning must mitigate arbitrary, complex noise, while modality heterogeneity also offers cross-modal correlations that may help identify and reduce it.The introduction treats noisy data as a fundamental challenge.Source: §1 · p. 1

Incomplete multimodal data

  • Incomplete multimodal learning must handle missing modalities, which can arise when patients with the same disease choose different examinations.The survey frames flexible and reliable methods for incomplete data as a research direction.Source: §1 · p. 1

Imbalanced multimodal data

  • Imbalanced modalities can bias fusion toward a stronger modality, leading models to take shortcuts and underexplore weaker modalities.Fusion methods may not outperform the predominant unimodal model on some modality-preferred applications.Source: §1 · pp. 1–2

Quality dynamically varying multimodal data

  • Modality quality can vary across samples, so applications such as low-light imaging may require integration that adapts to changing quality.RGB can be less informative than thermal imagery in low-light or backlight conditions.Source: §1 · p. 2

Survey scope

  • The survey focuses on multimodal fusion across clustering, classification, object detection, and semantic segmentation, organizing work into four challenge areas.The areas are noisy data, missing-modality imputation, balanced fusion, and dynamic fusion.Source: §1 · pp. 1–2

2 LEARNING ON NOISY MULTIMODAL DATA

The survey distinguishes noise within individual modalities from semantic noise between weakly aligned or unaligned pairs. It reviews reduction approaches and highlights gaps in general noise patterns and learning paradigms.

2 LEARNING ON NOISY MULTIMODAL DATA

  • Multimodal noise can arise from sensors, environmental interference, transmission losses, or weakly aligned and unaligned pairs; cross-modal correlations may help mitigate it.The survey distinguishes modal-specific noise from higher-level semantic noise.Source: §2 · p. 2

2.1 Modal-specific noise reduction

  • Modal-specific denoising methods depend on the input modalities and task, and often aggregate useful multimodal information while mitigating noise.The section presents visual noise reduction in multimodal image fusion as a showcase.Source: §2.1 · p. 2

2.1.1 Joint variation based fusion

  • Joint variation-based fusion and denoising optimizes across modalities, with total-variation approaches applied to image fusion in medical, navigation, infrared-visible, and classification settings.Examples include adaptive weighting for MRI and PET and feature fusion of hyperspectral imaging and LiDAR.Source: §2.1.1 · p. 2

2.2 Cross-modal noise reduction

  • Cross-modal noise includes shifted object locations and irrelevant image-text pairs; approaches use data filtering or registration, model-based rectification, and noise-robust regularization.Examples include CAT filtering, image registration, NCR, and NLIP.Source: §2.2 · p. 3

2.3 Discussion

  • Current noisy-data methods often target specific scenarios, leaving general noise patterns and learning paradigms relatively unexplored.The discussion identifies noise correlation across modalities, complementarity between noisy and clean modalities, and high-level semantic noise as research topics.Source: §2.3 · p. 4

3 INCOMPLETE MULTIMODAL LEARNING

Real-world multimodal data may be incomplete for practical reasons, while conventional multimodal models typically assume complete data. The survey categorizes methods by whether and how they impute missing information.

Causes and examples

  • Missing modalities can result from equipment damage, data loss, patient preferences, economic constraints, or limited medical resources.The survey gives unavailable browsing histories or credit scores and declined PET or cerebrospinal-fluid examinations as examples.Source: §3 · p. 4

Learning with incomplete data

  • Conventional multimodal models typically assume complete data and cannot directly handle samples with partially missing modalities.Incomplete multimodal learning aims to explore information from such data.Source: §3 · p. 4

Method taxonomy

  • The survey groups incomplete-data methods into imputation-based and imputation-free approaches, with imputation further divided into instance- and modality-level methods.The imputation-based categories are illustrated in Fig. 2.Source: §3 · p. 4

3.1 Imputation based incomplete multimodal learning

Imputation-based methods recover missing modalities or related information so conventional multimodal learning can be used downstream. The survey distinguishes heuristic, model-agnostic approaches from learned methods, including kernel or graph completion and raw-data recovery.

3.1 Imputation based incomplete multimodal learning

  • Imputation-based incomplete learning fills missing modalities or related information in data, kernels, or graphs before conventional multimodal learning is applied downstream.The survey further distinguishes model-agnostic from model-specific imputation.Source: §3.1 · p. 4

3.1.1 Model-agnostic imputation

  • Model-agnostic approaches use simple fills such as zero or modality means, while weighted factorization or dynamic weighting can reduce the influence of imputed values.Other examples impute missing graph similarities or kernel elements from valid modalities or averages.Source: §3.1.1 · p. 4

3.1.2 Learning based imputation

  • Learning-based imputation includes kernel- or graph-based completion and direct raw-data recovery.Kernel and graph approaches complete relationships in those representations rather than recovering raw modality values.Source: §3.1.2 · p. 4
  • Kernel methods include KCCA, which is limited to two modalities with one complete modality, and joint completion methods for multiple incomplete kernels.Other approaches use sparse reconstruction, matrix completion, or consensus representations for clustering.Source: §3.1.2 · pp. 4–5
  • Raw-data methods use GANs, autoencoders, semantic neighbors, latent prediction, matrix factorization, or prototypes to recover missing modalities.UIMC imputes distributions rather than deterministic values and uses uncertainty to reduce the effect of low-quality imputations.Source: §3.1.2 · pp. 5–6

3.2 Imputation-free incomplete multimodal learning

Imputation-free incomplete multimodal learning uses available modalities rather than reconstructing missing ones, with approaches organized around latent representations, graphs, kernels, and deep learning. Across these approaches, methods seek to recover shared structure or representations from incomplete observations, with some relying on missingness information or additional training conditions.

3.2 Imputation-free incomplete multimodal learning

  • Imputation-free methods use information from available modalities and are grouped into latent representation and projection, graph, kernel, and deep learning approaches.Unlike imputation-based methods, they focus on information corresponding to observed modalities.Source: §3.2 · p. 6

3.2.1 Latent representation and projection learning

  • Latent representation methods learn a common representation from partially aligned observed data, using factorization, weighting, or projection to account for missing instances.Weighted matrix factorization uses missing-location weights; related methods extend these ideas to more modalities and other tasks.Source: §3.2.1 · pp. 6–7

3.2.2 Graph learning based

  • Graph-based methods learn a consensus graph or representation from incomplete modality-specific relationships among available instances.Approaches include adaptive collaborative graph learning and co-regularized consensus representation learning.Source: §3.2.2 · p. 7

3.2.3 Kernel learning based

  • Kernel methods operate in kernel space, jointly recovering missing representations and learning a consensus representation, but their performance depends on initialized incomplete basis quality.EERIMVC uses an initialized consensus representation as prior knowledge; the initialization may use zero or average filling.Source: §3.2.3 · pp. 7–8

3.2.4 Deep learning based

  • Deep methods use encoders and reconstruction objectives to learn fused representations, often weighting fusion by modality availability, while some require fully observed training samples.The surveyed deep approaches also include classification methods for incomplete modalities or labels and medical applications.Source: §3.2.4 · p. 8

3.3 Discussion

Incomplete multimodal methods are divided into imputation-based and imputation-free approaches. The discussion highlights unresolved challenges around imputation quality, missingness information, and less-studied incomplete cases.

3.3 Discussion

  • Incomplete multimodal methods fall into imputation-based approaches and imputation-free approaches that exploit available modalities.Imputation-based methods either complete graphs or kernels or directly fill missing data.Source: §3.3 · p. 8
  • Assessment of the quality of imputed instances is commonly overlooked, while using prior missingness information to mask unknown modalities remains difficult.Such masking is difficult to bridge information gaps and imbalance caused by missing modalities.Source: §3.3 · pp. 8–9
  • Unpaired views and partially missing labels are identified as incomplete-data cases needing more attention.Source: §3.3 · p. 9

4 BALANCED MULTIMODAL LEARNING

Modalities differ in source, form, learning properties, and the amount of target-related information they provide. The section describes how these disparities can lead a multimodal model to rely more on a higher-quality modality and underfit others.

4 BALANCED MULTIMODAL LEARNING

  • Different data sources and forms give modalities distinct learning properties, including different convergence speeds, complicating joint learning.Source: §4 · p. 9
  • Even when modalities depict the same concept, they can provide different amounts of target-related information, as in meeting video paired with noisy street-car audio.Source: §4 · p. 9
  • Because deep networks are greedy, multimodal models may rely on a high-quality modality and underfit the others.The passage presents balancing modality differences as a focus of recent strategies to improve overall model performance.Source: §4 · p. 9

4.1 Property-discrepancy method

Property-discrepancy methods address differences in modality learning dynamics that uniform joint training may neglect. The section surveys objective, optimization, and architecture strategies for balancing those differences.

4.1 Property-discrepancy method

  • Uniform joint training may overlook modality-specific learning properties, and the jointly trained model can be outperformed by the best unimodal counterpart.The passage identifies learning-objective, optimization, and architecture approaches to address this discrepancy.Source: §4.1 · p. 9

4.1.1 Learning-objective-based method

  • Learning-objective methods add unimodal losses so modality-specific training can be adjusted through their loss weights.Wang et al. select weights using an overfitting-to-generalization ratio to balance differences in modality convergence speed.Source: §4.1.1 · p. 9

4.1.2 Optimization-based method

  • Optimization-based methods dynamically balance modality learning rates, assigning a lower rate to a modality closer to convergence.This supports modality-specific parameter optimization.Source: §4.1.2 · p. 9

4.1.3 Architecture-based method

  • Architecture-based methods can randomly drop the faster-converging modality pathway or use low-level cross-modal connections to support joint learning.The pathway dropping is intended to slow that modality's training; central connections link low-level audio and video features.Source: §4.1.3 · pp. 9–10

4.2 Quality-discrepancy methods

Differences in modality quality can make multimodal models favor the stronger modality and under-optimize others. The section organizes balancing approaches by learning objective, optimization, architecture, and data augmentation.

4.2 Quality-discrepancy methods

  • Multimodal models can outperform unimodal models overall while producing worse unimodal representations; the higher-quality modality is less under-optimized.The section attributes this imbalance to model preference for the dominant modality.Source: §4.2 · p. 10
  • Objective-based methods add losses to improve unimodal learning, including semantic alignment, confidence calibration, distillation, MMCosine, and margin-based robustness regularization.L_uni is designed to improve unimodal learning, especially for lower-quality modalities.Source: §4.2 · pp. 11–12

4.2.2 Optimization-based method

  • Optimization methods modulate gradient magnitude to slow dominant modalities and direct updates using class prototypes to improve slower-learning modalities.Examples extend gradient modulation to rumor detection and audiovisual video parsing; the latter also uses a modality-separated decision unit.Source: §4.2.2 · p. 12

4.2.3 Architecture-based method

  • Architecture-based methods balance modalities through temporal attention across unimodal branches, coordinated feature spaces, or variational feature fusion.The described applications include generative scenarios and segmentation.Source: §4.2.3 · p. 12

4.2.4 Data-augmentation-based method

  • Data-augmentation approaches train underused modalities more, mask well-learned modalities, or re-sample low-contributing modalities at the sample level.The Shapley-based valuation approach dynamically re-samples modalities during training.Source: §4.2.4 · p. 12

4.3 Discussion

Balanced multimodal learning addresses discrepancies in modality learning properties or data quality through objective, optimization, architecture, and augmentation strategies. The discussion identifies theoretical cooperation and extension to multimodal large language models as open directions.

4.3 Discussion

  • Balanced multimodal learning methods target modality discrepancies associated with multimodal data heterogeneity.The discussion identifies theoretical guidance for cooperation between modalities as a promising direction.Source: §4.3 · p. 12
  • Current methods focus mainly on typical discriminative tasks, with only a few generative tasks represented.The discussion raises multimodal large language models as a setting that may also face imbalance.Source: §4.3 · p. 12
  • Extending existing methods or designing new solutions for multimodal large language models remains an expected research direction.The discussion presents this as an opportunity rather than an established result.Source: §4.3 · p. 12

5 DYNAMIC MULTIMODAL FUSION

Because modality quality can vary across samples and settings, dynamic fusion aims to adaptively integrate task-specific information. The survey organizes approaches as heuristic, attention-based, and uncertainty-aware.

5.1 Heuristic dynamic fusion

  • Heuristic methods adapt fusion using illumination cues, feature importance, or task-specific policies that select modalities for each input.Examples include day/night weighting, channel exchange, and selecting modalities per video segment.Source: §5.1 · p. 13

5.2 Attention-based dynamic fusion

  • Attention-based approaches dynamically weight modality features or constrain cross-modal interactions, including through self-, channel-, spatial-, and transformer-based mechanisms.Fusion Bottlenecks restrict cross-modal interactions to bottleneck tokens and mid-late fusion layers to avoid expensive full pairwise attention.Source: §5.2 · pp. 13–15

5.3 Uncertainty-aware dynamic fusion

  • Uncertainty-aware methods estimate confidence or uncertainty to reweight modality contributions, using strategies such as sparse gating, subjective logic, entropy, and probabilistic distributions.The survey covers confidence-based gating, Dempster-Shafer evidence fusion, entropy-based noisy-or fusion, Gaussian uncertainty estimates, and NIG mixtures.Source: §5.3 · pp. 15–16
  • Theoretical analysis links uncertainty-aware weighting to a tighter generalization error bound than traditional static methods.The bound separates weighted unimodal loss from a covariance term representing uncertainty awareness; with an effective uncertainty estimator, that covariance should be negative.Source: §5.3 · p. 16

5.4 Discussion

  • Dynamic fusion methods address modality-quality variation across samples, time, or space, with heuristic, attention-based, and uncertainty-aware strategies differing in their focus.The survey identifies potential applications in models such as CLIP and in application-specific settings including autonomous driving and medical-image fusion.Source: §5.4 · p. 16

6 CONCLUSION

The conclusion recaps the survey’s low-quality multimodal-data challenges and points to broader research scope and unresolved problems.

6 CONCLUSION

  • The survey investigates noisy, partial, unbalanced, and quality-varying multimodal data challenges in the wild.Source: §6 · p. 16
  • Future work may examine low-quality multimodal data for foundation models such as CLIP and for tasks beyond fusion, including alignment and retrieval.Source: §6 · p. 16
  • The survey identifies other low-quality forms, including adversarial data, and aims to help catalog future work and unresolved problems.Source: §6 · p. 17
Loading 2404.18947v3…