PaperlayerView exact revision on arXiv

Source-linked AI summary

Provable Dynamic Fusion for Low-Quality Multimodal Data

Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, Xi Peng

arXiv:2306.02050v2cs.LGcs.CV

TL;DR

Multimodal fusion lacks sufficient theoretical justification for robustly handling low-quality data. This paper analyzes generalization, connects dynamic fusion with uncertainty estimation, and proposes QMF, with experiments supporting improved classification accuracy and robustness. Its conclusions remain bounded by uncertainty-estimation challenges and theoretical reliance on an assumption.

  • Problem

    Existing multimodal fusion methods can fail on imbalanced, noisy, or corrupted data, while theoretical criteria for robust fusion remain limited.

  • Method

    The paper analyzes decision-level dynamic fusion from a generalization perspective and proposes QMF, which uses energy-based uncertainty to characterize modality quality.

  • Results

    Experiments on multiple benchmarks support the theoretical observations and show QMF improves classification accuracy and robustness.

  • Takeaways & Limitations

    Dynamic fusion can be theoretically better generalized when fusion weights are negatively correlated with unimodal generalization errors, linking robust fusion design to uncertainty estimation.

  • Takeaways & Limitations

    QMF depends on uncertainty estimation, which was only mildly correlated with loss on NYU Depth V2 and SUN RGB-D, and the theory relies on Assumption 1.

Abstract

from arXiv · show

The inherent challenge of multimodal fusion is to precisely capture the cross-modal correlation and flexibly conduct cross-modal interaction. To fully release the value of each modality and mitigate the influence of low-quality multimodal data, dynamic multimodal fusion emerges as a promising learning paradigm. Despite its widespread use, theoretical justifications in this field are still notably lacking. Can we design a provably robust multimodal fusion method? This paper provides theoretical understandings to answer this question under a most popular multimodal fusion framework from the generalization perspective. We proceed to reveal that several uncertainty estimation solutions are naturally available to achieve robust multimodal fusion. Then a novel multimodal fusion framework termed Quality-aware Multimodal Fusion (QMF) is proposed, which can improve the performance in terms of classification accuracy and model robustness. Extensive experimental results on multiple benchmarks can support our findings.

1. Introduction

Multimodal fusion can exploit cross-modal information but is vulnerable when modality quality varies, motivating a theoretical and quality-aware approach to robust fusion. The paper introduces QMF and reports experiments supporting its theoretical observations.

  • Motivation: Real-world multimodal data can be imbalanced, noisy, or corrupted, and existing fusion methods may fail to outperform the best unimodal method.The problem is linked to unreliable modality quality in practical settings.
  • Motivation: The paper studies the theoretical advantage and criterion of robust multimodal fusion.
  • Empirical support: QMF consistently outperforms unimodal methods on low-quality data, while existing multimodal methods degrade relative to their best unimodal counterparts in high-noise regimes.
  • Approach: QMF is a decision-level fusion framework that uses energy-based uncertainty to characterize each modality's quality.
  • Theory: The analysis characterizes generalization error and identifies conditions under which dynamic fusion can outperform static fusion.The contribution connects fusion-weight behavior with unimodal generalization errors.

2. Related works

Related work frames multimodal fusion as a fundamental integration problem whose dynamic variants remain insufficiently understood, while uncertainty estimation offers tools for assessing prediction reliability and modality quality.

  • Multimodal fusion: Multimodal fusion integrates modality-wise features into a joint representation and is commonly organized as early, intermediate, or late fusion.
  • Multimodal fusion: Late fusion remains widely used because of its interpretability and practical simplicity.
  • Dynamic fusion: Dynamic fusion strategies have improved multimodal applications, but their inductive bias and reliability remain insufficiently understood.
  • Uncertainty estimation: Uncertainty estimation methods include Bayesian neural networks, deep ensembles, predictive confidence, Dempster-Shafer theory, and energy scores.
  • Uncertainty estimation: Energy scores are used to capture out-of-distribution uncertainty, while predictive confidence relates predicted class probability to empirical accuracy.

3. Theory

The theory analyzes late multimodal fusion from a generalization perspective, separating error into empirical loss, complexity, and covariance terms. It identifies conditions under which dynamic fusion has a tighter bound than static fusion and frames reliable weighting as the central challenge.

  • 3. Theory: The analysis studies decision-level late fusion, where modality-specific predictions are combined using static or input-dependent weights.Static weights are constant, whereas dynamic weights vary with the input sample.
  • 3. Theory: Theorem 1 bounds multimodal generalization error using weighted unimodal empirical loss, model complexity, and covariance between fusion weights and unimodal loss.The analysis uses logistic loss in a two-class classification setting and Rademacher complexity to characterize the bound.
  • 3. Theory: For static fusion, the fusion weights are constant, so the covariance term in the generalization bound is zero.This removes the covariance contribution that dynamic fusion can potentially make nonpositive.
  • 3. Theory: Dynamic fusion provably outperforms static fusion when the combined empirical-loss and complexity terms are no larger than in static fusion and Term-Cov ≤0.The criterion compares the upper bounds on dynamic and static generalization error.
  • 3. Theory: When dynamic and static fusion use the same unimodal architectures, their intrinsic complexity and empirical risk can remain invariant, simplifying the comparison.Under this setting, the key comparison depends on whether the dynamic weighting satisfies the theorem’s required condition.
  • 3. Theory: The main challenge for reliable dynamic fusion is learning modality weights that satisfy the theoretical conditions required by the bound comparison.The paper specifically identifies learning w_m^dynamic(x) satisfying Eq. 5 and Eq. 6 as the challenge.

4. Method

QMF realizes robust dynamic fusion by linking modality uncertainty to fusion weights, with theory showing when dynamic weighting can outperform static fusion. It estimates uncertainty using energy-based scores and regularizes them with training-loss information before dynamically weighting unimodal predictions.

  • 4. Method: QMF addresses robust dynamic fusion by connecting uncertainty estimation with modality-level weighting in decision-level late fusion.The framework is proposed to answer how robust dynamic fusion can be realized.
  • 4.1. Coincidence with Uncertainty Estimation: Under Assumption 1, uncertainty must be positively correlated with modal-specific loss, enabling fusion weights that decrease as uncertainty increases.The assumption uses Pearson correlation r(u_m, ℓ_m(x)) ≥ 0; the weighting function is linear with negative slope.
  • 4.1. Coincidence with Uncertainty Estimation: The theoretical analysis identifies conditions under which uncertainty-aware dynamic fusion can provably outperform conventional static fusion.The paper states that this opportunity follows from the relationship between uncertainty estimation and dynamic weighting.
  • 4.2. Enhance Correlation by Additional Regularization: QMF uses energy-based uncertainty, where negative energy is linearly aligned with modality density and more uniform predictions indicate higher uncertainty.The energy score is computed from classifier logits with a temperature parameter.
  • 4.2. Enhance Correlation by Additional Regularization: Because unregularized uncertainty estimates may not correlate sufficiently with modal loss, QMF adds sampling-based regularization using sample-wise training losses.The method uses historical training trajectories and average losses to regularize fusion weights, while the total objective combines cross-entropy and regularization losses.
  • 4. Method: During training, QMF computes uncertainty-aware weights, dynamically combines unimodal predictions, and updates unimodal predictors using the total loss.When one modality is noisy, its higher uncertainty yields a lower weight, reducing its influence relative to a cleaner modality.

5. Experiment

Experiments evaluate QMF across multimodal classification benchmarks, noisy modalities, uncertainty estimators, reliability, and ablations. QMF generally ranks among the top methods, outperforms MMBT and TMC on UPMC FOOD101, and benefits from uncertainty-aware weighting plus regularization.

  • Experimental setup: Experiments assess generalization, uncertainty-aware fusion, reliability, and ablations across scene-recognition and image-text classification tasks.The datasets include NYU Depth V2, SUN RGB-D, UPMC FOOD101, and MVSA; repeated-seed evaluation reports mean, standard-deviation, and worst-case accuracy where applicable.
  • Classification robustness: QMF usually ranks in the top three for average and worst-case accuracy under Gaussian or blank noise.On UPMC FOOD101, QMF outperforms the prior state-of-the-art methods MMBT and TMC.
  • Connection to uncertainty estimation: Dynamic fusion methods’ average and worst-case accuracy coincide with their uncertainty-estimation ability, while QMF achieves the best classification and uncertainty-estimation performance.The comparison evaluates QMF using prediction confidence and Dempster-Shafer evidence theory on NYU Depth V2 and SUN RGB-D.
  • Reliability of QMF: QMF fusion weights show the most effective perception of modality quality among the compared uncertainty-estimation methods, measured by correlation.The reliability analysis uses UPMC FOOD-101 and Pearson correlation between modality losses and fusion weights.
  • Ablation study: Adding the regularization term Lreg yields more reasonable fusion weights, while the full QMF produces the best performance.The ablation study combines uncertainty-aware weighting and Lreg on NYU Depth V2 with Gaussian noise.
  • Summary: Overall, the empirical results support the theoretical findings and indicate improved robustness across multiple datasets.The experiments are designed to test the proposed generalization and uncertainty-related claims.

6. Limitations

The paper identifies uncertainty estimation as a practical challenge for QMF and notes that its theoretical justification depends on Assumption 1. It also reports only mild Pearson correlation on two scene-recognition datasets.

  • Scope and practical limitation: QMF’s fusion weights depend on uncertainty estimation, which can be challenging in real-world settings.The paper suggests exploring novel uncertainty-estimation methods.
  • Theoretical limitation: The theoretical justification relies on Assumption 1, while prior work indicates that OOD detection is not learnable in some scenarios.The paper therefore identifies the generalization ability of dynamic multimodal fusion as an open problem.

7. Conclusions and Future works

The paper analyzes when dynamic multimodal fusion is robust, connects robustness with uncertainty learning, and proposes energy-based weighting supported by experiments on multiple benchmarks. It leaves broader settings and alternative uncertainty estimators for future work.

  • Conclusions: The paper provides rigorous analysis of when and what dynamic multimodal fusion methods are more robust on multimodal data in the wild.The analysis addresses the limited understanding of the inductive bias introduced by dynamic fusion.
  • Conclusions: The findings connect uncertainty learning with robust multimodal fusion and suggest a principle for designing dynamic fusion methods.The proposed work uses an energy-based weighting strategy and supports the findings with experiments on multiple benchmarks.
  • Future works: Future directions include exploring other uncertainty-estimation methods and proving dynamic fusion under a more general setting.These directions are stated as extensions beyond the paper’s energy-based weighting strategy and current theoretical setting.

A. Proofs

The proofs characterize generalization error for dynamic late fusion using Rademacher complexity and compare it with static fusion. The analysis identifies a covariance condition under which dynamic fusion has a tighter bound.

  • Generalization analysis: Rademacher complexity is used to quantify the generalization error of unimodal classifiers.The proof applies complexity-based learning theory to bound unimodal empirical and generalization errors.
  • Dynamic versus static fusion: Dynamic and static fusion share the same empirical-risk and intrinsic-complexity terms when optimized over the same function class.The comparison therefore focuses on the remaining covariance-related contribution.
  • Dynamic versus static fusion: When the correlation between fusion weights and unimodal losses is nonpositive, the corresponding covariance is nonpositive.This sign condition supports a tighter generalization bound for dynamic fusion than for static fusion.

B.1. Datasets details

The senses recognition datasets reorganize NYU Depth V2 into 10 categories and use 19 major scene categories from SUN RGB-D. FOOD-101 contains 60,101 training and 5,000 validation image-text pairs.

  • Senses recognition: NYU Depth V2 is reorganized into 10 categories: 9 usual senses and one “others” category.
  • Senses recognition: SUN RGB-D uses 19 major scene categories, each containing at least 80 images.
  • Image-text classification: FOOD-101 contains 60,101 image-text pairs in training and 5,000 image-text pairs in validation.

B.2. Implementation details

The experiments compare QMF with multiple fusion strategies across senses recognition and image-text classification, using modality-specific pretrained backbones and fixed training settings.

  • Senses recognition: Senses recognition compares late, align-based, concatenated-based, and MMTM attention-based fusion methods.
  • Image-text classification: Image-text classification compares late, concatenated-BOW, concatenated-BERT, and MMTM fusion methods.
  • Image-text classification: Image modality uses ImageNet-pretrained ResNet, while text uses pretrained BERT or BOW depending on the fusion baseline.
  • Training settings: Training uses sampling throughout, λ = 0.1, and temperature parameters T_m = 1 for all experiments.

C.1. Full results with standard deviation

The supplementary results report full comparisons and standard deviations under Gaussian and salt-pepper corruption, alongside a complete NYU Depth V2 ablation study.

  • Full results: Full results with standard deviation are reported in Tables 5 and 4.
  • Salt-pepper noise: Table 6 compares methods when 50% of modalities are corrupted with salt-pepper noise.
  • Noise robustness: The salt-pepper-noise results validate that the proposed method improves multimodal fusion performance under different noise types.
  • Ablation study: Table 4 provides the full ablation study on NYU Depth V2.
  • Gaussian noise: Table 5 compares methods when 50% of modalities are corrupted with Gaussian noise.
Loading 2306.02050v2…