PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusion

Mary Ogbuka Kenneth, Foaad Khosmood, Abbas Edalat

arXiv:2608.26879v1cs.LGcs.MM

TL;DR

Conventional multimodal fusion can fail to outperform the strongest unimodal baseline and can degrade the dominant modality. The paper proposes IAF, which shields the dominant pathway and strengthens weaker modalities through pre-fusion distillation. Across three benchmarks with different modality hierarchies, IAF preserves the dominant pathway at its unimodal ceiling and improves fused performance.

  • Problem

    Conventional fusion often fails to surpass the strongest unimodal baseline, while symmetric attention can degrade the dominant modality during integration.

  • Method

    IAF passes the dominant modality through fusion unchanged, lets weaker modalities attend to it as an anchor, and applies modality-aware distillation beforehand.

  • Results

    Across three benchmarks with differing modality hierarchies, IAF preserves the dominant pathway at its unimodal ceiling and consistently outperforms conventional fusion baselines.

  • Takeaways & Limitations

    Protecting strong modalities from cross-modal interference is an architectural principle for robust multimodal learning.

  • Takeaways & Limitations

    IAF requires empirical identification of the dominant modality, and its end-to-end effectiveness with jointly fine-tuned encoders remains unevaluated.

Abstract

from arXiv · show

Fusing multiple modalities is expected to improve model performance. However, on the MultiHuSE dataset, early, late, and symmetric attention fusion often fail to outperform the best unimodal baseline (text). Pathway isolation of a symmetric attention fusion model reveals that the text-pathway accuracy drops from 74.9% to 56.4% after fusion in one such setting, indicating that the dominant modality can be degraded during integration. We term this strong-modality collapse and argue that it helps explain why some multimodal models fail to surpass unimodal baselines. We propose Inverted Asymmetric Fusion (IAF), which avoids forcing mutual attention across modalities. The dominant modality is preserved by passing through fusion unchanged, while weaker modalities attend to it as a contextual anchor. Before fusion, weaker modalities are strengthened using Modality-Aware Knowledge Distillation. We evaluate IAF on three benchmarks with different modality hierarchies: text-dominant datasets (MultiHuSE, UR-FUNNY) and an audio-visual-dominant dataset (MUStARD). Pathway isolation shows that IAF preserves the dominant modality's internal accuracy at its unimodal ceiling across all tested configurations, whereas symmetric fusion degrades it by up to 18.5% on MultiHuSE. IAF improves over the strongest unimodal baseline by up to 8.25%.

1 Introduction

Conventional fusion can degrade a dataset’s strongest modality rather than surpassing the unimodal baseline. IAF addresses this failure by shielding the dominant pathway, strengthening weaker modalities beforehand, and testing across differing modality hierarchies.

  • 18.5 pp: Symmetric fusion degrades the dominant text modality on MultiHuSE, whereas IAF preserves it.
  • 74.9% to 56.4%: Pathway isolation shows the dominant text pathway losing accuracy after symmetric fusion.
  • Strong-modality collapse describes fusion undermining the model’s strongest modality through symmetric attention to weaker, noisier signals.
  • IAF lets the dominant modality bypass cross-modal attention while weaker modalities attend to it as a contextual anchor.
  • IAF is evaluated across three benchmarks spanning text-dominant and audio-visual-dominant modality hierarchies.
  • Modality-Aware Knowledge Distillation strengthens weaker encoders before asymmetric fusion.

2 Related Work

Multimodal fusion has progressed toward attention-based interaction, but uniform attention can ignore modality-strength differences. The paper combines structural shielding with pre-fusion distillation to limit interference and strengthen weaker encoders.

  • Attention-based fusion became a dominant paradigm after earlier concatenation, averaging, tensor, and low-rank approaches.
  • Uniform attention can force dominant modalities to attend to weaker, noisier signals and corrupt their internal representations.
  • Modality collapse reflects multimodal models exploiting only a subset of inputs as modalities overfit and generalise at uneven rates.
  • Structural shielding limits interference, but weaker encoders must first be strengthened for complementary contribution.
  • MAKD differs from common fusion-stage distillation by using the empirically dominant unimodal modality as teacher before asymmetric integration.
  • UR-FUNNY and MultiHuSE are text-dominant, whereas MUStARD gives greater weight to acoustic and visual cues.

3 Methodology

The framework first strengthens weaker modalities through hierarchy-aware distillation, then applies Inverted Asymmetric Fusion to protect the dominant modality while integrating contextual information from other streams.

  • Overview: The two-stage framework uses Modality-Aware Knowledge Distillation before Inverted Asymmetric Fusion.Distillation strengthens weaker modalities before fusion.
  • Modality-Aware Knowledge Distillation: The dominant modality acts as a teacher for weaker modalities before fusion.On MUStARD, weighted dual teachers supervise students when audio and video both dominate.
  • Modality-Aware Knowledge Distillation: The distillation loss combines hard-label cross-entropy, temperature-scaled KL divergence, and bottleneck feature alignment.The coefficients are α = 0.4, β = 0.35, and γ = 0.25.
  • Inverted Asymmetric Fusion: IAF shields the dominant modality from cross-modal attention while weaker modalities attend to it as a contextual anchor.The PURE pathway remains unmodified, whereas ATTEND pathways query the other modalities.
  • Adaptive Gating and Classification: Each modality has a dedicated classifier head, with a frozen standalone head for the PURE modality and trainable heads for ATTEND modalities.A gating network combines pathway logits using sample-adaptive weights computed from pre-attention features.
  • Pathway Analysis: IAF enforces dominant-pathway accuracy at the unimodal ceiling, while ATTEND pathway accuracy may fall below standalone baselines.The pathway analysis treats any dominant-modality deviation from its standalone accuracy as cross-modal interference.

4 Experiments

The experiments evaluate IAF on three humor and sarcasm benchmarks selected for contrasting modality hierarchies, using controlled features, baselines, and reporting protocols.

  • Benchmarks: The evaluation spans three multimodal humor and sarcasm benchmarks with contrasting modality hierarchies and task types.The tasks include binary and multiclass classification.
  • MultiHuSE: MultiHuSE is a 5-class dataset with 2,407 video utterances and a text-dominant hierarchy.Its strict 5-fold split groups performances by unique text ID to prevent script-level leakage.
  • UR-FUNNY: UR-FUNNY is a binary humor dataset of 16,514 context–punchline pairs with text stronger than audio and video.The experiments retain the benchmark’s GloVe, COVAREP, and OpenFace features.
  • MUStARD: MUStARD is a balanced binary sarcasm dataset of 690 utterances where audio and video are stronger than text.This hierarchy provides the clearest inversion relative to MultiHuSE.
  • Experimental Controls: All features are extracted offline with frozen pretrained encoders so performance differences reflect fusion architecture rather than encoder fine-tuning.No encoder parameters are updated during distillation or fusion training.
  • Baselines and Reporting: The study compares early, late, and symmetric fusion against IAF using identical features and hyperparameters.Accuracy and macroaveraged F1 are reported, with 5-fold means and standard deviations for MultiHuSE and MUStARD.

5 Results and Analysis

Across three benchmarks, IAF improves multimodal fusion while preserving the dominant modality, unlike symmetric fusion, which can degrade it. Ablations further show benefits from distillation, regularisation, and hierarchy-aware gating.

  • Unimodal and Fusion Results: 0.88–6.23% gains: knowledge distillation consistently improves student unimodal models before fusion.The largest gain is 6.23% on UR-FUNNY COVAREP audio.
  • Unimodal and Fusion Results: 2.06–6.48% gains: IAF exceeds the strongest unimodal text baseline across all four MultiHuSE configurations.Early, late, and symmetric fusion do not reliably surpass the text baseline.
  • Unimodal and Fusion Results: 70.72% accuracy: IAF surpasses MISA by 2.12% and symmetric fusion by 1.81% on UR-FUNNY.Across five seeds, IAF reaches 69.70 ± 0.61% mean accuracy, with every seed outperforming both baselines.
  • Unimodal and Fusion Results: 83.33% accuracy: IAF with E5 exceeds MUStARD’s state-of-the-art MHA with RoBERTa by 4.01%.The IAF-over-symmetric margin is 0.87–1.02 percentage points and falls within one standard deviation of both models.
  • Pathway Isolation: 18.5pp maximum degradation: symmetric fusion reduces dominant-modality pathway accuracy, while IAF preserves the unimodal baseline across all three datasets.On MultiHuSE, the E5 text pathway drops from 74.9% to 56.4% under symmetric fusion.
  • Ablation Study: 1.02–2.41% gains: distilled representations improve IAF over raw features on MultiHuSE, with additional gains on MUStARD and UR-FUNNY.These results support strengthening weaker modalities before fusion rather than relying only on architecture.
  • Ablation Study: Up to 4.07% drops: removing Mixup and Curriculum Modality Dropout reduces performance across all datasets.Curriculum Dropout provides a consistently larger isolated gain than Mixup.
  • Ablation Study: The learnable gate generally performs best, while the hierarchy gate remains competitive under strong modality imbalance.Uniform gating uses equal weights across text, audio, and video.

6 Conclusion

IAF is a two-stage framework that preserves the dominant modality while strengthening weaker modalities before fusion. Across text-dominant and audio-visual-dominant settings, it maintains dominant-pathway accuracy and improves fused performance.

  • IAF is introduced as a two-stage framework for mitigating strong-modality collapse in discriminative fusion.It preserves the dominant modality through a PURE/ATTEND hierarchy and strengthens weaker modalities with knowledge distillation before fusion.
  • IAF maintains the dominant modality’s pathway at its unimodal ceiling while improving fused performance across text-dominant and audio-visual-dominant settings.
  • IAF consistently outperforms early, late, and symmetric fusion baselines across three benchmarks with different modality hierarchies.
  • IAF surpasses prior published methods on UR-FUNNY and MUStARD.
  • Protecting strong modalities from cross-modal interference is identified as an important architectural principle for robust multimodal learning.

Limitations

The paper’s limitations concern hierarchy identification, frozen-feature evaluation, and narrow dataset scope. These constraints leave uncertainty about performance when modalities are similarly strong, encoders are jointly optimized, or tasks and languages differ.

  • IAF requires empirical identification of the dominant modality before fusion training, and structural asymmetry may diminish when modalities have similar predictive strength.
  • The inferred hierarchy reflects encoder performance rather than intrinsic task structure, because a stronger text encoder may overshadow a more task-relevant video encoder.
  • All experiments use frozen, pre-extracted features, so end-to-end joint optimization has not been evaluated.
  • The pathway analysis uses the original frozen unimodal classifier as a fixed probe, testing decision-boundary preservation rather than recoverability in principle.
  • All datasets are English-only and limited to humor and sarcasm, leaving generalization to other languages and tasks unresolved.

Ethical considerations

The reported experiments use existing public datasets and anonymized, pre-extracted features without new data collection or personally identifiable information. Hyperparameters were manually tuned rather than selected through automated search.

  • The study uses three publicly available datasets released under standard academic licences and conducts no new data collection or human annotation.
  • All experiments operate on pre-extracted, anonymised feature representations without access to personally identifiable information.
  • Hyperparameters were manually adjusted using validation performance until stable configurations were identified, followed by one final experiment per configuration.

B Role of Modalities Ablation

Bimodal ablations evaluate how removing each modality changes trimodal performance across the three benchmarks and encoder configurations. The results support text dominance on MultiHuSE and UR-FUNNY, while audio and video are co-dominant on MUStARD.

  • Full trimodal IAF outperforms every bimodal variant on all three datasets.
  • 65.85% is the MultiHuSE accuracy for the text-free audio-and-visual pair, confirming text’s dominant role.
  • 68.91% versus 65.29% shows that text-plus-audio outperforms text-plus-video on UR-FUNNY, consistent with Text > Audio ≈ Video.
  • 82.61% for the text-free audio-and-visual pair nearly matches 83.33% full trimodal performance on MUStARD, reflecting audio-video co-dominance.
  • Bimodal ablations with BERT configurations follow the same directional patterns as E5 configurations on MultiHuSE and MUStARD.
  • Regularisation ablations across datasets and encoder configurations show directionally consistent results across BERT- and E5-based configurations.

D Role of Gate Mechanism Ablation

Gate ablations show that learnable sample-adaptive weighting is usually strongest, while pathway analysis shows IAF preserves dominant-modality accuracy where symmetric fusion degrades it.

  • Gate mechanism ablation: Learnable gating attains the highest accuracy on UR-FUNNY and MUStARD, indicating benefits from sample-adaptive weighting.
  • Gate mechanism ablation: MultiHuSE is a mild exception: fixed hierarchy gating performs marginally better under extreme modality imbalance.The authors suggest learnable gating can over-emphasize the dominant modality in highly skewed conditions.
  • Gate mechanism ablation: Overall gate differences remain small, at ≤1.45%, indicating that IAF is largely insensitive to the precise gating strategy.
  • Pathway collapse analysis: Across all seven configurations, symmetric fusion reduces dominant-modality pathway accuracy by a mean of 11.6 pp, whereas IAF matches the unimodal ceiling every time.The pattern holds across datasets and encoder combinations.
  • Pathway collapse analysis: On MultiHuSE fold 4, the IAF text pathway reaches 76.7% accuracy, while symmetric fusion falls to 40.3% despite broadly similar feature manifolds.The result is consistent with classifier mismatch because the shared head is not optimized for single-pathway inputs.

F Gate Weight Analysis

IAF’s gate weights generally favor the empirically dominant modality while adapting across samples, and its distillation results show limited sensitivity to coefficient choices.

  • Population-level gate weights: The PURE modality receives the highest mean gate weight: text leads on MultiHuSE, while audio and video together exceed 0.80 on MUStARD.UR-FUNNY distributes weights more evenly, with text at 0.372, audio at 0.310, and video at 0.318.
  • Population-level gate weights: No modality exceeds a mean gate weight of 0.53 in any configuration, suggesting that IAF retains multimodal contributions.
  • Sample-level gate weights: Individual MultiHuSE samples range from near-exclusive text weighting above 0.95 to strong audio reliance, showing substantial within-dataset variation.
  • Sample-level gate weights: The gate adapts to sample-level modality reliability: audio receives about 90% weight when transcript content is relatively uninformative.
  • Distillation coefficient sensitivity: IAF fusion accuracy varies by at most 1.20 percentage points across four distillation-coefficient settings, with default and equal-weight variants both reaching 78.06%.
  • Knowledge distillation design: MAKD is a standalone pre-fusion stage that uses the empirically dominant unimodal modality as teacher to strengthen weaker encoders before fusion.Prior multimodal knowledge-distillation methods typically operate within the fusion stage itself.
Loading 2608.26879v1…