PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Representation Learning by Alternating Unimodal Adaptation

Xiaohui Zhang, Jaehong Yoon, Mohit Bansal, Huaxiu Yao

arXiv:2311.10707v2cs.LGcs.CV

TL;DR

Multimodal learning can favor dominant modalities, and critical modality data may be absent in real-world settings. MLA alternates unimodal optimization while retaining cross-modal information, and the authors report improved performance over prior methods in complete- and missing-modality scenarios.

  • Problem

    Some modalities can dominate multimodal learning while others are ignored, and critical modality data may be missing in real-world data.

    Source: §1 · p. 1

  • Method

    MLA alternates unimodal optimization, uses a shared head and gradient modification to retain cross-modal information, and fuses modality predictions at test time using uncertainty.

    Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 4

  • Results

    MLA is reported to improve multimodal performance over the best prior methods in complete- and missing-modality scenarios.

    Source: §1 · p. 2, §4.2.2 · p. 6

  • Takeaways & Limitations

    The authors report MLA as applicable to complete- and missing-modality learning and say it can enlarge the modality gap.

    Source: §1 · p. 2, §5 · p. 8

    The modality-importance coefficient relies on a hypothesis about prediction uncertainty.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Multimodal learning, which integrates data from diverse sensory modes, plays a pivotal role in artificial intelligence. However, existing multimodal learning methods often struggle with challenges where some modalities appear more dominant than others during multimodal learning, resulting in suboptimal performance. To address this challenge, we propose MLA (Multimodal Learning with Alternating Unimodal Adaptation). MLA reframes the conventional joint multimodal learning process by transforming it into an alternating unimodal learning process, thereby minimizing interference between modalities. Simultaneously, it captures cross-modal interactions through a shared head, which undergoes continuous optimization across different modalities. This optimization process is controlled by a gradient modification mechanism to prevent the shared head from losing previously acquired information. During the inference phase, MLA utilizes a test-time uncertainty-based model fusion mechanism to integrate multimodal information. Extensive experiments are conducted on five diverse datasets, encompassing scenarios with complete modalities and scenarios with missing modalities. These experiments demonstrate the superiority of MLA over competing prior approaches. Our code is available at https://github.com/Cecile-hi/Multimodal-Learning-with-Alternating-Unimodal-Adaptation.

1. Introduction

The introduction frames modality imbalance and missing data as challenges for multimodal learning, then presents MLA’s alternating training, shared-head, and uncertainty-based fusion approach. It reports promise across complete- and missing-modality settings.

1. Introduction

  • Some modalities dominate multimodal learning while others are ignored, and critical modality data may be missing in uncurated real-world settings.The paper identifies these challenges as modality laziness and missing modalities.Source: §1 · p. 1
  • MLA replaces joint optimization with alternating unimodal learning, aiming to reduce interference while integrating cross-modal information.The authors contrast this approach with methods that jointly optimize modality-specific encoders.Source: §1 · p. 1
  • A shared head captures cross-modal information, gradient modification addresses forgetting across modalities, and uncertainty-based test-time fusion combines modality predictions.Modality importance is gauged using prediction uncertainty.Source: §1 · pp. 1–2
  • MLA is reported to improve performance over the best prior methods in both complete- and missing-modality scenarios.The paper also reports that MLA can enlarge the modality gap.Source: §1 · p. 2

2. Related Work

Related work addresses imbalanced modality contributions and missing data through methods such as gradient balancing, imputation, and modality-specific missing-data approaches. The paper presents MLA as an alternative applicable to both conventional and missing-modality tasks.

2. Related Work

  • OGM-GE [26] balances modality contributions by encouraging intensive gradient updating from suboptimal modalities; QMF [48] mitigates effects of low-quality or noisy modalities by estimating an energy-based score of each modality.Source: §2 · pp. 2–3
  • Prior approaches to missing modalities include imputing data or embeddings, a strategy described as often unsuitable for large datasets or large missing-data blocks.The passage notes that imputation often works well on small data.Source: §2 · p. 3
  • Methods avoiding imputation include a generative approach for partial multi-view data and SMIL [22], a Bayesian meta-learning method for severely missing modalities.The paper characterizes these methods as tailored to missing-modality problems, limiting their generality across multimodal tasks.Source: §2 · p. 3
  • Unlike methods tailored to missing data, MLA is presented as applicable to conventional multimodal tasks and settings with missing modalities.The authors report that it outperforms baselines on these tasks.Source: §2 · p. 3

3. Multimodal Learning with Alternating Unimodal Adaptation

MLA replaces joint multimodal training with alternating single-modality optimization while retaining a shared head for cross-modal information. It protects that head from forgetting and fuses modality predictions at test time using uncertainty-based weights.

3. Multimodal Learning with Alternating Unimodal Adaptation

  • MLA alternates unimodal encoder training, protects the shared head from forgetting, and uses uncertainty-based dynamic fusion at inference.The shared head is optimized across modalities during training.Source: §3 · p. 3

3.1. Alternating Unimodal Learning

  • Each modality is optimized independently, reducing interference so less informative modalities are not overshadowed during joint learning.At each training step, data come exclusively from one modality; paired multimodal training data are not required.Source: §3.1 · pp. 3–4

3.2. Learning Cross-Modal Information without Modality Forgetting

  • A gradient modification matrix rectifies shared-head updates to limit interference with information encoded from previously trained modalities.The matrix is obtained using Recursive Least Squares and the update direction is orthogonalized against prior encoded features.Source: §3.2 · p. 4

3.3. Test-Time Dynamic Modality Fusion

  • At inference, MLA weights modality predictions using their uncertainty, assigning lower importance to predictions with higher entropy.The weighted predictions are combined to produce the test-time prediction.Source: §3.3 · pp. 4–5

4. Experiments

Experiments across five datasets evaluate MLA with complete and missing modalities, compare it with prior methods, and examine its components. MLA consistently outperforms other methods across the reported scenarios.

4.1.1 Experimental Setups

  • The evaluation uses five datasets, task-specific encoders, shared fully connected heads, and baselines spanning fusion, modality-laziness, and missing-modality methods.The datasets cover audio-video, image-text, and audio-image-text tasks; baselines use the same backbone architectures.Source: §4.1.1 · p. 5, §4 · p. 5

4.1.2 Result

  • MLA consistently outperforms all other methods across scenarios on audio-video, image-text, and audio-image-text datasets.Table 1 reports average test accuracy for single modalities and combined modalities.Source: Table 1 · p. 6, §4.1.2 · p. 6

4.2.1 Experimental Setup

  • For IEMOCAP missing-modality experiments, each modality is randomly masked at rates from 0.1 to 0.7, with the same rate across training, validation, and testing.The evaluation compares general and missing-modality-specific baselines using the same backbone models.Source: §4.2.1 · p. 6

4.2.2 Results

  • As missing rates increase, all methods degrade, while MLA outperforms general and missing-modality-specific baselines at every rate.The reported comparisons include Late Fusion, QMF, and MMIN.Source: §4.2.2 · pp. 6–7, Table 2 · p. 7

4.3.1 Ablation Study

  • Head gradient modification improves unimodal and multimodal performance, dynamic fusion improves multimodal performance, and combining both strategies performs best.Dynamic fusion is applied only during multimodal fusion and does not affect single-modality performance.Source: Table 3 · p. 7, §4.3.1 · p. 7

4.3.2 Analysis of Modality Isolation in Alternating Unimodal Learning

  • With audio or video masked during training, MLA's unimodal performance remains unaffected by the other modality's absence, unlike QMF.Figure 2 plots KS test accuracy against the missing-modality ratio in audio or video training data.Source: Figure 2 · p. 8, §4.3.2 · p. 8

4.3.3 Analysis of Modality Gap

  • On Food-101, MLA produces a larger text–vision modality gap than concatenation, alongside stronger performance.The paper describes the gap as modalities occupying separate regions in embedding space [21].Source: Figure 3 · p. 8, §4.3.3 · p. 8

4.3.4 Robustness Analysis of the Pre-trained Model

  • With CLIP-pretrained encoders on Food-101, MLA improves subordinate visual-modality performance and fusion performance.Table 4 reports average test accuracy over three seeds.Source: Table 4 · p. 8, §4.3.4 · p. 8

5. Conclusion

MLA reframes multimodal learning as alternating unimodal optimization, using gradient modification to preserve cross-modal information and dynamic fusion at test time. Experiments across five datasets show promise in mitigating modality laziness and improving performance with complete or missing modalities.

5. Conclusion

  • MLA decomposes joint multimodal optimization into alternating unimodal optimization to address modality laziness.It also incorporates dynamic fusion at test time and supports learning with complete or missing modalities.Source: §5 · p. 8
  • A gradient modification mechanism helps avoid modality forgetting while capturing cross-modal information during unimodal optimization.At test time, a dynamic fusion mechanism enables multimodal fusion.Source: §5 · p. 8
  • Across five datasets with different types of modalities, empirical results show promise in mitigating modality laziness and improving performance.The paper reports applicability to both complete- and missing-modality learning scenarios.Source: §5 · p. 8

CREMA-D (Creole Multimodal Affect Database)

The described datasets span audio-video, image-text, and audio-image-text data, with tasks including emotion recognition, scene understanding, food classification, and sentiment analysis.

CREMA-D (Creole Multimodal Affect Database)

  • CREMA-D contains audio and video recordings of Haitian Creole actors portraying emotional states for emotion-recognition research.The dataset includes a range of emotions expressed through speech and facial expressions.Source: CREMA-D (Creole Multimodal Affect… · p. 12
  • Kinetic-Sound combines synchronized video and audio recordings of everyday objects and actions for audio-visual scene understanding and object recognition.The dataset is used to explore how audio and visual information complement one another.Source: CREMA-D (Creole Multimodal Affect… · p. 12
  • Food-101 contains over 100,000 images across 101 food categories, while MVSA combines text, image, and audio data from social-media posts for sentiment analysis.Food-101 focuses on food classification; MVSA captures sentiment expressed in multimedia content.Source: CREMA-D (Creole Multimodal Affect… · p. 12
  • IEMOCAP pairs audio with motion-capture data from emotionally charged conversations between actors to support emotion-recognition and speech-processing research.Across these datasets, the described applications range from emotion recognition to scene understanding, food recognition, sentiment analysis, and speech and gesture research.Source: CREMA-D (Creole Multimodal Affect… · p. 12

A.2. Details of Baselines

The baseline set covers conventional fusion, modulation-based fusion, and methods designed to address modality imbalance. These methods differ in how they combine modality representations and update model parameters.

A.2. Details of Baselines

  • Summation fusion adds modality-specific classifier outputs and updates all encoder and classifier parameters using the fusion loss.It combines information element-wise without explicitly modeling cross-modal interactions.Source: §A.2 · p. 12
  • Concatenation fusion joins encoder feature vectors before a single fully connected layer, with the loss updating encoders and that layer.The method does not inherently capture cross-modal interactions.Source: §A.2 · p. 12
  • Late fusion trains each modality encoder and classifier independently, then averages their unimodal outputs.The approach has no cross-modal interactions and may miss early interactions between data sources.Source: §A.2 · p. 12
  • FiLM [27] uses a condition encoder to produce γ and β components that modulate another modality’s features before classification.Either audio or text can serve as the condition input.Source: §A.2 · p. 13
  • BiGated [19] uses one modality’s sigmoid-activated hidden state as a gate to modulate other modality states before fusion.Its fusion layer aggregates outputs from all modalities.Source: §A.2 · p. 13
  • OGM-GE [26] scales gradients with modality-specific coefficients, while QMF [48] combines multimodal loss with regularization for challenging samples.The QMF loss calculation follows the methodology described in the QMF papers.Source: §A.2 · p. 13

A.3.1 Learning with complete modality

The appendices describe encoder and training configurations for complete- and missing-modality experiments, report ablation results, and identify the scope and metrics of CLIP-based result tables.

A.3.1 Learning with complete modality

  • Complete-modality experiments use ResNet-18 for CREMA-D and KS, M3AE for Food-101 and MVSA, and CAVMAE with M3AE for IEMOCAP.The acoustic IEMOCAP encoder uses a pretrained CAVMAE audio model.Source: §A.3.1 · p. 13
  • Complete-modality experiments use a shared fully connected head, SGD with momentum 0.9, learning rate 0.001, and batch size 64.Features vary by dataset: CREMA-D and KS use fbank audio and concatenated transformed images; other datasets use tokenized BERT text and transformed images, with fbank audio for IEMOCAP.Source: §A.3.1 · pp. 13–14

A.3.2 Learning with missing modalities

  • Missing-modality IEMOCAP experiments extract text with BERT, visual features with MANet, and acoustic features with Wav2vec.The method descriptions identify these as the feature-extraction models for the three modalities.Source: §A.3.2 · p. 14
  • For missing-modality experiments, all methods use three-layer fully connected modality encoders and a fully connected classifier, with batch size 64 and learning rate 0.001.The learning rate is reduced each iteration by a decay ratio of 0.1.Source: §A.3.2 · p. 14

A.4.2 Full Results of Ablation Studies

  • The ablation section reports full results with 95% standard deviation.Source: §A.4.2 · p. 14

A.4.3 Full Results of CLIP

  • The CLIP results tables cover single-modality and combined-modality accuracy across audio-video, image-text, and audio-image-text datasets, missing-rate settings on IEMOCAP, and Food-101.The captions specify average test accuracy across three random seeds for the reported tables.Source: Table 5 · p. 15, Table 6 · p. 15, Table 7 · p. 15

Related papers

Loading 2311.10707v2…