PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing Modality Robustness in Multimodal Sentiment Analysis

Devamanyu Hazarika, Yingting Li, Bo Cheng, Shuai Zhao, Roger Zimmermann, Soujanya Poria

arXiv:2205.15465v1cs.CL

TL;DR

The authors call for attention to an equally important aspect alongside improvements in multimodal performance. The authors study diagnostic checks for modality errors across MSA models and benchmark datasets and examine modality perturbations as a robust-training approach. The study finds MSA models susceptible to performance drops from modality errors, while robust training and clean-test performance can coexist without an explicit trade-off.

  • Problem

    The authors call for attention to an equally important aspect alongside improvements in multimodal performance.

    Source: §1 · p. 1

  • Method

    The authors study diagnostic checks for modality errors across MSA models and benchmark datasets and examine modality perturbations as a robust-training approach.

    Source: §1 · p. 2, §3.2 · p. 3, §5 · p. 5

  • Results

    The study finds MSA models susceptible to performance drops from modality errors, while robust training and clean-test performance can coexist without an explicit trade-off.

    Source: §5 · p. 5

  • Takeaways & Limitations

    The paper offers simple, easy-to-integrate diagnostic checks and training methods for examining and improving robustness in MSA.

    Source: §5 · p. 5

    The diagnostic study leaves other potential errors, such as affine transformations to representations, for future work.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Building robust multimodal models are crucial for achieving reliable deployment in the wild. Despite its importance, less attention has been paid to identifying and improving the robustness of Multimodal Sentiment Analysis (MSA) models. In this work, we hope to address that by (i) Proposing simple diagnostic checks for modality robustness in a trained multimodal model. Using these checks, we find MSA models to be highly sensitive to a single modality, which creates issues in their robustness; (ii) We analyze well-known robust training strategies to alleviate the issues. Critically, we observe that robustness can be achieved without compromising on the original performance. We hope our extensive study-performed across five models and two benchmark datasets-and proposed procedures would make robustness an integral component in MSA research. Our diagnostic checks and robust training solutions are simple to implement and available at https://github. com/declare-lab/MSA-Robustness.

1 Introduction

The introduction frames modality robustness as essential for real-world MSA, where errors can affect source modalities, and asks whether language dependence makes models vulnerable. It reports a language-specific performance drop in an initial test and motivates broader diagnostics and robust training that need not reduce clean-set performance.

1 Introduction

  • Modality robustness matters for deployment, where source signals may encounter data loss, corruption, jitter, or privacy issues.Source: §1 · p. 1
  • The authors ask whether MSA models over-rely on language and what implications that dependence has for modality robustness.Language is described as the most effective modality in MSA.Source: §1 · p. 1
  • Removing language sharply lowers MISA performance, while removing audio or visual modalities produces no statistically significant drop.The comparison is on CMU-MOSI's test set.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The study proposes diagnostic checks for missing or noisy modalities across five state-of-the-art MSA models and two benchmark datasets.Source: §1 · p. 2
  • The authors study robust training strategies and report that robustness improvements do not trade off against final clean-test performance.The training methods are described as model-agnostic and easy to include in existing multimodal models.Source: §1 · p. 2

2 Related Works

Prior work has examined modality contributions and language-centered MSA, while robustness has received less attention. This paper distinguishes its diagnostics and training from earlier approaches by emphasizing integration into existing models and analysis across multiple models.

2 Related Works

  • Prior work examines modality contribution balancing and language-centered MSA frameworks, while this paper explores how language dominance relates to robustness.The cited approaches include Gat et al. (2020), Wu et al. (2021), and Han et al. (2021a).Source: §2 · p. 2
  • Earlier MSA robustness approaches include a factored model accommodating modality drops and training/testing modality drops with meta-learning.The works are Tsai et al. (2018) and Ma et al. (2021).Source: §2 · p. 2
  • Unlike those approaches, the paper's diagnostics and robust training require no sophisticated architecture, and its analysis spans multiple MSA models.Source: §2 · p. 2

3 Testing Robustness via Diagnostic Checks

Across MOSI and MOSEI, diagnostic checks find language representations sensitive to missing and noisy modality errors across the tested models. The checks intervene on language inputs, with errors applied to 30% of test data.

3.1 Experiment Setup

  • The experiments cover five diverse state-of-the-art models, ranging from RNN- to Transformer-based and word- to sentence-level variants.The models include MISA, BBFN, Self-MM, MMIM, and MulT.Source: §3.1 · p. 2
  • The evaluation uses CMU-MOSI and CMU-MOSEI, datasets of short-utterance videos with language, audio, and visual features.CMU-MOSEI is the larger counterpart, with richer annotations and more diverse samples.Source: §3.1 · p. 3

3.2 Proposed Diagnostic Checks

  • The study tests missing modalities by nulling a modality and noisy modalities by adding white Gaussian noise to representations.The diagnostic checks are applied to 30% of test data, with the study focused on language errors.Source: §3.2 · p. 3
  • Across MOSI and MOSEI, language modality is highly sensitive to missing and noisy errors across all tested models.The checks simulate errors on 30% of the test set.Source: §3.2 · p. 3, Figure 2 · p. 3, Figure 3 · p. 4

4 Robust Training

Modality perturbation during training improves robustness to missing and noisy language errors across tested models, while retaining performance similar to original models on clean data. The training approach balances missing and noisy perturbations.

4 Robust Training

  • Modality perturbation trains models on simulated errors by applying missing and noisy perturbations to the dominant modality, language.For each batch, a proportion of data is perturbed, with half receiving missing and half noisy perturbations; testing uses the diagnostic checks.Source: §4 · p. 4

4.1 Results

  • Robust training improves both missing- and noisy-modality diagnostics when 15% of training data receives each perturbation.The reported results average three random runs.Source: §4.1 · p. 4, Table 1 · p. 5
  • 31% (in F1) and 98% on noise: balanced perturbation reduces BBFN-MOSI relative drops on missing-language and noisy diagnostics, respectively.For BBFN-MOSI, missing-language diagnostic relative drop also reduces by 11% in Corr; noisy diagnostic relative drop reduces by 99%.Source: §4.1 · p. 4, Table 1 · p. 5
  • Robust training retains performance similar to original model variants on clean test data.The authors report no degradation in original performance.Source: §4.1 · pp. 4–5

5 Conclusion

The study finds that dominant modalities make state-of-the-art MSA models susceptible to performance drops under representation errors. It also finds that modality-perturbation robust training can improve robustness without an explicit performance trade-off.

5 Conclusion

  • Diagnostic checks reveal high susceptibility to performance drops when state-of-the-art MSA models encounter unwanted errors in their representations.The systematic study describes the dominant modality as double-edged.Source: §5 · p. 5
  • The authors study robust training methods that use modality perturbations to alleviate the identified robustness issues.Source: §5 · p. 5
  • Robustness and clean performance can co-exist without an explicit trade-off, though the authors note room for further improvement.Source: §5 · p. 5

A Model Details

The appendix specifies where modality interventions are applied in each of five models, relative to their encoded language representations.

A Model Details

  • In MISA, interventions occur before the language representation is projected into shared and private subspaces.Source: A Model Details · p. 7
  • In BBFN, interventions are applied after the language embedding.Source: A Model Details · p. 7
  • In Self-MM, interventions are placed after the encoded language features.Source: A Model Details · p. 7
  • In MMIM, interventions are applied after the encoded language representation.Source: A Model Details · p. 7
  • In MulT, interventions are applied after the encoded language representation.Source: A Model Details · p. 7
  • Table 2 provides the hyper-parameter configuration used to train the models.Source: Table 2 · p. 7

B Reproducing Results

The authors trained each model to approach the performance reported in its respective paper and provide the hyper-parameters used for reproduction.

B Reproducing Results

C Additional Results on Modality-Perturbation

The additional results examine varying perturbation proportions during training and testing. For MISA, 30% perturbation is reported as optimal, while a 30% setting also benefits MulT on MOSEI.

C Additional Results on Modality-Perturbation

Loading 2205.15465v1…