PaperlayerView exact revision on arXiv

Source-linked AI summary

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Devamanyu Hazarika, Roger Zimmermann, Soujanya Poria

arXiv:2005.03545v3cs.CLcs.LG

TL;DR

Multimodal affect analysis often relies on fusion despite gaps between heterogeneous modalities. MISA learns invariant and modality-specific representations before fusion, achieving gains over state-of-the-art approaches in sentiment analysis and humor detection.

  • Problem

    Existing multimodal sentiment approaches emphasize sophisticated fusion, while heterogeneous modalities retain distributional gaps that complicate multimodal learning.

  • Method

    MISA factorizes each modality into modality-invariant and modality-specific representations, then fuses them to predict affective states.

  • Results

    MISA achieves significant gains over state-of-the-art approaches across multimodal sentiment analysis and humor detection tasks.

  • Takeaways & Limitations

    Learning multimodal representations before fusion can support effective affective prediction without relying on complex fusion mechanisms.

  • Takeaways & Limitations

    The study evaluates sentiment and humor, leaving other affective dimensions and alternative fusion, similarity, and regularization choices for future analysis.

Abstract

from arXiv · show

Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.

1 INTRODUCTION

MISA addresses modality gaps in multimodal affect analysis by learning modality-invariant and modality-specific representations before fusion. It uses these disentangled representations across sentiment and humor detection benchmarks to support affective prediction.

  • Motivation: Most multimodal sentiment approaches focus on sophisticated fusion, but heterogeneous language, audio, and visual signals create distributional modality gaps.Existing fusion mechanisms include attention-based and tensor-based models.
  • Method: MISA maps each modality into a shared modality-invariant subspace with distributional alignment to reduce modality gaps and capture cross-modal commonalities.The shared information reflects common speaker motives and goals associated with an utterance’s overall affective state.
  • Method: MISA also learns modality-specific subspaces that preserve distinctive speaker-sensitive and stylistic characteristics useful for affect prediction.Examples include sarcasm and peculiar expressions that may be uncorrelated across modalities.
  • Training: The framework jointly uses distributional similarity, orthogonal, reconstruction, and task prediction losses to learn representative invariant and specific features.These losses target invariant-feature similarity, specific-feature separation, modality-feature representativeness, and task prediction.
  • Evaluation: MISA is evaluated on MOSI and MOSEI for multimodal sentiment analysis and on a multimodal humor detection task using the UR_FUNNY dataset.The paper tests the framework’s adaptability beyond sentiment analysis.

2 RELATED WORKS

Prior multimodal sentiment analysis work primarily differs in whether it models utterances independently or incorporates inter-utterance context, with substantial emphasis on sophisticated fusion. MISA instead prioritizes modality-invariant and modality-specific representation learning before fusion, while drawing on shared-private subspace methods and differing from DSN in its factorization setting.

  • Multimodal Sentiment Analysis: MSA research broadly comprises utterance-level models that analyze target utterances independently and inter-utterance models that use neighboring video utterances.
  • Utterance-level: Utterance-level approaches learn cross-modal dynamics through multiple-kernel, tensor-based, low-rank, and word-level fusion mechanisms.
  • Inter-utterance context: Inter-utterance models use hierarchical networks to represent individual utterances and sequential context, with later work improving attention, hierarchical fusion, or contextual modeling.
  • MISA distinction: MISA does not use contextual information or focus on complex fusion mechanisms; it emphasizes representation learning before fusion while remaining flexible enough to incorporate those components.
  • Subspace learning: Common-subspace research includes translation-based, correlation-based, and jointly learned shared-subspace models, while factorized-representation work motivates MISA’s modality-invariant and modality-specific representations.
  • Shared-private representations: MISA uses a discriminative deep architecture without approximate inference and differs from DSN by factorizing modalities within utterances rather than instances.

3 APPROACH … 3.4 Modality Fusion

MISA processes each video utterance through language, visual, and acoustic sequences, learns shared and modality-specific representations, and fuses the six resulting vectors with self-attention for affective prediction. The task supports categorical or continuous affective outputs from multimodal utterance inputs.

  • 3.1 Task Setup: Each video is segmented into utterances, each represented by language, visual, and acoustic low-level feature sequences.For modality m, the input is U_m ∈ R^T_m×d_m.
  • 3.1 Task Setup: The model predicts an utterance’s affective orientation either over C predefined categories or as a continuous intensity variable.The categorical target is y ∈ R^C, while the continuous target is y ∈ R.
  • 3.2 MISA: MISA comprises two stages: Modality Representation Learning and Modality Fusion.The framework is illustrated in Fig. 2.
  • 3.3 Modality Representation Learning: For each modality, a stacked bidirectional LSTM followed by a fully connected layer maps the utterance sequence to a fixed-sized vector.Each input sequence U_m is transformed into u_m ∈ R^d_h.
  • 3.3 Modality Representation Learning: Each utterance vector is projected into modality-invariant and modality-specific representations to combine shared information with modality-characteristic features.The invariant component uses a common subspace with distributional similarity constraints to reduce the heterogeneity gap, while the specific component captures unique modality characteristics.
  • 3.3 Modality Representation Learning: The invariant encoder shares parameters across language, visual, and acoustic modalities, whereas the specific encoder uses separate parameters for each modality.Both encoders use simple feed-forward neural layers.

3.5 Learning

MISA trains by minimizing a weighted combination of regularization components and task-specific loss. The regularizers align invariant representations, separate invariant from modality-specific representations, and preserve modality details through reconstruction.

  • Learning: The overall loss combines regularization components weighted by α, β, and γ, with each component enforcing desired subspace properties.These interaction weights determine each regularization component’s contribution to the overall loss L.
  • Similarity loss: The similarity loss uses Central Moment Discrepancy to reduce discrepancies between modalities’ shared representations and align common cross-modal features.CMD matches order-wise moment differences between representation distributions.
  • Difference loss: The difference loss applies soft orthogonality constraints so modality-invariant and modality-specific representations capture different, non-redundant aspects of the input.Orthogonality constraints are also imposed between modality-specific vectors.
  • Reconstruction loss: The reconstruction loss discourages trivial modality-specific representations by reconstructing each modality vector from the sum of its invariant and specific hidden representations.Reconstruction uses a decoder and mean squared error between the original and reconstructed modality vectors.
  • Task-specific loss: The task-specific loss evaluates prediction quality during training, using cross-entropy for classification and mean squared error for regression.The loss choice depends on the task type.

4 EXPERIMENTS

The experiments evaluate MISA on multimodal sentiment analysis using MOSI and MOSEI, and on multimodal humor detection using UR_FUNNY. They specify benchmark metrics and features, and compare MISA with established sentiment and humor baselines.

  • Datasets: MOSI contains 2,198 utterance-video segments from 93 videos and 89 speakers, with sentiment scores ranging from −3 to 3.The dataset consists of YouTube monologues with manually annotated continuous opinion scores.
  • Datasets: MOSEI contains 23,453 annotated utterances from 5,000 videos, 1,000 speakers, and 250 topics.MOSEI provides more utterances and broader sample, speaker, and topic diversity than MOSI.
  • Datasets: UR_FUNNY provides multimodal TED-talk punchlines with associated context, speaker and topic diversity, and binary humor/nonhumor labels.The dataset is used for multimodal humor detection.
  • Evaluation: MOSI and MOSEI sentiment intensity use MAE and Pearson correlation, alongside Acc-7, Acc-2, and F-Score classification metrics.Acc-7 covers seven sentiment classes from −3 to 3, while binary accuracy includes distinct labeling approaches.
  • Implementation: For fair comparisons, the experiments use benchmark-provided low-level features and replace traditional GloVe text features with pre-trained BERT for MOSI and MOSEI.UR_FUNNY results are reported with both GloVe and BERT because its state-of-the-art baseline uses GloVe features.
  • Baselines: MISA is compared against utterance-level and inter-utterance baselines, including ICCN for sentiment analysis and C-MFN for humor detection.The comparison set includes temporal, attention, graph-based, tensor-fusion, common-subspace, recurrent, contextual-attention, and multitask models.

5 RESULTS AND ANALYSIS

MISA outperforms prior models across multimodal sentiment and humor benchmarks, including BERT-based comparisons, while ablations and representation analyses support the value of multimodal, regularized invariant and modality-specific representations.

  • Benchmark Results: MISA achieves the best performance on MOSI and MOSEI across combined regression and classification metrics, surpassing baselines including ICCN.The utterance-level model also outperforms contextual models despite using less information.
  • Benchmark Results: MISA shows a pronounced improvement over contextual SOTA C-MFN on humor detection, while its GloVe variant performs at par with BERT-based TFN.The result highlights the importance of effective multimodal representation modeling for humor detection.
  • Benchmark Results: MISA beats the BERT-based ICCN across all metrics, indicating that its gains are not solely attributable to improved language features.The comparison isolates multimodal modeling as a critical factor in the performance improvements.
  • Ablation Analysis: The full multimodal model performs best, while removing language causes the sharpest performance drop, demonstrating complementary modalities and language dominance.The text advantage may reflect higher-quality manual transcriptions and BERT’s pretraining relative to raw audio and visual signals.
  • Representation and Regularization Analysis: All regularization losses decrease across training and validation, while t-SNE analyses show modality-invariance without regularization is not learned and modality-specific representations remain informative.The best performance occurs when all losses are retained, with particular sensitivity to similarity and difference losses.
  • Variant Analysis: The final MISA design outperforms its variants; invariant-only learning is too restrictive, whereas combining invariant and specific representations yields the best performance.The analysis attributes this restriction to modalities not always sharing the same polarity stimulus.

6 CONCLUSION

MISA factorizes modalities into modality-invariant and modality-specific features, fuses them for affective prediction, and achieves significant gains despite using simple feed-forward layers. Future work will extend the framework across affective dimensions, fusion schemes, and similarity/difference loss choices.

  • Conclusion: MISA factorizes modalities into modality-invariant and modality-specific features, then fuses them to predict affective states.The framework is presented as a multimodal affective framework.
  • Conclusion: Despite comprising simple feed-forward layers, MISA achieves significant gains over state-of-the-art approaches in multimodal sentiment analysis and humor detection.The reported tasks are multimodal sentiment analysis and humor detection.
  • Future Work: Future work will analyze MISA for other affective dimensions, including emotions, and combine it with other fusion schemes for further improvements.These directions are explicitly identified as planned extensions.
  • Future Work: The authors also plan to study alternative metrics and regularization choices for similarity and difference loss modeling.The passage notes that these losses allow various metric and regularization choices.

A BASELINE MODELS

The paper evaluates against diverse multimodal baseline models spanning memory-based, graph-based, tensor-based, and contextual-utterance approaches. These baselines capture intra-modal, cross-modal, and hierarchical multimodal interactions through varied fusion and sequence-modeling mechanisms.

  • Memory- and sequence-based baselines: MFN, RAVEN, MARN, and MV-LSTM represent memory- and LSTM-based baselines for modeling intra-modal, cross-modal, or context-dependent interactions.MFN stores intra- and cross-view interactions; RAVEN re-adjusts word embeddings using multimodal context; MARN uses a hybrid LSTM memory component.
  • Fusion-based baselines: RMFN, Graph-MFN, TFN, and LMF cover multi-stage, graph-based, and tensor-based fusion strategies for combining multimodal signals.Graph-MFN adds a dynamic interaction graph to MFN, while TFN uses an outer-product tensor to capture uni-, bi-, and tri-modal interactions.
  • Contextual-utterance baselines: The contextual-utterance baselines frame multimodal analysis around information from the surrounding video utterances.The paper explicitly introduces a contextual-utterance family before listing its constituent models.
  • Contextual-utterance baselines: BC-LSTM, CH-FUSION, CIM-MTL, and CIA include structured prediction, hierarchical fusion, and multitask sentiment-emotion modeling among contextual baselines.BC-LSTM models the whole video bidirectionally; CH-FUSION composes bi-modal and tri-modal fusion; CIM-MTL jointly analyzes sentiment and emotion.

B DATASET SIZES

This section identifies Table 5 as the source for dataset sizes, measured by the number of utterances in each dataset.

  • Dataset sizes: Table 5 reports the sizes of each dataset in terms of utterance counts.The passage does not provide the individual dataset values.

C HYPER-PARAMETER SELECTION

Hyper-parameters are selected by grid search on dataset validation sets, using the model with the best validation classification or regression loss. The search varies activation functions, α, β, γ, and dropout.

  • Selection procedure: Grid search over finite hyper-parameter sets selects the model with the best validation classification or regression loss.The validation sets provided in the datasets are used for hyper-parameter selection.
  • Search space: The activation-function candidates are leakyrelu, prelu, elu, relu, and tanh.The search also considers α ∈ {0.7, 1.0}, β ∈ {0.3, 0.7, 1.0}, and γ ∈ {0.7, 1.0}.
  • Search space: Dropout values are searched over {0.1, 0.5, 0.7}.

D NETWORK TOPOLOGY

Figure 6 presents the network topologies of the final models used for each dataset, including utterance-level modality-invariant and modality-specific representations and language encoded with BERT(U_l).

  • Network topologies: Figure 6 describes the network topologies used by the final models for the different datasets.The figure provides the topology descriptions for each dataset.
  • Network topologies: The topologies include utterance-level modality-invariant and modality-specific representations.These representation types are explicitly identified in the topology descriptions.
  • Network topologies: Language is represented using BERT(U_l) in the depicted topologies.The language branch is labeled “Language — BERT( Ul )”.
Loading 2005.03545v3…