PaperlayerView exact revision on arXiv

Source-linked AI summary

ConsMSA: Semantic Distribution Consistency Learning for Multimodal Sentiment Analysis

Pan Wang, Lipeng Ke, Huajun Ying, Pritish Mohapatra, Rohan Sarkar, Suresh Lakhani, Sankar Venkataraman, Jingtong Hu

arXiv:2610.31731v1cs.LGcs.MM

TL;DR

Multimodal sentiment analysis faces redundant and conflicting token evidence within and across modalities, which can add noise and unnecessary computation. ConsMSA couples semantic consistency with predictive relevance to guide learning and token optimization. It reports state-of-the-art performance on three benchmarks and competitive results after discarding 90% of tokens.

  • Problem

    In multimodal sentiment analysis, token evidence can be redundant within a modality or semantically conflicting across modalities, complicating coherent sentiment modeling.

    Source: §1 · p. 1, §3.3 · p. 3

  • Method

    ConsMSA computes intra- and inter-modality consistency in a shared sentiment space and couples it with predictive relevance to guide regularization, reweighting, and token pruning.

    Source: §1 · p. 2, §3.3 · p. 4, §3.5 · p. 5

  • Results

    ConsMSA outperforms prior state-of-the-art approaches on three benchmarks and maintains competitive performance after discarding 90% of tokens.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The results suggest that explicitly optimizing semantic consistency is a practical route to accurate, compute-efficient multimodal sentiment models.

    Source: §5 · p. 9

    The dataset’s long-tailed distribution poses a challenge for fine-grained sentiment understanding and motivates future exploration.

    Source: §4.6 · p. 9

Abstract

from arXiv · show

Multimodal sentiment analysis (MSA) aims to predict human sentiments by integrating signals from different modalities such as text, video, and audio. However, raw multimodal sequences often suffer from semantic inconsistency--exhibiting redundancy or conflicts within and across modalities--which hinders robust understanding and increases computational cost. To this end, we introduce ConsMSA, which explicitly formalizes semantic distribution consistency across both $\textit{intra}$- and $\textit{inter}$-modality, providing a consistency-aware mechanism for robust and efficient multimodal sentiment prediction. Specifically, ConsMSA projects multimodal token features into a shared sentiment space to compute an Intra- and Inter-modality Consistency Score ($I^2CS$). By coupling this score with predictive relevance, we formulate consistency-aware importance signals that are utilized: (i) as a consistency regularizer to align latent distributions during training, (ii) to derive semantic-aware weights for adaptive multimodal token reweighting, and (iii) as a practical criterion to prune redundant or conflicting tokens. Extensive experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate that ConsMSA achieves state-of-the-art performance while remaining robust under aggressive token compression--retaining only 10% of tokens yields comparable accuracy. These results establish semantic distribution consistency as a promising foundation for synergizing predictive robustness with computational efficiency.

1. Introduction

ConsMSA treats intra- and inter-modality semantic inconsistency as a source of noise and redundant computation in multimodal sentiment analysis. It couples consistency with predictive relevance to guide learning and token optimization, and reports state-of-the-art results across three benchmarks with robustness under token compression.

1. Introduction

  • Intra-modal redundancy or neutral signals and inter-modal semantic conflicts inject noise into fusion and make models spend computation on tokens that do not support coherent sentiment representations.Sarcasm is one example of cross-modal conflict.Source: §1 · pp. 1–2
  • ConsMSA couples predictive relevance with I2CS to guide consistency regularization, adaptive token reweighting, and pruning of redundant or conflicting tokens.The signals reward tokens that are both informative and semantically coherent.Source: §1 · p. 2
  • The framework formalizes consistency within and across modalities, with I2CS using JS divergence to quantify semantic agreement in the shared semantic space.This frames consistency as a single metric spanning intra- and inter-modality.Source: §1 · p. 2
  • ConsMSA achieves state-of-the-art results on CMU-MOSI, CMU-MOSEI, and CH-SIMS while preserving competitive performance under aggressive token compression.The introduction reports these benchmark and compression outcomes without specifying metric values.Source: §1 · p. 2

2. Related Work

Prior MSA work improves fusion through representation alignment or reduces computation through token selection, but these approaches do not explicitly assess token consistency with sentiment decisions. ConsMSA measures prediction-level agreement and couples it with relevance to support both consistency learning and token optimization.

2. Related Work

  • Representation-alignment methods improve features for fusion but do not explicitly quantify whether token-level evidence agrees with the downstream sentiment decision.The cited approaches include shared/private factorization, metric learning, adversarial alignment, and decoupled distillation.Source: §2 · p. 2
  • Existing token-selection methods use attention, entropy, or learned pruning, but feature-space scores may discard low-saliency tokens that are sentiment-critical.This leaves a gap between token scoring and the sentiment decision boundary.Source: §2 · p. 2
  • ConsMSA measures sentiment prediction-level agreement with JS divergence and combines it with relevance to support consistency regularization, reweighting, and compression.The framework overview places these importance signals before optimized tokens are concatenated for final prediction.Source: §2 · p. 2, Figure 2 · p. 3

3. Method

ConsMSA represents multimodal tokens as sentiment distributions and measures their intra- and inter-modality consistency. It couples consistency with predictive relevance to regularize learning and guide token reweighting and pruning.

3.1. Preliminaries

  • MSA is framed as predicting a categorical or continuous sentiment label from audio and text inputs represented as token-level features.The method treats each token as inducing a sentiment distribution in a shared semantic space.Source: §3.1 · p. 3

3.2. Overview

  • ConsMSA projects contextual tokens into a shared semantic distribution space, couples consistency with relevance, and uses the resulting signals for regularization, reweighting, and compression.The optimized tokens are concatenated for final prediction.Source: §3.2 · p. 3

3.3. Intra- and Inter-modality Consistency Score(I2CS)

  • JS divergence between token-induced sentiment distributions provides a symmetric, finite, bounded basis for measuring intra- and inter-modality consistency.IntraCS compares tokens within a modality, while InterCS compares tokens across paired modalities; lower I2CS indicates higher consistency.Source: §3.3 · pp. 3–4

3.4. I2CS-based Consistency Loss

  • Consistency is enforced more strongly on confident tokens by weighting I2CS with token relevance, while avoiding over-penalization of uncertain tokens.The relevance term measures confidence in the latent sentiment prediction.Source: §3.4 · p. 4

3.5. Token Optimization Module via I2CS

  • The importance signal rewards predictive confidence and penalizes semantic disagreement, enabling differentiable soft weights and TopK selection of tokens.Soft gating gives greater weight to confident, consistent evidence; hard selection retains the top-K tokens for subsequent computation.Source: §3.5 · p. 5

Algorithm 1 End-to-end ConsMSA Pipeline

  • The pipeline takes raw audio and text inputs with sentiment labels and returns a prediction and total loss.The listed steps include semantic projection and prediction through modality heads.Source: Algorithm 1 End-to-end ConsMSA Pipeline · p. 5

3.6. Final Learning Objective

  • The joint objective optimizes prediction accuracy, consistency regularization, and modality-specific supervision.The task loss is the primary target-space supervision signal, and modality-specific losses supervise predictions before fusion.Source: Algorithm 1 End-to-end ConsMSA Pipeline · p. 5

4. Experiments

ConsMSA is evaluated across three multimodal sentiment datasets, with results supporting its performance, component choices, and robustness to token compression. Further analyses examine masking signals and token-level consistency, while MOSI class imbalance constrains fine-grained accuracy.

4.1. Datasets and Evaluation Metrics

  • Evaluation spans MOSI, MOSEI, and Chinese CH-SIMS, using Acc-7, Acc-2, F1, and MAE, plus Acc-5 and Acc-3 for CH-SIMS.Source: §4.1 · p. 6

4.2. Implementation Details

  • Experiments use RoBERTa text embeddings and Data2Vec-Audio representations, trained with AdamW on a single NVIDIA A100 GPU.The reported setup uses a 5 × 10−6 initial learning rate, batch size 8, and early stopping with patience 8.Source: §4.2 · p. 6

4.3. Main Results

  • ConsMSA achieves the best results across all metrics on MOSI and MOSEI with both BERT-based T+A+V and RoBERTa-based T+A settings.The comparison covers prior state-of-the-art methods.Source: §4.3 · pp. 6–7, Table 1 · p. 7
  • 1.31% Acc-3 (69.37% to 70.68%) and 1.39% Acc-5 (49.38% to 50.77%) gains over MMML establish bi-modal ConsMSA as CH-SIMS state of the art.The main-text comparison uses the T+A setting.Source: Table 2 · p. 7, §4.3 · p. 7

4.4. Ablation Study

  • Removing either consistency term, relevance, gating, or either auxiliary loss degrades MOSI performance, supporting the integrated design.The ablations cover IntraCS, InterCS, Rel, LMS, and LI2CS.Source: §4.4 · pp. 7–8, Table 3 · p. 8

4.5. Further Analysis

  • 10% token retention remains highly robust; the optimal retention ratio differs by dataset, at 0.6 for CH-SIMS and 0.8 for MOSI.On MOSI, hard selection at 0.8 yields Acc-2 90.85%, versus 90.09% for soft selection at 1.0.Source: Table 4 · p. 8, §4.5 · p. 8
  • 51.46% Acc-7 and 0.557 MAE mark ConsMSA’s best masking results; KL-div underperforms it, while cosine and L2 fall below random masking.Source: Table 5 · p. 8, §4.5 · p. 8

4.6. Visualization

  • After consistency-aware training, intra- and inter-modality token consistency improves; on MOSI, HN and HP accuracy is lower, with only 46 and 20 samples, respectively.The paper identifies this long-tailed distribution as a challenge for fine-grained sentiment understanding.Source: Figure 4 · p. 9, §4.6 · p. 9, Figure 5 · p. 9

5. Conclusion

ConsMSA models semantic distribution consistency within and across modalities, using I2CS in a shared prediction space to support regularization, reweighting, and token compression. Across three benchmarks, it outperforms prior state-of-the-art approaches and remains competitive after discarding 90% of tokens.

5. Conclusion

  • Across three benchmarks, ConsMSA outperforms prior state-of-the-art approaches and maintains competitive performance after discarding 90% of tokens.The results support semantic consistency optimization as a practical route to accurate, compute-efficient multimodal sentiment models.Source: §5 · p. 9
  • I2CS in a shared prediction space provides a unified signal for consistency regularization, relevance-aware reweighting, and token compression.ConsMSA explicitly models semantic distribution consistency within and across modalities.Source: §5 · p. 9

Impact Statement

The impact statement identifies intra- and inter-modality consistency as important to multimodal learning and describes the proposed learning paradigm as adaptable to other multimodal tasks. Robust consistency learning with missing modalities is left for future work.

Impact Statement

  • Modeling intra- and inter-modality consistency offers a new perspective for exploiting multimodal learning.The statement identifies this as an important insight of the study.Source: Impact Statement · p. 9
  • The consistency-aware learning paradigm can be extended to other multimodal tasks by adapting its scoring inputs.The impact statement characterizes the paradigm as general and flexible.Source: Impact Statement · p. 9
  • Robust consistency learning under modality-missing settings is identified as future work.Source: Impact Statement · p. 9

A.1. Configuration of Hyperparameters

The appendix reports that benchmark hyperparameters were selected using validation performance and gives a broad single-GPU training-time range.

A.1. Configuration of Hyperparameters

  • Hyperparameters for each benchmark were optimized based on validation-dataset performance.Table 6 presents the configurations; its caption identifies T as text and A as audio and says token length depends on the BERT/RoBERTa backbone.Source: §A.1 · p. 12, Table 6 · p. 12
  • Training takes 30 minutes to several hours on a single NVIDIA A100-40G GPU.Source: §A.1 · p. 12

A.2. Details for the Three-Modality Implementation

For three-modality use, ConsMSA averages each target modality’s pairwise scores with the other modalities and uses the result to guide token optimization.

A.2. Details for the Three-Modality Implementation

  • For each target modality, ConsMSA averages its two independently computed pairwise scores with the other modalities to guide token optimization.This extends the bimodal scoring procedure to the trimodal setting while retaining the same optimization principles.Source: §A.2 · p. 12

Impact of Different Modality in the RoBERTa-based

ConsMSA benefits from complementary modalities and holistic cross-modal consistency, with reported gains across MOSI and CH-SIMS. Its token compression also reduces inference latency while keeping runtime memory nearly unchanged.

Impact of Different Modality in the RoBERTa-based

  • With all modality-pair consistency signals, the BERT-based model achieves the best performance across metrics, including 47.96% Acc-7 and 0.698 MAE.Removing any pair degrades performance, while text-involving pairs make larger individual contributions.Source: Impact of Different Modality in the… · p. 12, Table 8 · p. 12

Impact of Modality Selection in the RoBERTa-based

A.4. Brief Introduction of All Baselines

  • The baseline appendix describes varied MSA strategies, including shared/private-space separation, modality adaptation gates, mutual-information objectives, expert fusion, and tensor-based fusion.Examples include MISA, MAG-BERT, MMIM, EMOE, and TFN.Source: §A.4 · p. 14

Related papers

Loading 2610.31731v1…