PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Multimodal Sentiment Analysis with Incomplete Modalities via Semantic-aware Completeness based Reconstruction

Han-Jun Choi, Byunggill Joe, Saim Shin, Jin Yea Jang

arXiv:2609.10950v1cs.CLcs.AIcs.LG

TL;DR

Text-centric multimodal sentiment analysis degrades when incomplete or noisy inputs remove sentiment-relevant information, while existing completeness estimates may not reflect semantic loss. The paper estimates semantic completeness, uses it to guide reconstruction, and stabilizes joint training; across three benchmark datasets, the approach consistently outperforms 12 competitive baselines.

  • Problem

    Text-centric multimodal sentiment methods can suffer substantial degradation when modalities are partially missing or noisy, especially when incomplete text removes key sentiment cues.

  • Method

    The method estimates semantic completeness with TPSC supervision, adaptively combines incomplete text with auxiliary-modality proxy features, and uses AOS to stabilize joint optimization.

  • Results

    The proposed method consistently outperforms 12 competitive baselines across three benchmark datasets with varying missing rates.

  • Takeaways & Limitations

    Semantic-aware completeness estimation supports more reliable reconstruction of missing textual semantics for multimodal sentiment analysis.

  • Takeaways & Limitations

    The study focuses on reconstructing semantic information loss in text and relies on pseudo-labeling that may introduce noise into supervision.

Abstract

from arXiv · show

Recent multimodal sentiment analysis studies increasingly adopt text-centric fusion approaches to exploit the rich sentiment information inherent in the textual modality. However, these approaches often suffer from performance degradation during inference due to partially missing or noisy data in real-world scenarios, especially when sentiment-related cues are missing. To address this issue, we introduce a new completeness estimation approach that quantifies the degree of sentiment-relevant information preserved in incomplete data to guide the reconstruction of missing semantics. Furthermore, we propose a training strategy that stabilizes multi-task learning while jointly optimizing sentiment prediction and completeness estimation. Extensive experiments and in-depth analyses on three benchmark datasets demonstrate that the proposed approach enables more accurate semantic reconstruction, leading to more precise sentiment prediction.

1 Introduction

Text-centric multimodal sentiment methods are vulnerable when incomplete data removes sentiment cues, because missing-rate completeness can misrepresent semantic loss. The paper proposes semantic-aware completeness estimation, adaptive reconstruction, and stabilized multitask optimization, and reports consistent gains over 12 baselines on three datasets.

  • Text-centric approaches perform strongly with complete data but degrade substantially when modalities are partially missing or noisy, especially when text is incomplete.
  • Missing-rate completeness can overestimate or underestimate preserved sentiment information when key cues are absent or retained despite similar missing rates.Figure 1 contrasts a missing key cue, such as “love,” with a case where sentiment meaning remains intact.
  • The method estimates incomplete-text completeness semantically, uses TPSC pseudo-labels, and adaptively combines incomplete text with auxiliary-modality proxy features for reconstruction.
  • The paper introduces semantic-aware completeness estimation to quantify semantic preservation in incomplete data.
  • AOS mitigates gradient conflicts between completeness estimation and sentiment prediction in the shared encoder, stabilizing hierarchical multitask learning.
  • The proposed method consistently outperforms 12 competitive baselines across three benchmark datasets with varying missing rates.

2 Related Work

Multimodal sentiment methods integrate textual, acoustic, and visual information, while text-centric approaches treat text as dominant and other modalities as auxiliary. Existing methods often assume complete modalities, and reconstruction approaches generally do not explicitly model semantic information loss in incomplete text.

  • Text-centric methods use text as the dominant modality and audio and visual signals as auxiliary sources to enhance textual representations.
  • Assuming fully available modalities limits robustness in real-world inference, particularly when important textual semantics are lost.
  • Reconstruction-based approaches restore missing semantics, including methods that derive proxy features from audio and vision and weight reconstruction using text missing rates.
  • Existing reconstruction approaches do not explicitly account for semantic information loss in incomplete text.

3 Proposed Method

TCMR estimates how much sentiment-relevant information remains in incomplete text and uses that estimate to guide semantic reconstruction before text-centric sentiment fusion. Its training alternates completeness learning with reconstruction and sentiment optimization to reduce multi-task gradient conflict.

  • Framework Overview: TCMR estimates textual completeness and reconstructs missing semantics by adaptively weighting incomplete text with proxy features from auxiliary modalities.The reconstructed representation is then passed to a text-centric multimodal fusion module for sentiment prediction.
  • Text Completeness Estimation: Completeness is defined by preserved sentiment-related information and supervised with pseudo-labels derived from a text-only classifier's target-class probability.The classifier is trained on complete text, then its probability for the ground-truth polarity class on incomplete text becomes the completeness label.
  • Proxy Feature Generation: The Importance-aware Proxy Feature Generator adaptively balances audio and vision contributions using modality-specific generators and a sigmoid-based gating network.The gating weight controls the relative contribution of auxiliary modalities to proxy feature generation.
  • Semantic Reconstruction: The reconstructed text representation integrates incomplete text with generated proxy features under the predicted completeness weight and is trained toward complete representations with mean squared error.Auxiliary modalities are additionally self-reconstructed from their incomplete representations.
  • Alternating Optimization Strategy: Naive end-to-end optimization can suppress completeness learning through gradient conflict and shortcut solutions, leaving the completeness estimator at a poor local minimum.The sentiment task may be minimized without learning reliable completeness estimation.
  • Alternating Optimization Strategy: AOS alternates two phases within each epoch: it first optimizes completeness parameters with the completeness loss, then fixes them while optimizing reconstruction and sentiment modules.This separates completeness learning from the remaining objectives during training.

4 Experiments

Experiments evaluate TCMR across MOSI, MOSEI, and SIMS under random modality missingness, showing that semantic-aware completeness estimation supports robust reconstruction and sentiment prediction. TCMR-TPSC consistently outperforms baselines across increasing missing rates and achieves strong overall benchmark results.

  • Experimental Setup: The experiments use MOSI, MOSEI, and SIMS with random modality erasure during training and evaluation across missing rates from 0 to 0.9.Training samples missing rates uniformly from [0, 1.0), while testing uses ten settings in increments of 0.1.
  • Experimental Setup: The study compares TCMR-TPSC with missing-rate-based and lexicon-derived completeness variants alongside competitive baselines.TCMR-MRSC uses text missing rate, whereas TCMR-LDSC derives labels from token-level sentiment strengths.
  • Main Results: TCMR-TPSC achieves state-of-the-art performance across all MOSI evaluation metrics, improving Acc-5 by 6.68% and reducing MAE by 6.37% versus LNLN.Among TCMR variants, TCMR-TPSC performs best across datasets because it captures semantic loss more accurately than missing-rate or token-score alternatives.
  • Main Results: TCMR-TPSC consistently outperforms baseline methods as the missing rate increases across the three benchmark datasets.Performance curves compare accuracy and MAE across missing rates on MOSI, MOSEI, and SIMS.

5 Ablation Study

The ablation study examines optimization strategies and model components, finding that coordinated multi-task optimization and IPFG are important for TCMR performance. Sequentially freezing the completeness-learning stage produces the weakest results across datasets.

  • Experimental Design: The ablation compares CompNet-first, CompNet-later, End2End, and component-removal settings on MOSI, MOSEI, and SIMS.Tables 3 and 4 report the framework ablations for the two dataset groups.
  • Optimization Strategy: CompNet-first consistently achieves the lowest performance across all datasets.Freezing the shared BERT encoder after completeness learning can misalign completeness estimation with sentiment prediction.
  • Optimization Strategy: Continuously updating the shared BERT encoder across completeness and sentiment objectives is necessary while preserving their hierarchical training structure.CompNet-later is competitive on MOSI but performs worse on MOSEI, whereas End2End remains suboptimal versus TCMR (Full).
  • Component Analysis: Removing IPFG consistently degrades performance, supporting its role in generating useful proxy features for reconstruction.The ablation evidence links IPFG removal with lower results across datasets.

6 In-depth Analysis

In-depth analyses show that sentiment-related tokens drive classification sensitivity and that TCMR handles missing-text cases more reliably than LNLN. Qualitative comparisons further indicate that semantic-aware completeness better reflects information loss than missing-rate or token-lexicon heuristics.

  • Sentiment Token Sensitivity: Masking affective tokens such as [’boring’], [’not’], and [’fun’] frequently causes sentiment misclassification.This supports using target-probability changes to estimate preservation of sentiment-relevant information.
  • Case Study: TCMR produces correct predictions under varying missing-text conditions, whereas LNLN often misreconstructs semantics and predicts sentiment incorrectly.The difference is attributed to TCMR’s more accurate completeness estimation and adaptive reliance on auxiliary modalities.
  • Completeness Labels: MRSC cannot capture semantic loss from token proportions alone, while LDSC and TPSC better reflect the importance of missing sentiment-related tokens.Table 5 compares the completeness labels assigned under random missing conditions.
  • Completeness Labels: LDSC can misestimate completeness because SentiWordNet assigns sentiment scores according to a token’s most frequent sense rather than its contextual meaning.The analysis highlights positive expressions such as [’fan’] and [’dig’] as examples.

7 Conclusion

The work reinterprets semantic information loss through changes in target-class probability, introduces pseudo-label supervision and gradient-conflict mitigation, and targets more stable joint learning for completeness estimation and sentiment prediction.

  • Target-class probability changes are used to reinterpret semantic information loss in incomplete data and address degradation from noisy or missing modalities.
  • Pseudo-labels supervise a completeness estimator, while the optimization strategy mitigates gradient conflicts between completeness estimation and sentiment prediction.

Limitations

The study’s scope is limited to textual semantic reconstruction, pseudo-label supervision, and the reported experimental setting, leaving extensions to other modalities and noise reduction for future work.

  • TCMR reconstructs semantic information loss only in the textual modality, leaving audio and vision extensions for future work.
  • Pseudo-labeling can introduce noisy supervision, motivating confidence-aware filtering, complementary LDSC use, and human verification for low-confidence samples.
  • MOSEI is the larger, more diverse, and more challenging benchmark among the three datasets, increasing variability across modalities.
  • The experiments use predefined sentiment-class mappings and report dataset-specific classification and regression metrics.

C Gradient Conflict Analysis

The analysis examines gradient conflicts in joint completeness-estimation and sentiment training, compares optimization behavior, and studies how sentiment-token masking affects pseudo-completeness labels.

  • Gradient Conflict Analysis: Joint optimization can create gradient conflicts that bias learning toward some tasks while under-optimizing others.
  • Gradient Conflict Analysis: Table 9 evaluates gradient conflict rates and convergence, including comparisons among end-to-end training, GradNorm, and PCGrad.
  • Lexicon-derived Completeness Label: LDSC estimates completeness from retained sentiment-related lexical information using SentiWordNet-derived token scores.
  • Token Sensitivity Analysis: The sensitivity procedure masks one token at a time and identifies tokens whose removal flips the classifier’s predicted polarity.
  • Token Sensitivity Analysis: As the confidence threshold increases, key sentiment tokens remain prominent while semantically ambiguous tokens disappear across MOSI and MOSEI.

D.3 Upper-Bound Analysis

The upper-bound analysis tests direct use of target pseudo-labels, while failure-case analysis examines pseudo-label noise and its relationship to dataset complexity and generalization.

  • Upper-Bound Analysis: TCMR-ub consistently outperforms TCMR across all metrics, indicating that target pseudo-labels provide an effective reconstruction-guidance signal.
  • Failure Cases of TPSC: Incorrect TPSC labels can arise when the pretrained classifier misclassifies samples even on complete text.
  • Failure Cases of TPSC: MOSEI’s lower classifier accuracy likely produces noisier pseudo-labels that can partially explain its weaker generalization performance.
  • Failure Cases of TPSC: Misclassifications mainly occur in ambiguous and sarcastic expressions, although low TPSC scores limit their impact on completeness estimation.

E.1 Metric-Level Comparison

TCMR-TPSC generally outperforms comparison methods across benchmark datasets and evaluation analyses, while metric choice can affect how sentiment quality is represented.

  • Metric-Level Comparison: Classification metrics can miss fine-grained sentiment differences because continuous predictions are discretized into classes before evaluation.Rounding can also produce inconsistencies near class boundaries.
  • Metric-Level Comparison: TCMR-TPSC consistently outperforms CENet and P-RMF across benchmark datasets, except for a few classification metrics.Table 13 compares TCMR-TPSC with CENet and P-RMF on MOSI, MOSEI, and SIMS.
  • Metric-Level Comparison: TCMR achieves the second-fastest training speed after TF-Mamba while using a lightweight MLP-based CompNet.The added AOS optimization steps do not significantly increase overall training cost.
  • Metric-Level Comparison: TCMR achieves statistically significant performance improvements across most evaluation metrics on MOSI.Figure 9 reports statistical significance using error bars and asterisk thresholds.
Loading 2609.10950v1…