Source-linked AI summary
Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis
Wenmeng Yu, Hua Xu, Ziqi Yuan, Jiele Wu
TL;DR
The paper targets the difficulty of learning both consistent and modality-specific representations from unified multimodal annotations without costly unimodal labels. It generates unimodal supervision self-supervisedly, jointly trains unimodal and multimodal tasks with adaptive weighting, and reports strong results across three datasets. Auto-generated labels are reliable and stable, while generated audio and vision labels remain limited by preprocessed features.
Problem
Unified multimodal annotations restrict learning differentiated modality information, while obtaining additional unimodal annotations is costly in time and labor.
Method
The method generates independent unimodal supervisions self-supervisedly, jointly trains unimodal and multimodal tasks, and adjusts subtask weights according to label differences.
Results
The method surpasses current state-of-the-art results on MOSI and MOSEI, achieves performance comparable with human-annotated unimodal labels on SIMS, and shows reliable auto-generated labels.
Takeaways & Limitations
Joint unimodal and multimodal learning provides a self-supervised route to learning consistent and differentiated modality representations without human unimodal annotations.
Takeaways & Limitations
Generated audio and vision labels are not significant enough because of limitations in the preprocessed features.
Abstract
from arXiv · showhide
Representation Learning is a significant and challenging task in multimodal learning. Effective modality representations should contain two parts of characteristics: the consistency and the difference. Due to the unified multimodal annotation, existing methods are restricted in capturing differentiated information. However, additional uni-modal annotations are high time- and labor-cost. In this paper, we design a label generation module based on the self-supervised learning strategy to acquire independent unimodal supervisions. Then, joint training the multi-modal and uni-modal tasks to learn the consistency and difference, respectively. Moreover, during the training stage, we design a weight-adjustment strategy to balance the learning progress among different subtasks. That is to guide the subtasks to focus on samples with a larger difference between modality supervisions. Last, we conduct extensive experiments on three public multimodal baseline datasets. The experimental results validate the reliability and stability of auto-generated unimodal supervisions. On MOSI and MOSEI datasets, our method surpasses the current state-of-the-art methods. On the SIMS dataset, our method achieves comparable performance than human-annotated unimodal labels. The full codes are available at https://github.com/thuiar/Self-MM.
1 Introduction
The paper addresses the difficulty of learning modality-specific information when multimodal annotations provide unified supervision. It proposes self-supervised unimodal labels and adaptive multi-task weighting to learn consistent and differentiated representations.
- Representation learning must capture both consistent and complementary information across modalities.
- Unified multimodal annotation makes it difficult for forward-guidance methods to capture modality-specific information.
- The method generates unimodal labels from multimodal labels and modality representations without human-annotated unimodal labels.
- A momentum-based update gives greater weight to unimodal labels generated later because early auto-generated labels are unstable.
- The self-adjustment strategy weights subtasks according to differences between auto-generated unimodal and human-annotated multimodal labels.
- Experiments on three benchmark datasets validate the stability and reliability of auto-generated unimodal labels, with performance exceeding current state-of-the-art results.
2 Related Work
Related work covers multimodal sentiment representation learning, Transformer-based language processing, and multi-task learning. The paper jointly learns unimodal and multimodal tasks so different tasks provide similarity and differentiated information.
- Multimodal Sentiment Analysis: Multimodal sentiment analysis combines verbal, visual, and acoustic information, with prior work focusing on representation learning and multimodal fusion.
- Multimodal Sentiment Analysis: The paper uses late fusion and jointly learns unimodal and multimodal tasks with a self-supervised strategy.
- Multimodal Sentiment Analysis: The multimodal task learns similarity information, while unimodal tasks learn differentiated information.
- Transformer and BERT: The paper fine-tunes pretrained BERT as a language feature extraction module for its tasks.
- Multi-Task Learning: Multi-task learning faces challenges in parameter sharing and balancing the learning processes of different tasks.
- Multi-Task Learning: The method adopts hard parameter sharing and a weight-adjustment strategy to balance unimodal subtasks.
3 Methodology
Self-MM jointly learns one multimodal task and three unimodal subtasks to obtain information-rich modality representations. Its self-supervised ULGM generates unimodal supervision from multimodal labels and modality representations, while adaptive weighting emphasizes samples with larger supervision differences.
- Overall architecture: Self-MM jointly trains one multimodal task and three unimodal subtasks, sharing the bottom representation-learning network.The unimodal subtasks exist only during training, and the multimodal prediction is used as the final output.
- Feature representation: The model processes text with 12-layer BERT and audio and vision with pretrained toolkit features followed by single-directional LSTMs.The first-word vector from BERT represents the sentence, while end-state hidden vectors represent audio and vision sequences.
- Task architecture: The three unimodal tasks share modality representations with the multimodal task, project them into a common feature space, and predict unimodal results with linear regression.The modality representations are concatenated and projected into a lower-dimensional fusion space for multimodal prediction.
- Unimodal label generation: ULGM generates unimodal supervision from multimodal annotations and modality representations using relative distances to positive and negative class centers.It is non-parameterized and computes modality-specific offsets from representation geometry; equal-weight summation combines the resulting relationships.
- Unimodal label generation: A momentum-based update policy stabilizes dynamically generated unimodal labels by weighting later labels more heavily across training epochs.The labels reportedly stabilize after about 20 iterations, after which unimodal-task training gradually becomes stable.
- Optimization objectives: The unimodal-task loss uses the difference between unimodal and multimodal labels as a sample weight, directing learning toward samples with larger differences.The optimization uses L1Loss as the basic objective.
4 Experimental Settings
The experiments evaluate Self-MM on three public multimodal sentiment-analysis datasets against established baselines and related methods, using standard classification and regression metrics.
- Datasets: Self-MM is evaluated on MOSI, MOSEI, and SIMS, three public multimodal sentiment-analysis datasets.MOSI contains 2,199 clips, MOSEI contains 23,453 utterances, and SIMS contains 2,281 refined clips.
- Baselines: The comparison includes established multimodal sentiment-analysis baselines and state-of-the-art methods, including TFN, LMF, and MISA.The cited baselines cover tensor fusion, low-rank fusion, memory fusion, factorization, attended variation embedding, multimodal transformers, and modality-specific representation learning.
- Evaluation Comparisons: Table 3 reports results on SIMS, where Self-MM is compared with TFN, LMF, and Human-MM using human-annotated unimodal labels.Human-MM replaces auto-generated unimodal labels with human-annotated unimodal labels.
- Evaluation Metrics: The experiments report Weighted F1 score and binary accuracy for classification, plus Mean Absolute Error and Pearson correlation for regression.Except for MAE, higher values denote better performance.
- Ablation Settings: Table 4 evaluates Self-MM with different combinations of multimodal, text, audio, and vision tasks.The table labels these tasks as M, T, A, and V, respectively.
5 Results and Analysis
Self-MM improves multimodal sentiment results across MOSI, MOSEI, and SIMS, while its auto-generated unimodal labels stabilize and reflect modality-specific sentiment differences.
- Quantitative Results: Self-MM surpasses reproduced MISA and MAG-BERT on most evaluations and achieves significant gains over unaligned TFN and LMF on MOSI and MOSEI.It remains competitive with aligned models under the same comparison.
- Quantitative Results: Self-MM outperforms TFN and LMF on SIMS and achieves comparable performance with Human-MM using human-annotated unimodal labels.These results cover an unaligned-data setting and indicate applicability across different data scenarios.
- Ablation Study: Adding unimodal subtasks significantly improves performance over the single-task model, with text and audio helping more than vision.The M, T, V and M, T, A combinations are comparable to or better than the full M, T, A, V combination.
- Case Study: As iterations increase, auto-generated unimodal-label distributions stabilize, with faster convergence on MOSEI than on MOSI and SIMS.The authors associate faster convergence on the larger dataset with more stable class centers.
- Case Study: MOSI case studies show unimodal labels receiving negative offsets when individual modalities lean negative despite positive multimodal labels, and a positive offset in another case.The figure distinguishes human-annotated M labels from auto-generated V, T, and A labels.
6 Conclusion and Future Work
The paper introduces self-supervised unimodal subtasks to learn modality-specific representations without human unimodal labels. It reports stable, reliable auto-generated labels, while identifying limitations in generated audio and vision labels caused by pre-processed features.
- Conclusion: Self-MM jointly introduces unimodal subtasks and a self-supervised label-generation strategy to learn modality-specific representations while reducing human annotation costs.The authors describe this as a new perspective on multimodal representation learning.
- Future Work: Generated audio and vision labels are not significant enough because of limitations in the pre-processed features.Future work will explore an end-to-end multimodal learning network and the relationship between unimodal and multimodal learning.