Source-linked AI summary
Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanqiuyue Yang, Yuanzhe Qiu, Tie Cheng, Xiaoteng Li, Hua Xu, Kai Gao
TL;DR
MSA research seeks to use acoustic and visual cues alongside text, but prior work often relies more on text and does not fully use non-verbal behaviors. This paper presents the CH-SIMS v2.0 dataset and AV-MC, which uses mixup to simulate unobserved non-verbal contexts. The reported experiments and annotations demonstrate the significance of making full use of non-verbal behaviors.
Problem
Prior MSA approaches over-rely on text and fail to make full use of non-verbal behaviors.
Source: §1 · p. 2Method
The paper extends CH-SIMS as CH-SIMS v2.0 and proposes AV-MC, which uses mixup to simulate potential unobserved non-verbal behaviors corresponding to the same spoken words.
Source: §1 · pp. 2–3Results
The paper reports that its experiments and annotations demonstrate the significance of making full use of non-verbal behaviors.
Source: §1 · p. 3, §6 · p. 14Takeaways & Limitations
The conclusion frames the work as an effort to improve the contribution of non-verbal cues to sentiment analysis.
Source: §6 · p. 14The authors report that AV-MC performs poorly in sentiment inference due to the limitation of its simple late-fusion strategy.
Source: §5.3 · p. 12
Abstract
from arXiv · showhide
Multimodal sentiment analysis (MSA), which supposes to improve text-based sentiment analysis with associated acoustic and visual modalities, is an emerging research area due to its potential applications in Human-Computer Interaction (HCI). However, the existing researches observe that the acoustic and visual modalities contribute much less than the textual modality, termed as text-predominant. Under such circumstances, in this work, we emphasize making non-verbal cues matter for the MSA task. Firstly, from the resource perspective, we present the CH-SIMS v2.0 dataset, an extension and enhancement of the CH-SIMS. Compared with the original dataset, the CH-SIMS v2.0 doubles its size with another 2121 refined video segments with both unimodal and multimodal annotations and collects 10161 unlabelled raw video segments with rich acoustic and visual emotion-bearing context to highlight non-verbal cues for sentiment prediction. Secondly, from the model perspective, benefiting from the unimodal annotations and the unsupervised data in the CH-SIMS v2.0, the Acoustic Visual Mixup Consistent (AV-MC) framework is proposed. The designed modality mixup module can be regarded as an augmentation, which mixes the acoustic and visual modalities from different videos. Through drawing unobserved multimodal context along with the text, the model can learn to be aware of different non-verbal contexts for sentiment prediction. Our evaluations demonstrate that both CH-SIMS v2.0 and AV-MC framework enables further research for discovering emotion-bearing acoustic and visual cues and paves the path to interpretable end-to-end HCI applications for real-world scenarios.
1 INTRODUCTION
The introduction identifies text-predominant MSA and limitations in resources and non-verbal representation learning as barriers to using acoustic and visual cues. It presents CH-SIMS v2.0 and AV-MC as resource and modeling contributions, with experiments intended to show the significance of using non-verbal behaviors.
Motivation and challenges
- MSA is described as text-predominant: prior reports show about a 30% binary-accuracy drop on MOSI when text is removed, from 80%+ with text to about 54% without it.The introduction links reliance on text to imperfect textual input, including potential ASR errors, and notes that textual ambiguity and irony pose challenges.Source: §1 · p. 2
Contributions
- The paper presents CH-SIMS v2.0, an extension of CH-SIMS with 4,402 supervised instances and over 10,000 unsupervised data, collected across 11 scenarios.The supervised data includes unimodal annotations; the scenarios are intended to simulate real-world HCI settings.Source: §1 · p. 2
- AV-MC uses mixup to simulate potential unobserved non-verbal behaviors corresponding to the same spoken words, and adapts to supervised and semi-supervised training.The framework uses the dataset’s unimodal annotations to make models aware of varied non-verbal behaviors.Source: §1 · pp. 2–3
- The introduction reports that experiments provide an intuitive demonstration of the significance of making full use of non-verbal behaviors.The paper describes these experiments as baselines and an intuitive demonstration.Source: §1 · p. 3
2 RELATED WORKS
The related work reviews multimodal sentiment datasets, prior MSA representation and fusion methods, and Mixup approaches. It identifies limited diversity in CH-SIMS and limited use of non-verbal behaviors as motivations for extending the dataset and applying input-level Mixup to MSA.
2.1 Multimodal Sentiment Analysis Dataset
- Time-consuming unimodal annotation limits CH-SIMS dataset size and non-verbal diversity; CH-SIMS v2.0 adds unlabeled instances with more expressive non-verbal behaviors.The added data enriches multimodal context with few annotation workloads.Source: §2.1 · p. 3
2.2 Multimodal Sentiment Analysis
- Earlier MSA work emphasizes joint representation learning and multimodal fusion, while the paper says these approaches fail to make full use of non-verbal behaviors.The reviewed methods include cross-modal attention, tensor fusion, and modality-specific or modality-invariant representations.Source: §2.2 · p. 3
2.3 Mixup
- The paper transfers input-level Mixup to MSA to enhance acoustic and visual representation learning on a multitask late-fusion backbone.The review distinguishes input-level from hidden-level Mixup and notes prior use in semi-supervised learning and text classification.Source: §2.3 · pp. 3–4
3 CH-SIMS V2.0 DATASET
CH-SIMS v2.0 expands scenario diversity and emphasizes emotion-bearing non-verbal behavior, with supervised and unlabeled video segments. Its modality-isolated annotations show differences from multimodal labels, and the paper specifies default text, acoustic, and visual features.
3.1 Data Collection
- CH-SIMS v2.0 adds diverse video scenarios and deliberately includes expressive, emotion-bearing non-verbal behaviors.The collected scenes include melodrama, interviews, modern TV, talk shows, vlogs, films, costume TV, and variety shows.Source: §3.1 · pp. 4–5
- The dataset contains 4,402 supervised segments and 10,161 unsupervised segments; original CH-SIMS instances are relabeled within the supervised set.The unsupervised segments have a more diverse duration distribution, while their transcripts are unrefined and may contain noise.Source: §3.1 · p. 5, Figure 2 · p. 5
3.2 Data Annotation
- Unimodal labels are collected under modality isolation: acoustic clips obscure words, visual clips are silent without transcripts, and text labels use transcripts alone.Annotation proceeds in the order text, acoustic, visual, then multimodal.Source: §3.2 · p. 6
- The annotation comparison reports an 83.41% positive-to-negative instance ratio and significant differences between unimodal and multimodal labels.Acoustic annotations are most similar to multimodal annotations, while textual annotations are least similar.Source: §3.2 · p. 6, Figure 3 · p. 6
3.3 Feature Extraction
- Default features comprise 768-dimensional BERT contextual word-vector sequences, 25-dimensional eGeMAPS acoustic low-level descriptors extracted with OpenSMILE at 16000 Hz, and 177-dimensional frame-level visual features.Acoustic features are padded or truncated to a sequence length of 925.Source: §3.3 · pp. 6–7
4 ACOUSTIC VISUAL MIXUP CONSISTENT (AV-MC) FRAMEWORK
AV-MC is designed to prepare MSA models for unobserved multimodal context by mixing acoustic and visual representations and training with supervised and unsupervised data.
4.1 Modality Mixup Module
- AV-MC mixes shuffled and original modality instances through weighted averaging to create potential representations for learning from unobserved multimodal context.The mixing weight is sampled from a Beta distribution within [0, 1].Source: §4 · p. 7, §4.1 · pp. 7–8
4.2 Supervised and Semi-supervised learning with AV-MC Framework
- Text uses a one-layer feed-forward encoder on [CLS], while acoustic and visual sequences use stacked bidirectional LSTMs followed by one-layer feed-forward networks.Source: §4.2 · p. 8
- AV-MC concatenates unimodal representations into a multimodal representation and uses independent classifiers for unimodal and multimodal sentiment prediction.Source: §4.2 · p. 8
- The regression loss weights unimodal and multimodal tasks, while a separate weighted consistency loss combines acoustic and visual mixup tasks.Source: §4.2 · pp. 8–9
- For supervised and unsupervised data, acoustic and visual mixed representations use the corresponding unimodal classifiers, with predictions trained to match direct predictions.Source: §4.2 · pp. 8–9
- Each training epoch first updates parameters using supervised and unsupervised instances with regression and mixup-consistency losses, then uses supervised instances with regression loss alone.Source: §4.2 · p. 9
5 EXPERIMENTS AND DISCUSSION
The experiments and discussion evaluate AV-MC on CH-SIMS v2.0 benchmarks and analyze its performance through case studies and ablations. Findings highlight gains in sentiment prediction, especially for weak-emotion instances, alongside continuing challenges in fine-grained prediction and multimodal fusion.
5.1 Dataset splits and Evaluation Metric
- CH-SIMS v2.0 is partitioned into training, validation, and test sets in a ratio close to 9:2:3; performances are evaluated using classification and regression, with Acc2, F1_Score, Acc2_weak, MAE, and Corr.Source: §5.1 · p. 9
5.2.1 Baselines.
- The benchmark models fall into two groups: those trained with unified multimodal annotations and those using unimodal annotations to guide unimodal representation learning.The latter group includes MTFN, MLF_DNN, and MLMF.Source: §5.2.1 · p. 10
5.2.2 Benchmark Model Performances.
- Benchmarks perform relatively poorly on Acc2_weak and regression metrics; unimodal annotations help intensity prediction, and simple late fusion can be competitive with state-of-the-art models.The performance gap between unimodal and multimodal annotations also suggests that multimodal annotations can mislead unimodal representation learning.Source: §5.2.2 · pp. 10–11, Table 3 · p. 11
- AV-MC outperforms existing MSA benchmarks, especially on Acc2_weak under supervised settings, and improves further with CH-SIMS v2.0 (u).The authors attribute the weak-emotion result to modality mixup and the further improvement to the potential of unsupervised data.Source: Table 2 · p. 11, §5.2.2 · p. 11
5.3 Case Study
- AV-MC performs well when more than two modality annotations are consistent with ground-truth multimodal annotations; in case #I-2, despite no obvious textual emotion, its prediction has less than 0.1 difference using acoustic and visual cues.The authors note a limitation of the simple late fusion strategy.Source: §5.3 · p. 12
5.4 Ablation Study
- On CH-SIMS v2.0, removing both acoustic-visual mixup and unimodal multitask supervision from the AV-MC multitask late-fusion backbone causes performance to decline by 4.80% on Acc2, 5.84% on Acc2_weak, and 11.08% on MAE.Source: Table 4 · p. 13, §5.4 · p. 13
- Ablations and constructed scenarios indicate that acoustic and visual mixup improve sentiment-intensity prediction by helping the model perceive non-verbal context, with visual mixup especially contributive.Mixup contributes more to instances #AV-2, #AV-5, and #AV-6, which might be unseen during training; the findings support alleviating text predominance.Source: §5.4 · pp. 13–14
- Compared with (a), (b) shows reduced overlap between modality representations, indicating that unimodal annotations help the multitask late-fusion backbone preserve modality-specific information.Source: §5.4 · p. 14
- Compared with (b), (c) shows greater within-modality separation, indicating that acoustic-visual mixup embeds virtual non-verbal contexts in latent space and enhances fine-grained sentiment perception.The mixup strategy provides more potential non-verbal contexts.Source: §5.4 · p. 14
6 CONCLUSION
The paper introduces CH-SIMS v2.0, a semi-supervised dataset with fine-grained unimodal and multimodal annotations, and proposes AV-MC, which mixes acoustic and visual modalities. The authors identify annotation discrepancies as evidence of emotion-bearing nonverbal cues and outline future research directions.
6 CONCLUSION
- CH-SIMS v2.0 is a semi-supervised dataset built from videos with expressive nonverbal behaviors and fine-grained unimodal and multimodal annotations.The authors describe it as an enhancement and extension of CH-SIMS.Source: §6 · p. 14
- The discrepancy between textual and multimodal annotations intuitively reveals the importance of emotion-bearing acoustic and visual cues for prediction.Source: §6 · p. 14
- AV-MC creates augmentations by mixing acoustic and visual modalities, enriching nonverbal behaviors so models can learn their contributions.The authors describe AV-MC as an early attempt based on CH-SIMS v2.0.Source: §6 · p. 14
- The authors expect CH-SIMS v2.0 to guide research toward real multimodal fusion approaches that alleviate text-predominance.Source: §6 · p. 14
- Future research can explore interpretable approaches for discovering decisive modalities and multimodal pretrained approaches using unimodal annotations and unsupervised data.Source: §6 · p. 14