Source-linked AI summary
A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, Bo Xu
TL;DR
Multimodal ERC requires modeling interactions across utterances and modalities rather than relying mainly on text. SDT combines transformer-based interaction modeling, hierarchical gated fusion, and self-distillation, and it outperforms prior baselines on IEMOCAP and MELD.
Problem
Existing ERC studies mainly model textual context and speaker dependencies, leaving multimodal interactions, modality contributions, and modal representations insufficiently addressed.
Method
SDT uses intra- and inter-modal transformers, hierarchical gated fusion, and self-distillation of hard and soft labels to improve modality representations.
Results
SDT outperforms all baselines on IEMOCAP and MELD, including MMTr by 1.68% accuracy and 2.17% weighted F1-score on IEMOCAP, and UniMSE by 2.46% accuracy and 1.09% weighted F1-score on MELD.
Takeaways & Limitations
The results demonstrate the effectiveness of combining multimodal interaction modeling, adaptive fusion, and self-distillation for multimodal ERC.
Takeaways & Limitations
SDT misclassifies similar emotions and minority emotions, and often predicts MELD’s majority neutral class.
Abstract
from arXiv · showhide
Emotion recognition in conversations (ERC), the task of recognizing the emotion of each utterance in a conversation, is crucial for building empathetic machines. Existing studies focus mainly on capturing context- and speaker-sensitive dependencies on the textual modality but ignore the significance of multimodal information. Different from emotion recognition in textual conversations, capturing intra- and inter-modal interactions between utterances, learning weights between different modalities, and enhancing modal representations play important roles in multimodal ERC. In this paper, we propose a transformer-based model with self-distillation (SDT) for the task. The transformer-based model captures intra- and inter-modal interactions by utilizing intra- and inter-modal transformers, and learns weights between modalities dynamically by designing a hierarchical gated fusion strategy. Furthermore, to learn more expressive modal representations, we treat soft labels of the proposed model as extra training supervision. Specifically, we introduce self-distillation to transfer knowledge of hard and soft labels from the proposed model to each modality. Experiments on IEMOCAP and MELD datasets demonstrate that SDT outperforms previous state-of-the-art baselines.
I. INTRODUCTION
Multimodal ERC must model context- and speaker-sensitive dependencies while integrating textual, acoustic, and visual evidence. SDT addresses this through cross-modal interaction modeling, adaptive fusion, and self-distillation.
- ERC recognizes each utterance’s emotion in context, supporting applications such as opinion mining, health care, and empathetic dialogue systems.
- Most existing ERC methods focus on textual conversations and do not leverage acoustic or visual modalities, despite emotions being expressed verbally, vocally, and facially.
- Multimodal ERC requires intra- and inter-modal interaction modeling, dynamic weighting of modalities, and expressive modal representations.
- SDT uses intra- and inter-modal transformers, hierarchical gated fusion, and self-distillation to address these multimodal ERC requirements.Self-distillation transfers hard- and soft-label knowledge from the proposed model to modality-specific students.
- The paper evaluates positional and speaker embeddings, transformer components, self-distillation losses, and hierarchical gated fusion through experiments and ablations.
C. Knowledge Distillation
The paper situates SDT within knowledge distillation and defines its multimodal ERC pipeline. Self-distillation uses the model’s own outputs to improve modality-specific representations without an additional teacher network.
- Knowledge distillation transfers knowledge from a large teacher network to a small student network through outputs, features, or inter-layer relationships.
- Self-distillation is an online-distillation variant in which a single network teaches another part of itself, avoiding an extra network.
- SDT uses output-based soft-label supervision because modality features differ substantially and matching fused multimodal features with unimodal features is inappropriate.
- The ERC pipeline predicts each utterance’s emotion from textual, acoustic, and visual modality sequences in a conversation.
- SDT combines a modality encoder, hierarchical gated fusion, emotion classifier, and self-distillation with two loss types for learning better modal representations.
Positional Embeddings:
SDT augments modality sequences with positional and speaker information before transformer-based interaction modeling. Intra-modal transformers refine each modality, while inter-modal transformers transfer information across modalities.
- Positional Embeddings:: Positional embeddings augment convolved modality sequences with utterance-order information.The position index and dimension index determine the embedding components.
- Positional Embeddings:: Speaker embeddings augment modality sequences with speaker-specific information represented by trainable vectors.Each speaker is mapped from a one-hot identity vector through a trainable embedding matrix.
- Positional Embeddings:: The resulting representation combines modality features with positional and speaker embeddings to encode context and speaker information.
- Intra- and Inter-modal Transformers:: Intra-modal transformers use one modality’s sequence as queries, keys, and values to capture interactions among its utterances.
- Intra- and Inter-modal Transformers:: Inter-modal transformers use one modality as queries and another as keys and values, enabling information transfer between modalities.
D. Hierarchical Gated Fusion
Hierarchical gated fusion first filters cross-modal representations and enhances each modality, then learns utterance-specific weights to combine the enhanced modalities into a multimodal representation.
- The hierarchical gated fusion module contains unimodal- and multimodal-level gates for enhancing modality representations and learning their relative weights.
- Unimodal-level Gated Fusion:: A sigmoid gate filters irrelevant information from each cross-modal representation before further modality enhancement.The gate is computed from a learned weight transformation and applied element-wise.
- Unimodal-level Gated Fusion:: Enhanced representations from the other two modalities are combined through a fully connected layer to obtain each modality’s enhanced sequence representation.
- Multimodal-level Gated Fusion:: A softmax gate dynamically learns the textual, acoustic, and visual weights for each utterance.
- Multimodal-level Gated Fusion:: The weighted enhanced modality representations form the final multimodal sequence used by the emotion classifier.
E. Emotion Classifier
The emotion classifier produces utterance-level emotion probabilities, while self-distillation trains modality-specific students using supervision from the multimodal teacher. The students learn from both hard labels and softened teacher outputs.
- Emotion prediction: The classifier converts fused representations into probability distributions over emotion categories and selects each utterance’s highest-probability label.The classifier uses fully connected and softmax layers; predictions are made with argmax.
- Self-distillation: Self-distillation transfers knowledge from the proposed multimodal model to textual, acoustic, and visual modality students.Each student includes a modality encoder, unimodal gated fusion, and classifier, and is trained during model training.
- Self-distillation: The student outputs emotion probabilities for each utterance, with temperature τ used to soften the distributions.A higher τ produces a softer distribution over classes.
- Self-distillation: Students are trained to learn enhanced modality representations for m ∈ {t, a, v}.The training objective uses two kinds of losses for each modality-specific student.
Cross Entropy Loss:
The training procedure combines hard-label and soft-label supervision, while evaluation uses conversation datasets with different sizes and emotion inventories. The overall objective weights the three loss components equally in the reported experiments.
- Cross Entropy Loss: Cross-entropy supervision trains each modality student against the ground-truth emotion labels.This directly introduces hard-label knowledge into the student to improve its modality representation.
- KL Divergence Loss: KL-divergence supervision makes each student’s softened output approximate the teacher’s softened output.This transfers soft-label knowledge to the student.
- Datasets: The evaluation uses IEMOCAP and MELD, which differ in conversation structure, dataset size, and emotion categories.IEMOCAP contains 153 conversations and 7,433 utterances with six emotions, whereas MELD contains 1,433 conversations and 13,708 utterances with seven emotions.
- Overall loss: The three loss types are equally weighted in the reported experiments, with γ1 = γ2 = γ3 = 1.The coefficients control the weights of the three kinds of losses.
Evaluation Metrics:
The study evaluates multimodal ERC with overall accuracy and weighted F1-score, alongside emotion-specific metrics, and compares SDT with sequence-, graph-, and fusion-based baselines. Baseline implementations are aligned with the authors’ extracted features where possible.
- Evaluation Metrics: Overall accuracy and weighted average F1-score measure overall performance, while per-emotion accuracy and F1-score provide class-level results.These metrics are reported for the overall task and for each emotion class.
- Baselines: Compared baselines include recurrent, graph-based, hierarchical-transformer, dynamic-fusion, cross-modal, and contrastive multimodal models.Examples include DialogueRNN, MMGCN, DialogueTRM, MM-DFN, MMTr, and UniMSE.
- Baselines: The baseline systems generally combine textual, acoustic, and visual features into multimodal utterance representations.The cited baselines differ in how they model contextual dependencies and fuse modalities.
- Implementation: All baselines are re-run except MMTr and UniMSE because their source code was unavailable.The authors also explain why their extracted features could not be used to implement UniMSE.
DialogueRNN*
SDT achieves the strongest reported overall performance on both IEMOCAP and MELD, and ablations indicate that its transformer components and combined self-distillation losses contribute to performance. The evaluation uses common data splits and reports comparisons against reimplemented baselines.
- Experimental setup: The experiments use the same data splits for all implemented models, including reimplemented baselines with the authors’ extracted features.The reported comparison includes reimplemented DialogueRNN, MMGCN, DialogueTRM, and MM-DFN variants.
- Overall Results: 1.68% accuracy and 2.17% weighted F1-score are SDT’s improvements over MMTr on IEMOCAP.SDT performs better than all baselines on IEMOCAP and improves most emotion-class F1-scores.
- Ablation on Transformer-based Model: Removing any positional embedding, speaker embedding, intra-modal transformer, or inter-modal transformer degrades performance.The ablation study identifies all four components as useful, with positional and speaker embeddings having considerable effects on both datasets.
- Ablation on Self-distillation: The model performs best when both cross-entropy and KL-divergence self-distillation losses are included.The two losses are described as complementary, supporting transfer from both hard and soft labels.
Effect of Different Modalities:
Textual features lead unimodal emotion recognition, while combining modalities improves over unimodal inputs. SDT’s hierarchical gated fusion outperforms alternative fusion strategies and produces more separated multimodal representations with self-distillation.
- Modal Contributions: Textual modality performs far better than acoustic or visual modalities alone, indicating its leading role in emotion recognition.
- Modal Contributions: Bimodal combinations outperform their corresponding unimodal results, showing that acoustic and visual information complement textual features.
- Fusion Strategies: Hierarchical gated fusion significantly outperforms Add, Concatenation, and Uni-Cat-Transformer fusion strategies.The fusion module filters irrelevant unimodal information before dynamically weighting modalities.
- Training Losses: All training losses converge on IEMOCAP, although the visual student has larger losses than the textual and acoustic students.The authors attribute the visual student’s larger losses potentially to an unsuitable learning rate and propose adaptive modality-specific learning rates for future work.
- Representation Visualization: Self-distillation yields better separation of emotion representations, reducing mixing among similar and neutral emotions.The model’s representations become more clustered even without self-distillation, while self-distillation further separates emotion categories.
- Representation Visualization: With or without self-distillation, the model forms clusters corresponding to speaker gender.
E. Case Study
The case studies examine SDT’s multimodal predictions, attention patterns, emotion confusions, and performance under emotional shifts. They show successful examples alongside errors concentrated around similar or imbalanced emotions and emotional transitions.
- Case Study: SDT correctly identifies all utterance emotions in the MELD case study, whereas DialogueRNN* and MMGCN* misclassify the third utterance.The competing models predict “surprise” instead of the correct emotion, possibly because question marks often express surprise.
- Case Study: Text-only attention for the fourth utterance relies heavily on the third and fifth utterances, while SDT correctly recognizes it as “disgust.”The third utterance’s neutral emotion may dominate the text-only prediction through darker attention heads.
- Error Analysis: SDT confuses similar emotions on IEMOCAP and MELD, including happy–excited, angry–frustrated, and surprise–anger pairs.The confusion matrices also show a tendency to predict “neutral” for other MELD emotions.
- Error Analysis: Recognizing minority emotions such as fear and disgust remains difficult on MELD because of unbalanced data.The authors identify similar emotions and imbalanced emotion distributions as continuing ERC challenges.
- Error Analysis: SDT performs worse on utterances with emotional shifts than on utterances without them.The paper relates this difficulty to latent factors such as speaker personality and intent, which SDT and most existing models do not consider.
APPENDIX ATTENTION VISUALIZATION
The appendix visualizes multi-head attention weights used to form enhanced acoustic and visual modality representations in SDT.
- Attention Visualization: Multi-head attention weights are visualized for the transformers that form enhanced acoustic and visual modality representations.The visualization concerns the attention mechanisms inside SDT’s modality-specific processing.