Source-linked AI summary
Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation
Xiaohui Song, Longtao Huang, Hui Xue, Songlin Hu
TL;DR
ERC must recognize emotions that vary with conversational context, while supervised contrastive learning can struggle with class imbalance. This paper combines a prototypical contrastive loss with distance-based curriculum learning and reports state-of-the-art performance on three benchmarks.
Problem
Supervised contrastive learning can struggle with class imbalance because majority-class samples may lack negatives and minority-class samples may lack positives within a batch.
Source: §3.3 · p. 4Method
The paper combines a supervised prototypical contrastive loss with a distance-based difficulty measure that schedules training examples from easy to hard.
Source: §1 · p. 2Results
The approach achieves state-of-the-art performance on IEMOCAP, MELD, and EmoryNLP.
Source: §6 · pp. 8–9Takeaways & Limitations
SPCL works well on class-imbalanced data and is less sensitive to training batch size, reducing computing-resource requirements.
Source: §6 · p. 8The approach uses many hyperparameters that require additional computing resources to search.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Capturing emotions within a conversation plays an essential role in modern dialogue systems. However, the weak correlation between emotions and semantics brings many challenges to emotion recognition in conversation (ERC). Even semantically similar utterances, the emotion may vary drastically depending on contexts or speakers. In this paper, we propose a Supervised Prototypical Contrastive Learning (SPCL) loss for the ERC task. Leveraging the Prototypical Network, the SPCL targets at solving the imbalanced classification problem through contrastive learning and does not require a large batch size. Meanwhile, we design a difficulty measure function based on the distance between classes and introduce curriculum learning to alleviate the impact of extreme samples. We achieve state-of-the-art results on three widely used benchmarks. Further, we conduct analytical experiments to demonstrate the effectiveness of our proposed SPCL and curriculum learning strategy. We release the code at https://github.com/caskcsg/SPCL.
1 Introduction
Emotion recognition in conversation must account for context because similar utterances can convey different emotions, while class imbalance and text-insufficient examples challenge contrastive learning. The paper combines SPCL with distance-based curriculum learning and reports state-of-the-art results on three benchmarks.
1 Introduction
- ERC requires context because emotions can vary across conversation turns, and even the same utterance can convey different emotions in different contexts.Textual information may also be insufficient to distinguish emotions in some conversations, making extreme samples a potential source of performance degradation.Source: §1 · p. 1, Figure 1 · p. 1
- A distance-based difficulty measure sorts training examples from easy to hard as a curriculum strategy for alleviating the impact of extreme samples.The paper reports experimental evidence for the strategy's effectiveness.Source: §1 · p. 2
- The combined SPCL loss and curriculum learning strategy achieves state-of-the-art results on three widely used benchmarks.The paper also reports that experiments demonstrate the effectiveness of both components.Source: §1 · p. 2
- SPCL combines prototypical networks with supervised contrastive learning to support imbalanced ERC without requiring large batches.It samples representations from per-category queues to form temporary prototypes, ensuring positive and negative class examples during training.Source: §1 · p. 2
- The authors present combining supervised contrastive learning and curriculum learning for ERC as a contribution they believe is a first.They explicitly qualify the novelty claim with “to the best of our knowledge.”Source: §1 · p. 2
2 Related Work
Related ERC work is organized around context modeling, while contrastive-learning work includes approaches for sentence embeddings, supervised learning, smaller-batch learning, and emotion discrimination.
2.1 Emotion Recognition in Conversation
- ERC approaches are grouped as sequence-based, graph-based, or knowledge-enhanced, with methods modeling utterance sequences, graph structures, or external knowledge.The discussion also notes hybrid curriculum and speaker-guided encoder-decoder frameworks.Source: §2.1 · pp. 2–3
2.2 Contrastive Learning
- Contrastive-learning approaches include SimCSE for sentence embeddings, supervised extensions of batch contrastive learning, methods reducing large-batch dependence, and CoG-BART for distinguishing emotions.The cited examples are SimCSE (Gao et al., 2021), Khosla et al. (2020), Yeh et al. (2021), and CoG-BART (Li et al., 2021).Source: §2.2 · p. 3
3 Methodology
The method defines real-time emotion recognition from prior conversation turns and builds speaker- and context-aware representations with a prompt-based encoder. It combines prototype-based supervised contrastive learning with a distance-based curriculum, then trains the encoder on progressively harder subsets.
3.1 Definition
- In real-time ERC, the model predicts the emotion at turn t using only conversation turns through t.Source: §3.1 · p. 3
3.2 Context Modeling
- The prompt-based encoder represents a target utterance using the most recent k utterances and speakers as context.Its full input concatenates the context with a prompt; training also uses a pair sharing the context but having a different prompt and label.Source: §3.2 · p. 3
3.3 Supervised Prototypical Contrastive Learning for ERC
- SPCL adds category prototypes to supervised contrastive learning, providing each sample at least one positive and |E|−1 negative pairs.Prototypes are means of randomly selected support representations drawn from per-category queues.Source: §3.3 · p. 4
3.4 Curriculum Learning
- The curriculum scores sample difficulty using distances between emotion representations and class centers, then samples training subsets that shift from mostly easy to mostly hard.The sampling-based schedule is described as providing smoother difficulty variation than sequentially splitting the training set.Source: §3.4 · pp. 4–5
3.5 Training and Evaluation
- Training computes representations and difficulty scores, sorts the training set, samples a subset, and optimizes the context encoder with SPCL.The paper also describes a separate linear layer trained with cross-entropy for label prediction in other cases.Source: §3.5 · p. 5
4 Experimental Settings
Experiments evaluate the approach on three ERC benchmarks, using weighted-F1 to account for class imbalance. The setup uses SimCSE-based experiments with fixed test reporting from the best development-set checkpoint.
4.1 Experimental Setup
- Experiments use AdamW and a cosine learning-rate schedule, cap training-sample length at 256, and report test results from the checkpoint selected on the development set.The experiments were conducted on Nvidia V100 GPUs.Source: §4.1 · p. 6
4.2 Datasets
- The evaluation covers MELD, EmoryNLP, and IEMOCAP, which contain dialogue utterances labeled with emotion categories.MELD and EmoryNLP each use seven emotion labels; IEMOCAP uses nine.Source: §4.2 · p. 6
4.3 Metrics
- Because all three benchmarks are class-imbalanced, the authors use weighted-F1 for every experiment.Source: §4.3 · p. 6
5 Results and Analysis
The approach achieves state-of-the-art results on three benchmarks, and ablations favor SPCL and curriculum learning. Further experiments examine class imbalance, batch-size sensitivity, and the curriculum's sample ordering.
5.1 Main Results
- 0.28% on IEMOCAP, 0.73% on MELD, and 0.7% on EmoryNLP are the reported margins over the previous state-of-the-art methods.The comparisons are CoMPM on IEMOCAP, CoMPM on MELD, and SGED + DAG-ERC on EmoryNLP, respectively.Source: §5.1 · pp. 6–7
5.2 Ablation Study
- SPCL outperforms SupCon and cross-entropy on all three datasets, while curriculum learning also consistently improves performance with SupCon and SPCL.Curriculum learning did not help much with cross-entropy; the authors attribute this to cosine distance being unsuitable for representations optimized with that loss.Source: §5.2 · p. 7
5.3 Using SPCL on Class Imbalanced Data
- On the imbalanced MELD training set, SPCL beats SupCon at batch sizes 4, 8, 16, and 32; from batch size 32 to 4, weighted-F1 drops 4.1% with SPCL versus 6.95% with SupCon.The constructed set uses 1,024 neutral samples and 128 joy samples, with 32 samples for each of surprise, anger, sadness, disgust, and fear.Source: §5.3 · p. 7, Figure 4 · p. 7, Table 4 · p. 7
5.4 Using SPCL with Small Batch Size
- On IEMOCAP, reducing batch size from 32 to 4 drops performance 0.82% with SPCL and 5.63% with SupCon, showing SPCL is less sensitive to batch size when each category has enough samples.Source: §5.4 · p. 8, Table 5 · p. 8
5.5 Qualitative Analysis of Curriculum Learning
- The difficulty measure ranks central samples as easiest and boundary samples as hardest, while sampling shifts from mostly easy examples early to mostly hard examples by the last epoch.Directly sorting by difficulty produced unsatisfactory results, so the authors use a sampling-based curriculum to provide smoother difficulty variation.Source: §5.5 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
6 Conclusion
The proposed SPCL loss and distance-based curriculum learning strategy are presented as effective for emotion recognition in conversation. The approach achieves state-of-the-art performance on all three evaluated datasets.
6 Conclusion
- State-of-the-art performance is reported on all three datasets.The conclusion names IEMOCAP, MELD, and EmoryNLP.Source: §6 · pp. 8–9
- SPCL combines Prototypical Network methods with supervised contrastive learning and is reported to outperform traditional supervised contrastive loss.It is also described as less sensitive to training batch size and effective on class-imbalanced data.Source: §6 · p. 8
- A distance-based difficulty measure and curriculum learning are introduced to alleviate the impact of extreme samples.Source: §6 · p. 8
Limitations
The work identifies limitations in hyperparameter search costs, compatibility of its difficulty measure, and variability across random seeds.
Limitations
- The method introduces many hyperparameters, requiring additional computing resources for search.Source: Limitations · p. 9
- The difficulty measure is incompatible with most existing ERC methods unless their emotion representations are distance-aware.Source: Limitations · p. 9
- Multiple random sampling can produce unstable performance, with results averaged across multiple seeds.The authors report significant variance between seed-specific results in practice.Source: Limitations · p. 9