Source-linked AI summary
MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, Rada Mihalcea
TL;DR
Emotion recognition in conversations lacks a large multimodal, multi-party dataset, limiting research on settings with more than two speakers. The paper extends EmotionLines into MELD, combining textual dialogues with audio and visual data and evaluating strong conversational baselines. The experiments report gains from multimodal fusion, while identifying class imbalance and visual speaker identification as continuing constraints.
Problem
Emotion recognition in conversations lacks a large multimodal conversational dataset, while existing conversational approaches are mostly limited to dyadic conversations.
Method
The paper extends EmotionLines into MELD and evaluates contextual conversational baselines using textual and audio features alongside multimodal inputs.
Results
Multimodal fusion improves emotion recognition performance by 3%, while multimodal DialogueRNN achieves 67.56% F-score for sentiment classification.
Takeaways & Limitations
MELD provides raw videos, audio segments, transcripts, and baseline features for conversational emotion recognition and multimodal empathetic response generation.
Takeaways & Limitations
Future work must improve contextual modeling and address the challenge of identifying a speaker’s face when multiple people appear in MELD videos.
Abstract
from arXiv · showhide
Emotion recognition in conversations is a challenging task that has recently gained popularity due to its potential applications. Until now, however, a large-scale multimodal multi-party emotional conversational database containing more than two speakers per dialogue was missing. Thus, we propose the Multimodal EmotionLines Dataset (MELD), an extension and enhancement of EmotionLines. MELD contains about 13,000 utterances from 1,433 dialogues from the TV-series Friends. Each utterance is annotated with emotion and sentiment labels, and encompasses audio, visual and textual modalities. We propose several strong multimodal baselines and show the importance of contextual and multimodal information for emotion recognition in conversations. The full dataset is available for use at http:// affective-meld.github.io.
1 Introduction
Emotion recognition in conversations is limited by insufficient large multimodal, multi-party data and requires modeling both conversational context and signals across modalities. MELD extends EmotionLines with a larger multi-party multimodal resource and establishes baselines for this setting.
- Research gap: Emotion recognition in conversations lacks a large multimodal conversational dataset and remains less studied than general multimodal emotion recognition.
- Research challenges: Conversational emotion recognition must account for contextual dependencies, emotion shifts, and multiple interlocutors.
- Multimodal motivation: Multimodal cues can supplement misleading or difficult textual signals, helping classify emotions and sentiments in conversational utterances.
- Research challenges: Approximately 42% of MELD utterances contain fewer than five words, making context and additional modalities relevant for interpreting short utterances such as “yeah” or “okay.”
- MELD contributions: MELD extends EmotionLines into a multimodal dataset containing multi-party conversations, with more than 13,000 utterances and audio, visual, and textual sources.
- MELD contributions: The paper establishes DialogueRNN as a strong baseline for emotion recognition in multi-party dialogues through interparty dependency modeling.
2 EmotionLines Dataset
EmotionLines is a Friends-based multi-speaker dialogue dataset created for textual emotion analysis. Its construction sampled 1,000 dialogues and assigned utterance-level emotion labels through crowdsourced majority voting.
- Dataset origin: EmotionLines contains dialogues from the sitcom Friends, with utterances produced by multiple speakers.
- Dataset construction: The dataset was formed by grouping dialogues into four utterance-length ranges and randomly sampling 250 dialogues from each group.
- Dataset construction: The resulting EmotionLines dataset contains 1,000 dialogues.
- Annotation: Utterances were labeled using six Ekman emotions plus Neutral and Non-Neutral categories.
- Annotation: Five Amazon Mechanical Turk workers annotated each utterance, with majority voting producing the final label and an overall Fleiss’ kappa of 0.34.
3 Multimodal EmotionLines Dataset (MELD)
MELD extends EmotionLines into a multimodal, multi-party resource through timestamp alignment, filtering, audiovisual extraction, and multimodal re-annotation. The resulting dataset supports emotion and sentiment analysis across multiple speakers and modalities.
- Dataset construction: Timestamp constraints require utterances to occur in increasing order within the same episode and scene, filtering cross-scene or cross-episode anomalies.
- Dataset construction: MELD retains dialogues while adding visual, audio, and textual modalities for each utterance.
- Dataset re-annotation: Three annotators label each utterance using video clips, with majority voting determining final labels and 89 utterances across 11 dialogues removed after unanimous disagreement.
- Dataset re-annotation: 0.43 Fleiss’ kappa exceeds EmotionLines’ 0.34, indicating higher annotation agreement after incorporating additional modalities.
- Dataset re-annotation: 2,772 EmotionLines utterances marked non-neutral without a definite emotion label are contrasted with 89 disagreements removed from MELD.
- Comparison with existing datasets: Multi-party MELD contains more than 13,000 emotion-labeled utterances, nearly twice the annotated utterances in IEMOCAP and SEMAINE, while also offering a dyadic variant.
4 Experiments
The experiments evaluate multimodal and contextual baselines for emotion and sentiment classification on MELD. Results show gains from multimodal fusion, conversational context, and speaker-specific modeling, while class imbalance and fine-grained emotions remain challenging.
- Baseline Models: DialogueRNN models multi-party context by tracking global and individual speaker states, then uses an emotion GRU with attention for classification.The experiments compare DialogueRNN and bcLSTM in unimodal and multimodal settings.
- Results: 67.56% F-score is achieved by multimodal DialogueRNN for sentiment classification, surpassing multimodal bcLSTM at 66.68% F-score.Multimodal DialogueRNN also outperforms its unimodal counterparts, although fusion improves over text by about 1.4%.
- Results: 57.03% F-score for the textual classifier exceeds 41.79% F-score for the audio classifier in emotion classification.The experiments report particularly poor performance for disgust, fear, and sadness, partly because of dataset imbalance.
- Results: 3% improvement in emotion recognition performance comes from multimodal fusion, although the multimodal classifier performs worse than the textual classifier for sadness.Removing infrequent fear and disgust emotions improves results in the 5-class setting, with the multimodal variant performing significantly better.
- Additional Analysis: 1.4% to 2.5% improvement over text-CNN demonstrates the contribution of contextual modeling in the MELD experiments.Text-CNN does not use conversational context or multimodality.
- Additional Analysis: 63% of correct test predictions attend most strongly to utterances from different speakers, while DialogueRNN outperforms bcLSTM by a 1-2% margin.These findings support the value of inter-speaker information in multi-party emotion recognition.
5 Future Directions
Future work should improve contextual modeling, visual speaker-feature extraction, and multimodal fusion for MELD. The paper also reports that audio features do not help significantly in the current baselines.
- Contextual modeling, reasoning about decisions, emotional influences, and emotion shifts are identified as promising future directions.
- Visual feature extraction remains challenging because identifying a speaker’s face is difficult when multiple people appear in the video.
- Figure 4 summarizes the distance between target and context utterances using emotion GRU attention scores.
- Audio features do not help significantly in the reported results, motivating improved extraction for auxiliary modalities.
- Concatenation fusion outperforms unimodal baselines by about 1-3%, while advanced methods such as MARN may offer further improvement.
6 Applications of MELD
MELD supports emotion-aware and multimodal dialogue applications. Its uses include empathetic response generation, user emotion and personality modeling, and training multimodal dialogue systems.
- MELD can train emotion classifiers used as emotional receptors in generative dialogue systems for empathetic responses.
- MELD can support emotion and personality modeling of users in conversations.
- MELD can train multimodal dialogue systems that use speakers’ voices and facial expressions to generate responses.
- MELD alone is not large enough to train an end-to-end dialogue system, but its construction procedures can help create larger multimodal corpora.
7 Conclusion
The paper introduces MELD as a multimodal, multi-party conversational emotion recognition dataset and provides data, features, and baseline results for future research.
- MELD is introduced as a multimodal, multi-party conversational emotion recognition dataset.
- The dataset provides raw videos, audio segments, transcripts, and features used in baseline experiments.
- MELD is intended as a training corpus for conversational emotion recognition and multimodal empathetic response generation.
- Future work can explore multimodal fusion algorithms, ERC frameworks, and new audio, visual, and textual features.