Source-linked AI summary
Exposing Weaknesses in Emotion Recognition in Conversations
Amir Ben Khalifa, Fanny Bezancon, Amine Trabelsi, Bessam Abdulrazak
TL;DR
ERC benchmarks often force context-dependent utterances into one emotion label, making it unclear whether disagreements reflect poor model understanding or annotation ambiguity. The paper combines zero-shot LLM evaluation, human re-annotation, and agreement analysis, then introduces an LLM-as-Judge evaluator. It finds systematic failure patterns and concludes that many apparent errors reflect genuine uncertainty rather than poor emotion understanding.
Problem
Single-label ERC benchmarks assume each utterance has one unambiguous emotion, despite contextual cases where multiple labels may be plausible.
Source: Introduction · pp. 1–2Method
The study evaluates LLMs zero-shot with conversational context, re-annotates representative data with additional humans, analyzes agreement, and introduces an LLM-as-Judge evaluator.
Source: Introduction · p. 2Results
Systematic failures concentrate around linguistic cues such as negations, exclamations, and interjections, while human re-annotation shows that many apparent errors reflect annotation ambiguity.
Source: Introduction · p. 2, Consistent Failure Modes in Utterances · p. 4, Conclusion · p. 7Takeaways & Limitations
Single-label evaluation is insufficient for ERC, motivating evaluation that judges each emotion independently by its plausibility in conversational context.
Source: Introduction · p. 2, Conclusion · p. 7Many ERC instances lack clear emotional consensus even among humans, which constrains the validity of single-label ground truth.
Source: Re-Annotation Setup and Agreement… · p. 5
Abstract
from arXiv · showhide
Emotion Recognition in Conversations (ERC) aims to identify speakers' emotions in multi-turn dialogue. Accurate emotion recognition can support a wide range of applications, including empathetic conversational agents, mental health support, and educational technologies. While many recent approaches rely on task-specific fine-tuning, such models may exploit dataset-specific cues. A central yet rarely questioned assumption in ERC is that each utterance can be assigned a single unambiguous emotion label. To investigate this assumption, we study ERC using Large Language Models (LLMs) in a zero-shot setting while incorporating preceding conversational turns as context. We show that aggregate metrics mask systematic failures. Errors concentrate around utterances containing negations, exclamations, and interjections. This pattern is consistent across all evaluated models, suggesting limitations in the benchmarks rather than model-specific weaknesses. A controlled re-annotation study involving four human annotators supports this finding: strong agreement is observed in only 35 percent of cases, with neutral utterances dominating high-agreement instances, while many emotional categories fall into low-agreement regimes. These findings suggest that many apparent model errors reflect genuine annotation ambiguity rather than poor emotion understanding. Standard single-label evaluation is therefore insufficient. To address this limitation, we introduce an LLM-as-Judge framework that evaluates each emotion independently according to its plausibility in the conversational context rather than enforcing a single-label decision.
Introduction
ERC research commonly assumes each conversational utterance has one unambiguous emotion label, but contextual intent and linguistic cues can support multiple valid interpretations. This study tests that assumption through zero-shot LLM evaluation, human re-annotation, and an LLM-as-Judge framework.
- ERC predicts the emotion of each utterance in multi-turn dialogue and supports applications including empathetic dialogue systems and mental health monitoring.Source: Introduction · p. 1
- Single-label ERC benchmarks can misrepresent emotion understanding because context, speaker intent, and subtle linguistic cues may make multiple labels plausible.Source: Figure 1 · p. 1, Introduction · p. 2
- The study combines systematic zero-shot evaluation with human re-annotation and inter-annotator agreement analysis to distinguish model errors from annotation ambiguity.Source: Introduction · p. 2
- An LLM-as-Judge framework evaluates whether each candidate emotion is plausible given the utterance and surrounding dialogue, moving beyond single-label constraints.Source: Introduction · p. 2
Related Work
ERC has progressed from feature-based, recurrent, and graph-based models toward LLM-based methods that use contextual and auxiliary information. However, reported gains may depend on dataset-specific cues and may not generalize across benchmarks.
- Earlier ERC methods modeled temporal dynamics, speaker interactions, and conversational structure with recurrent and graph-based architectures.Source: Related Work · p. 2
- Recent approaches use LLMs and auxiliary signals such as scene descriptions and persona cues to improve emotion recognition.Source: Related Work · p. 2
- These improvements often rely on dataset-specific contextual signals and do not generalize well across benchmarks.Source: Related Work · p. 2
Experimental Setup
The study evaluates zero-shot LLM emotion recognition across diverse ERC benchmarks using conversational context, complementary metrics, and human-agreement analysis. The setup compares model performance with class imbalance, per-emotion behavior, and annotation consistency.
- The study uses two stages: zero-shot evaluation across ERC datasets, followed by re-annotation of representative subsets and inter-annotator agreement analysis.Source: Experimental Setup · pp. 2–3
- The benchmarks differ in genre, emotion inventories, and class imbalance, with DailyDialog containing over 80% neutral utterances.Source: Datasets · p. 3
- All preceding dialogue turns with speaker attribution are included so target utterances are evaluated with conversational history.Source: Prompting Protocol · p. 3
- Accuracy, Weighted F1, Macro F1, Cohen’s κ, and False Positive Rate provide complementary views of overall performance, class imbalance, human consistency, and unsupported predictions.Source: Evaluation Metrics · p. 3
Disagreement Analysis
Aggregate zero-shot metrics conceal systematic weaknesses, especially for minority and ambiguous emotions. Errors repeatedly cluster around specific linguistic forms, while manual inspection indicates that several apparent mistakes may reflect genuine annotation ambiguity.
LLM performance Analysis
- Weighted-F1 scores of 0.60–0.75 on MELD and DailyDialog coexist with persistent failures on minority emotions, while EmoryNLP Weighted-F1 is 0.31–0.39.Source: LLM performance Analysis · p. 4
- Across benchmarks, models perform better on dominant classes but struggle with rare or ambiguous categories, so Weighted-F1 can mask class-level weaknesses.Source: LLM performance Analysis · p. 4
Consistent Failure Modes in Utterances
- Manual inspection and rule-based tagging identify negations, exclamations, interjections, interrogations, and short replies as recurring failure-mode categories.Source: Consistent Failure Modes in Utterances · p. 4
- Across models, DailyDialog errors are highest for negations and exclamations at around 45–50%, followed by interjections and interrogations.Source: Consistent Failure Modes in Utterances · p. 4, Figure 3 · p. 4
- Multiple emotional interpretations can be plausible for the same utterance, motivating human-agreement analysis to separate model limitations from data ambiguity.Source: Consistent Failure Modes in Utterances · p. 4
ERC Benchmarks
The re-annotation study finds that agreement is usually moderate or low, especially for minority emotions, revealing ambiguity tied to unclear intents, missing lexical cues, and context.
Agreement patterns
- Most annotated utterances fall into moderate (3/5) or low (2/5) agreement regimes, while high-consensus cases are a minority dominated by neutral emotion.Source: Re-Annotation Setup and Agreement… · p. 5, Figure 4 · p. 5
- Minority emotions such as fear, disgust, sadness, peaceful, and powerful are concentrated in lower-agreement regimes across datasets.Source: Re-Annotation Setup and Agreement… · p. 5
Implications
- These findings expose a structural limitation of single-label benchmarks because many utterances lack a clear human emotional consensus.Source: Re-Annotation Setup and Agreement… · p. 5, Linguistic and Contextual Factors… · p. 6
Sources of disagreement
- A decision-tree classifier predicts agreement levels with accuracies of 0.84 for MELD, 0.76 for DailyDialog, 0.68 for IEMOCAP, and 0.58 for EmoryNLP.The results support the view that disagreement is linked to observable linguistic and contextual cues rather than being random.Source: Table 3 · p. 5, Linguistic and Contextual Factors… · p. 6
- Ambiguous utterances commonly have unclear intent, multiple competing intents, absent lexical emotion cues, and unclear context.Source: Linguistic and Contextual Factors… · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
Emotion Scenarios
The multi-annotation-aware evaluation treats a prediction as correct when it matches at least one human-annotated emotion, while benchmark comparisons show persistent difficulty for minority emotions.
Multi-annotation evaluation
- A prediction is considered correct when it matches at least one human-annotated emotion rather than only the original dataset label.Source: Emotion Scenarios · p. 6, Table 4 · p. 7
Benchmark results
- Multi-annotation-aware accuracy results show strong inter-model agreement, largely driven by neutral and positive emotions, while minority emotions remain challenging.Source: Table 4 · p. 7
- Table 5 compares LLM-as-Judge models within each ERC dataset.Source: Table 5 · p. 7
Evaluation with LLM-as-Judge
The LLM-as-Judge framework evaluates each candidate emotion independently for plausibility in full conversational context, allowing multiple valid interpretations. Across judges and datasets, it produces strong and consistent agreement with human annotations.
Framework
- The framework presents one candidate emotion at a time and asks whether it is plausible given the utterance and its conversational context.Source: Evaluation with LLM-as-Judge · p. 7
- Unlike direct emotion prediction, the judge can accept multiple emotional interpretations of the same utterance.Source: Evaluation with LLM-as-Judge · p. 7
Results
- Compatibility scores range from 0.69 to 0.91 across models and benchmarks, while accuracy ranges from 0.68 to 0.86.Source: Evaluation with LLM-as-Judge · p. 7
- DailyDialog consistently performs strongest and IEMOCAP remains challenging, with only modest differences across model families and sizes.Source: Evaluation with LLM-as-Judge · p. 7
- False Positive Rates remain low across models and datasets, indicating conservative plausibility judgments.Source: Evaluation with LLM-as-Judge · p. 7
Conclusion
The paper combines zero-shot LLM evaluation, human re-annotation, agreement analysis, cue investigation, and LLM-as-Judge evaluation to examine limitations of single-label ERC benchmarks.
- A substantial portion of ERC data is not well captured by strict single-label annotations, and many apparent model errors reflect genuine uncertainty.Source: Conclusion · p. 7
- The LLM-as-Judge framework offers a way to move beyond strict labeling without requiring extensive human re-annotation.Source: Conclusion · p. 7