Source-linked AI summary
Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory
Divyesh Bommana, Mohammad Saim, Tianyu Jiang
TL;DR
Existing emotion benchmarks usually predict one person’s emotion, missing cases where two agents respond oppositely to one shared event. CHIARO introduces a human-annotated, appraisal-grounded benchmark for this paired inference task and evaluates and trains emotion models with it. The strongest models remain below human agreement, while combining CHIARO with existing data improves performance across CHIARO and several external benchmarks.
Problem
Existing emotion benchmarks usually model one person in isolation, leaving joint opposed-valence reactions to a shared event insufficiently evaluated.
Method
CHIARO is a 1,000-sentence human-annotated benchmark where two agents receive opposed-valence emotions from a shared causal event, grounded in appraisal theory.
Results
The strongest LLM reaches 67.3 macro-F1, below human agreement, while existing emotion classifiers perform near chance on CHIARO.
Takeaways & Limitations
CHIARO combined with a matched-size GoEmotions slice beats either source alone on CHIARO and six of ten external emotion benchmarks.
Takeaways & Limitations
CHIARO is English-only, draws narratives from one online community, and restricts the task to positive–negative valence pairs.
Abstract
from arXiv · showhide
Emotion recognition benchmarks often predict one emotion per text, missing many real-world scenarios where two people arrive at opposing emotions from a single shared event. For example, a child kicks the seat in front of her in excitement while the passenger ahead grows angry. We introduce CHIARO, a 1,000 human-annotated sentence benchmark for contrastive emotion inference grounded in appraisal theory. Each scene describes one causal trigger eliciting a positive emotion in one person and a negative emotion in the other, drawn from a ten-class taxonomy. We benchmark seven frontier LLMs and four off-the-shelf emotion classifiers. The strongest LLM reaches 67.3 macro-F1, well below human agreement, while existing emotion classifiers score near chance. Beyond evaluation, CHIARO also serves as a training signal. When combined with an existing emotion corpus, the resulting downstream classifier improves on CHIARO itself and on six of ten external emotion benchmarks, which positions our dataset as a complementary signal for emotion recognition.
1 Introduction
CHIARO addresses emotion inference when two people experience opposing emotions from one shared event, a case existing single-person benchmarks largely miss. It introduces a 1,000-sentence appraisal-grounded benchmark and shows substantial gaps between current models and human agreement.
- Motivation: Shared events can elicit opposing, unstated emotions because people attend to different aspects of the situation.The benchmark targets both emotions as the unit of analysis for interpersonal systems, story generation, and multi-party dialogue analysis.
- Research gap: Existing emotion corpora generally predict one person’s emotion in isolation, while implicit-emotion tasks still focus on a single experiencer.CHIARO instead requires contrasting emotions tied to one shared cause.
- Contribution: CHIARO is a 1,000-sentence benchmark for two-person contrastive emotion inference, grounded in appraisal theory and balanced across five positive and five negative classes.The task links one shared event to opposed valences for two agents.
- Evaluation: Seven frontier LLMs and four off-the-shelf emotion classifiers are evaluated, with the strongest frontier model below human agreement and existing classifiers transferring at chance level.The comparison tests both general-purpose reasoning systems and single-agent emotion classifiers.
- Training use: Combining CHIARO with a matched-size slice of an existing emotion dataset beats either source alone on CHIARO and six of ten external emotion benchmarks.This positions CHIARO as a complementary training resource rather than only an evaluation set.
2 Related Works
Related work establishes that emotion can be distributed across targets, inferred without affect words, or diverge across agents, but no existing dataset jointly requires role-based opposed-valence attribution from one event. CHIARO targets that conjunction directly.
- Existing emotion research: Emotion research spans affective classification, sentiment polarity, dimensional annotation, large-scale corpora, and embodied inference.This literature provides broad coverage but does not by itself define CHIARO’s paired-agent target.
- Adjacent task formulations: Compositional sentiment work shows that conflicting polarities can attach to distinct targets, while implicit-emotion tasks remove explicit affect words.These strands address related forms of complexity without jointly modeling two agents’ opposed emotions.
- Theoretical grounding: Appraisal theory explains divergent reactions to one event through agents’ different goals and agency, while mixed-emotion research supports the coexistence of opposite-valence states.Together, these perspectives motivate contrastive emotion inference as an agent-relative problem.
- Interpersonal relations: Interpersonal emotion regulation formalizes directional influence between agents’ expressive behavior and affective response, informing CHIARO’s shared-agent setup.The related framework contributes a relational perspective beyond isolated emotion labels.
- Research gap: No existing dataset jointly requires detecting opposed valence and assigning the correct polarity to each agent by role.CHIARO addresses this gap with paired physical and non-physical scenario variants for causal grounding and role-aware attribution.
3 Task and Dataset Creation
CHIARO defines contrastive emotion inference as paired role-based prediction for two agents sharing one event, with one positive and one negative emotion inferred from context. Its construction combines appraisal-oriented source narratives, controlled generation, paired causal modes, and validation against explicit emotion leakage.
- Task definition: The task predicts one emotion per agent from a ten-class taxonomy, with exactly one positive and one negative emotion assigned to the two roles.Each scene is represented as a paired assignment over agents A and B.
- Construction pipeline: CHIARO scenes require situational inference rather than explicit affect vocabulary and are constructed through source selection, two-stage generation, and validation or correction.The pipeline is designed to preserve causal context while discouraging keyword shortcuts.
- Emotion taxonomy: The ten labels comprise five positive emotions—joy, pride, relief, gratitude, and excitement—and five negative emotions.The taxonomy is derived from GoEmotions and filtered using clustering, arousal, and appraisal distinctions.
- Trigger grounding: Each target emotion is paired with a mandatory event or trigger so that its situational basis is recoverable from the generated text.Examples include an identifiable helper for gratitude and an avoided prior threat for relief.
- Source narratives: AITA narratives provide interpersonal conflicts and implicit appraisal cues such as fairness, agency, harm, and benefit for generating contrastive scenes.The source genre was selected because its events naturally create opposing affective stakes without naming emotions directly.
- Two-stage generation: Generation first drafts a two-agent scene and then renders paired physical and non-physical versions sharing the same underlying contrast.Physical triggers directly change an agent’s situation, whereas non-physical triggers rely on social or environmental cues and theory-of-mind reasoning.
- Validation: The corpus excludes affect-bearing words and applies six validators covering valence, lexical constraints, length, references, span consistency, and role collisions.Failed versions return to the model for bounded correction.
4 State-of-the-Art LLMs
The evaluation finds a substantial gap between frontier LLM performance and human agreement on CHIARO, with errors concentrated in fine-grained positive emotions and generally better performance on non-physical scenes.
- 67.3 macro-F1 is the best LLM result, achieved by GPT-5.5 against adjudicated human gold, while human agreement reaches 93.0 macro-F1.
- 31.4 to 89.3 is the range of GPT-5.5’s per-class F1, from joy to fear, with the three lowest-F1 emotions on the positive side.
- Positive emotions account for most of GPT-5.5’s gap with human agreement, while negative-polarity imbalance is weaker.
- GPT-5.5 reaches 64.9 macro-F1 on physical scenes versus 70.0 on non-physical scenes.
5 Emotion Classifiers
Existing single-agent classifiers transfer poorly to CHIARO, whereas a RoBERTa-large model trained on CHIARO and GoEmotions improves both in-distribution and many external benchmark results.
- 11.8–29.0 macro-F1 is the range for three GoEmotions-trained encoders on CHIARO, placing them 36–54 points below the LLM mean.
- 79.6 versus 29.0 macro-F1 is ModernBERT-large’s performance on filtered GoEmotions and CHIARO, respectively, with architecture, scoring, and label set held fixed.
- 69.5% is CHIARO-only RoBERTa-large accuracy on the held-out CHIARO test split, compared with 44.0% for RoBERTa-base on the same training data.
- 73.5% versus 69.5% is the Combined model’s in-distribution accuracy relative to CHIARO-only, and Combined leads both baselines on six of ten external datasets.
- CHIARO-only remains stronger on EmotionX-2019 and MELD, GoEm-only is slightly stronger on GoEmotions and CARER, and Combined leads on mixed-demand benchmarks.
6 Conclusion
CHIARO is a 1,000-sentence human-annotated benchmark for two-agent contrastive emotion inference, where opposite-valence emotions arise from one shared causal event and are grounded in appraisal theory. Results show substantial gaps between current models and human agreement, while combined training makes CHIARO useful as a complementary signal.
- CHIARO is a 1,000-sentence human-annotated benchmark for two-agent contrastive emotion inference.
- Each scene presents opposite-valence emotions tied to one causal event, with inference grounded in appraisal theory rather than affect vocabulary.
- 65.3 macro-F1 is the seven-model LLM mean, below 93.0 macro-F1 for human annotators, with ¯κ = 0.827 inter-annotator agreement.
- Four off-the-shelf emotion classifiers transfer to CHIARO at near-chance level despite covering all ten labels, isolating per-agent attribution as the missing piece.
- The union of CHIARO with a matched-size GoEmotions slice beats either source alone on CHIARO and six of ten external emotion benchmarks.
Limitations
CHIARO is designed as an English-language evaluation benchmark and complementary training signal, but its source community, generator, annotation pool, valence pairing, and scale constrain its coverage. The paper identifies broader, more diverse data and expansion beyond AITA as next steps.
- CHIARO is English-only and grounded in r/AmItheAsshole, skewing its situations, social norms, and interpersonal scripts toward U.S. and Anglophone framings.
- Generation uses a single model, gpt-5.2, so residual stylistic and topical biases may remain despite lexical constraints, repair loops, and human adjudication.
- A larger and more demographically diverse annotation pool could tighten human agreement and reduce annotation biases.
- The positive–negative valence restriction excludes same-polarity different-emotion cases and scenarios where one or both agents are emotionally neutral.
- CHIARO is sized for evaluation and complementary training rather than deployment-scale training, with the released pipeline supporting future expansion.
C Contrastive Scenario Types
CHIARO organizes generated scenes into six contrastive scenario types that specify how one shared situation produces different outcomes or appraisals for two agents. Generation and validation enforce causal, valence, lexical, role, and span constraints.
- Six contrastive scenario types encode different causal structures linking the two agents’ outcomes, refining the OCC fortunes-of-others grid.
- Zero-sum gain/loss uses a scarce resource whose mutually exclusive allocation produces affectively charged gain and denial for comparable agents.
- Side-effect spillover models a positively motivated activity that creates a negative byproduct for another agent, such as seat-kicking that bothers a passenger.
- Competing preferences describe opposing needs in one environment, where satisfying one preference automatically works against the other.
- Success vs. failure contrasts parallel attempts in which one agent succeeds and the other fails, with appraisal shaped by comparison, locus, and controllability.
- Every version must contain exactly one positive and one negative emotion, avoid banned lexical or behavioral cues, and ground both agents’ evidence in one event.
F Training Hyperparameters
All three RoBERTa-large checkpoints use the same AdamW optimization and training setup, differing only in their training corpus.
- All three RoBERTa-large checkpoints use AdamW with learning rate 2 × 10^-5, weight decay 0.01, batch size 16, bf16 precision, and five epochs.
- Best-on-validation-accuracy checkpoint selection and seed 42 are used for every checkpoint.
- RoBERTa-large has approximately 355M parameters, and each checkpoint takes approximately 2–3 hours on one NVIDIA A100 GPU.
G Taxonomy: Discard Mapping and Mandatory Triggers
CHIARO reduces the source taxonomy to ten retained emotions using explicit appraisal-based discard criteria and mandatory situational triggers. Its staged generation and balanced sampling produce a final release near the uniform class distribution.
- Discard mapping: Table 7 maps eighteen non-retained GoEmotions categories to retained CHIARO emotions or outright drops using explicit discard criteria.The criteria include cluster deduplication, intensity siblings, appraisal overlap, and lacking fixed valence.
- Mandatory triggers: Table 8 pairs each retained emotion with a mandatory situational trigger inserted into the Stage 1 generation prompt.These triggers provide recoverable disambiguation cues despite lexical constraints and sentences without affect vocabulary.
- Balanced sampling: 20.0% disgust and 3.4% sadness in early generation revealed strong class imbalance before quota-based balanced sampling was introduced.Gratitude accounted for 17.0% of early generations.
- Balanced sampling: Every emotion was brought within roughly ±1.5 percentage points of the uniform 10% baseline, and the final release contains 1,000 adjudicated scenes.The released corpus inherits this balance after adjudication.
J Per-Emotion Breakdown for All LLMs
Per-emotion results for GPT-5.5 show that positive emotions, especially joy and gratitude, vary most across the breakdown and have the lowest absolute F1, while negative emotions are comparatively flat.
- Positive emotions: Joy and gratitude show the widest variance and lowest absolute F1 among the positive emotions for GPT-5.5.The breakdown reports precision, recall, F1, and support against adjudicated human gold.
- Negative emotions: Negative emotions have comparatively flat per-emotion performance across the GPT-5.5 breakdown.The comparison uses predictions pooled across both agent slots.
K Evaluation Prompt
The evaluation gives all models the same multiple-choice task for two agents reacting to one event with contrasting emotions. Each agent chooses among five gold-polarity emotions, with answer letters rotated across scenes and outputs constrained to two lines.
- Prompt design: All seven LLMs and Emollama-chat-7B receive the same multiple-choice prompt for two agents reacting to one shared event.The prompt asks for one best answer for each agent.
- Prompt design: Each agent receives five answer options corresponding to the five emotions of that agent’s gold polarity.Correct option letters are rotated across scenes.
- Output format: The required response format is exactly two lines: one letter for Agent A and one for Agent B.The system message explicitly forbids additional output.