AffectReveal: Event-Grounded Emotion Recognition Beyond Visual Appearances

1 INTRODUCTION

• AffectReveal addresses affect-critical event ambiguity and appearance-induced circularity, which can undermine retrieval-based emotion reasoning.A topically plausible event may imply the wrong emotion if its identity, role, relationship, or outcome is misread; validating it with the same potentially misleading expression is circular.Source: §1 · p. 2
• AffectReveal constructs and independently verifies evidence-anchored event alternatives, then checks the recovered event against non-facial visual facts before prediction.The original unmasked input and verified event evidence are jointly used for final prediction.Source: §1 · p. 2
• AffectReveal improves UAR and accuracy in all 16 model-setting comparisons, with average UAR gains of 5.26–10.53 points.It also outperforms fine-tuned visual-only counterparts in all 12 corresponding accuracy comparisons without updating downstream parameters.Source: §1 · p. 2
• EGER requires recovering the external event needed to interpret a focal person’s visible reaction, because decisive context may lie beyond the observed media.The same tears or smile can accompany different emotions, and the event’s identity, focal-person role, relationship, or outcome may be decisive.Source: §1 · pp. 1–2
• EGER-Bench pairs visually insufficient or misleading observations with source-grounded event annotations across two domains and four visual settings.It contains 10,052 video clips and 10,734 images across 11 emotions; event context raises human recognition accuracy from 33.96% to 72.08%.Source: §1 · pp. 1–2

Visual and Context-Aware Emotion Recognition

• Context-aware emotion models incorporate posture, scenes, objects, actions, and interactions, but their evidence remains within the observed media (Kosti et al., 2017; 2019; Lee et al., 2019; Hoang et al., 2021; Wu et al., 2022; Mittal et al., 2020; Yang et al., 2022; 2023; Li et al., 2026).EGER concerns events not fully observable in the input rather than a particular downstream model architecture (Liu et al., 2023; Chen et al., 2024; Cheng et al., 2024; Lian et al., 2025; Peng et al., 2026).Source: §2 · pp. 2–3

Emotion Causes and Event-Level Affect Understanding

• Emotion-cause methods typically identify causes from provided conversations, documents, or complete multimodal sequences, whereas EGER requires external event recovery and determining the focal person’s role, relationship, and outcome (Xia & Ding, 2019; Rashkin et al., 2019; Poria et al., 2021; Wang et al., 2022).Source: §2 · p. 3

Retrieval-Augmented Affect Reasoning

• Retrieval-augmented reasoning can acquire external evidence, but AffectReveal targets cases where topically relevant information may still imply the wrong emotion (Lewis et al., 2020; Izacard & Grave, 2021; Yao et al., 2022; Schick et al., 2023; Qin et al., 2024; Madaan et al., 2023; Asai et al., 2024; Gou et al., 2024; Wang et al., 2026).It verifies evidence-grounded alternatives over affect-critical event details and cross-checks the recovered event against non-facial visual evidence.Source: §2 · p. 3

3.1 TASK DEFINITION

• EGER models emotion as depending on the observed media and a latent affect-determining event that must be recovered at inference.The event is unavailable at inference; an external information space may be queried, while emotion prediction remains the primary task.Source: §3.1 · p. 3

3.2 DATA COLLECTION

• EGER-Bench draws from news/interviews and television series, yielding 10,052 short videos and 10,734 images across 11 emotions and four visual settings.Discovery uses reaction-focused seed patterns; video clips average 6.1 seconds, and the image set includes representative video frames and 682 event-grounded news images.Source: §3.2 · p. 3, Figure 2 · p. 4

3.3 EVENT-GROUNDED ANNOTATION

• Retained samples have one of 11 emotion labels, determined from the focal person’s reaction and role in the complete event rather than appearance alone.Samples with low visual quality, no identifiable focal person, or inadequate event grounding are excluded; reference materials are not provided in the standard setting.Source: §3.3 · p. 4
• Event context raised human accuracy from 33.96% to 72.08% on 480 instances.Different annotators assessed each instance with restricted visual input alone or with its reference event context.Source: Table 1 · p. 4, §3.3 · p. 4

3.4 VISUAL INPUT SETTINGS AND SPLITS

• The four settings vary video versus image input and whether subtitles or scene text remain visible; no audio or separately supplied transcript is used.Text-retained and text-removed pairs share underlying frames, and the benchmark uses an approximately 6:1:3 train/validation/test split.Source: §3.4 · p. 4

4.1 OVERVIEW

• AffectReveal addresses event ambiguity and expression-induced circularity by recovering four event factors, checking face-masked context, and predicting with verified evidence and the unmasked input.The four factors are event identity, focal-person role, interpersonal relationship, and outcome; they organize verification rather than exhaustively defining event context.Source: §4.1 · p. 5

4.2 AFFECT-CRITICAL EVENT RECOVERY

• Visual clues and ranked event-oriented searches yield a primary event summary with its description, time, location, affect-critical details, and supporting sources.Clues include people, scenes, actions, visible text, distinctive objects, and observable reactions.Source: §4.2 · p. 5
• For uncertain event factors, AffectReveal constructs evidence-grounded alternatives, independently checks them, and retains only supported or partially supported alternatives for arbitration.Arbitration combines the primary event, retained alternatives, visual clues, and retrieved evidence without directly predicting an emotion label.Source: §4.2 · pp. 5–6

4.3 EXPRESSION-DISENTANGLED EVENT VERIFICATION

• Face-masked verification aligns atomic internal and external facts bidirectionally, scores support, and measures observable compatibility without requiring every event fact to appear in-media.The strongest counterpart for each fact is retained, and support is aggregated in both directions.Source: §4.3 · p. 6

4.4 EVENT-CONDITIONED EMOTION INFERENCE

• The final evidence package combines external context, in-media context, and their verification relation with the original unmasked input, while leaving downstream model parameters unchanged.The package is provided to the downstream model for emotion prediction.Source: §4.4 · p. 6

5.1 EXPERIMENTAL SETUP

• The experiments use four image/video settings with and without visible text, report Accuracy, UAR, and WAF, and treat UAR as primary because emotion classes are imbalanced.Evaluated downstream models include general-purpose and emotion-oriented systems, with fine-tuned variants for locally deployable models.Source: §5.1 · p. 6

5.2 MAIN RESULTS ACROSS MODELS AND INPUT SETTINGS

• AffectReveal improves UAR, accuracy, and WAF for every untuned downstream model across all four settings, with average UAR gains of 7.05 for Emotion-LLaMA, 5.26 for AffectGPT, and 10.53 for Qwen2.5-Omni-7B.It also outperforms fine-tuned visual-only counterparts in all 12 accuracy comparisons, and applying it to fine-tuned models yields further gains across every model and setting.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 DIAGNOSTIC VALUE OF EXTERNAL EVENT CONTEXT

• Reference event context raises UAR from 24.15% to 39.36%, while AffectReveal reaches 34.76% and closes 69.8% of the gap using automatically recovered evidence.The reference context is a diagnostic rather than perfect oracle because it may omit relevant role, relationship, or outcome details.Source: §5.3 · pp. 7–8, Table 3 · p. 8

5.4 DOES AFFECT-CRITICAL VERIFICATION IMPROVE EVENT RECOVERY?

• Under the same retrieval budget, affect-critical alternatives raise fully or partially aligned event recovery from Generic Multi-Search’s 38.88% to 42.86% and reduce mismatches to 57.14%.The largest factor-level gain is interpersonal relationship, from 25.53% to 31.06%; event identity, focal-person role, and outcome also improve.Source: §5.4 · p. 8, Table 4 · p. 8

5.5 HOW DOES EVENT VERIFICATION CONTRIBUTE TO EMOTION RECOGNITION?

• External event context raises UAR from 17.11% to 27.84%, while adding in-media context alone gives no further improvement; modeling its support relation raises UAR to 29.25% and WAF to 27.11%.Removing face masking reduces WAF by 2.74 points, and replacing atomic support with holistic comparison causes smaller but consistent decreases across all metrics.Source: §5.5 · pp. 8–9, Table 3 · p. 8

5.6 QUALITATIVE ANALYSIS

• In the Olympic example, correcting event identity and outcome changes the prediction from happiness to sadness; in the television example, correcting event identity and relationship changes it from happiness to surprise.Primary Retrieval identifies the focal person in both cases but retrieves affectively incorrect event details.Source: §5.6 · p. 9, Figure 4 · p. 9

• EGER and EGER-Bench address recognition of visual reactions whose affect-determining events are not fully observable.The benchmark is introduced alongside the task formulation.Source: §6 · p. 9
• AffectReveal verifies factorized event alternatives and cross-checks recovered context against expression-disentangled visual evidence.It improves recognition across multiple models and visual settings and complements task-specific fine-tuning without updating downstream parameters.Source: §6 · p. 9
• Future work will extend EGER to dynamic, interactive settings where systems seek evidence and revise emotion interpretations as events unfold.Source: §6 · p. 9

• Generative AI tools assisted with language polishing, text compression, and LATEX formatting during manuscript preparation.Source: AI USE STATEMENT · p. 10
• The authors reviewed and revised all AI-assisted content and independently verified the technical claims, citations, experimental results, and numerical analyses.Source: AI USE STATEMENT · p. 10
• The authors state that generative AI was not used as an authoritative source or to fabricate data or results, and that they take full responsibility for the submission.Source: AI USE STATEMENT · p. 10

• Emotion recognition from human behavior raises privacy, subjective-interpretation, cultural-variation, and potential-misuse concerns.Source: ETHICS STATEMENT · p. 10
• EGER-Bench is intended solely for research on event-grounded emotion understanding, not objective psychological assessment of individuals.Source: ETHICS STATEMENT · p. 10
• The authors do not advocate using AffectReveal or EGER-Bench for surveillance, clinical assessment, employment screening, law enforcement, or other high-stakes decisions about individuals.The benchmark is constructed from news/interview media and television-series content, with source provenance retained; releases will respect applicable licenses and copyright restrictions.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Sections 3–5 describe dataset construction, the AffectReveal framework, and experimental settings; the appendix supplies prompts, evaluation protocols, implementation details, and inference configurations.Source: REPRODUCIBILITY STATEMENT · p. 10

APPENDIX CONTENTS

• Appendix A reports additional EGER-Bench results with Qwen3.5-Omni-Plus, Appendix B evaluates vision-only transfer on MER2023, and Appendix C compares Qwen-based event-recovery assessments with human annotations.Source: APPENDIX CONTENTS · p. 14
• Appendix D analyzes performance across News/Interview and TV Series, while Appendix E provides prompt templates and inference configuration for AffectReveal’s core components.Source: APPENDIX CONTENTS · p. 14

A ADDITIONAL RESULTS WITH QWEN3.5-OMNI-PLUS

• 7.16 points is the average UAR improvement for Qwen3.5-Omni-Plus across four EGER-Bench settings; the largest gain is VID-V, from 24.15 to 34.76.IMG-V UAR rises from 20.35 to 29.16; with visible text, UAR rises from 34.91 to 39.50 for VID-VS and from 27.08 to 31.70 for IMG-VS.Source: Table 5 · p. 14, A ADDITIONAL RESULTS WITH… · p. 14
• The Qwen3.5-Omni-Plus results complement locally deployable models and show benefits beyond models whose parameters or training procedures are accessible.Source: A ADDITIONAL RESULTS WITH… · p. 14

B VISION-ONLY TRANSFER TO MER2023

• The MER2023 vision-only transfer evaluation tests whether recovered event evidence can help conventional visual emotion recognition when the observed media is limited; it does not establish EGER.Source: B VISION-ONLY TRANSFER TO MER2023 · p. 14

C VALIDATION OF AUTOMATIC EVENT RECOVERY EVALUATION

• Six annotators categorized recovered event contexts for 500 randomly selected VID-VS test samples as Match, Partial Match, or Mismatch.Source: C VALIDATION OF AUTOMATIC EVENT… · p. 14, Figure 5 · p. 15
• 64.4% agreement was observed between Qwen-based judgments and human annotations in the event-recovery assessment.Source: Figure 5 · p. 15, C VALIDATION OF AUTOMATIC EVENT… · p. 15
• Most assessments fell along the confusion matrix diagonal, while disagreements mainly involved neighboring categories, especially partially recovered events.Source: C VALIDATION OF AUTOMATIC EVENT… · p. 15, Figure 5 · p. 15

D PERFORMANCE ACROSS SOURCE DOMAINS

• Across all samples, AffectReveal improves accuracy from 28.74% to 29.80%, UAR from 28.09% to 29.25%, and WAF from 25.09% to 27.11%.It improves all three metrics in both domains; gains are larger on TV Series than News/Interview for accuracy (2.08 vs. 0.29 points), UAR (1.60 vs. 0.58), and WAF (3.78 vs. 0.76). The larger TV Series gains are consistent with the need to resolve narrative outcomes, character roles, and relationships beyond the focal visual input.Source: D PERFORMANCE ACROSS SOURCE DOMAINS · pp. 15–16, Table 7 · p. 16

E PROMPT TEMPLATES AND INFERENCE CONFIGURATION

• The fixed inference setup assigns staged thinking to Qwen3.5-Flash and greedy decoding to Qwen2.5-Omni-7B.Qwen2.5-Omni-7B uses max_new_tokens=512; its video input for Non-Facial In-Media Context is sampled at 1 FPS with audio disabled. Qwen3.5-Flash uses automatic tool selection during retrieval calls.Source: E PROMPT TEMPLATES AND INFERENCE… · p. 16

E.1 AFFECT-CRITICAL EVENT RECOVERY

• Visual Cue-Grounded Retrieval extracts event-identifying cues while separating observed facts from hypotheses and withholding a final emotion label.The structured analysis covers people, scene, actions, visible text, distinctive and affective cues, and uncertainties; it supports primary event retrieval and evidence-grounded alternative construction.Source: §E.1 · pp. 16–17
• AffectReveal constructs retrieved-evidence-anchored alternatives that change one affect-critical slot, independently verifies them, then arbitrates using discriminative evidence rather than voting.If evidence does not distinguish interpretations sufficiently, arbitration can retain uncertainty or conflict rather than force a winner.Source: §E.1 · pp. 17, 19, 20
• Event recovery prioritizes event identity, the main character’s role and relationship, and the outcome, while explicitly preserving uncertainty and avoiding a final emotion label.When sufficient material exists, the context is 45–110 English words and must not introduce events absent from the retrieval note.Source: §E.1 · p. 21

E.2 EXPRESSION-DISENTANGLED EVENT VERIFICATION

• Verification extracts non-facial event clues, decomposes internal and external contexts into up to four factual items each, and scores every cross-context pair as 0.0, 0.5, or 1.0.The extracted facts concern actions, posture, objects, environment, interactions, and visible event clues; intermediate scores are disallowed.Source: §E.2 · pp. 21–22

E.3 EVENT-CONDITIONED EMOTION INFERENCE

• Verified context is appended to each baseline’s original classification instruction while the original unmasked visual input remains unchanged.The baseline-specific instruction is retained, and no additional context-rewriting model is used.Source: §E.3 · p. 23