PaperlayerView exact revision on arXiv

Source-linked AI summary

LongEmo: Towards Emotion Understanding and Reasoning in Long Videos

Shuo Zhang, Yifan Zhou, Han Wang, Jinsong Zhang, Jingyu Li, Hongbing Li, Zhejun Zhang, Chengyi Zhao, Yuquan Hao, Yitong Liu, Jiyin Li, Ruiqi Tang, Zixuan Lin, Yi Luo, Xurui Zhang, Ronghao Chen, Huacan Wang, Lei Li

arXiv:2609.40079v1cs.CVcs.AI

TL;DR

Affective reasoning in existing methods is largely confined to short clips, while emotions in extended narratives depend on past experiences and unfolding events. The paper introduces LongEmoBench and LongEmo, which uses event memories and query-relevant retrieval for long-range reasoning. Across evaluations, existing methods struggle with long-video emotion reasoning, while LongEmo achieves state-of-the-art performance.

  • Problem

    Existing affective benchmarks and methods largely focus on short clips, leaving long-range reasoning about emotions shaped by past experiences and unfolding events underexamined.

    Source: §1 · p. 1

  • Method

    LongEmoBench evaluates scene- and episode-level emotion reasoning, while LongEmo constructs an Event Memory Graph and retrieves relevant event streams for progressive multimodal reasoning.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Across 17 evaluated approaches, standard MLLMs and long-video methods struggle with complex emotion reasoning, while LongEmo achieves state-of-the-art performance and improves 14.78 points over the strongest baseline.

    Source: §1 · p. 2, §5.1 · p. 9

  • Takeaways & Limitations

    The findings support integrating local multimodal evidence with long-range context through structured event memories and relational retrieval for long-range emotional reasoning.

    Source: §6 · p. 10

    LongEmoBench may underrepresent subtle and culturally diverse emotional behavior, and its emotion annotations can admit multiple reasonable interpretations.

    Source: §6 · p. 9

Abstract

from arXiv · show

While recent Multimodal Large Language Models (MLLMs) have shown promise in affective computing, their reasoning capabilities are largely confined to short video clips with limited interactions. However, real-world emotions are not merely isolated instantaneous reactions but dynamic and cumulative processes deeply shaped by past experiences and ongoing events. To bridge this gap, we introduce LongEmoBench, a benchmark dedicated to emotion understanding and reasoning in long videos. It assesses progressive capabilities scaling from continuous scene interactions to complex episodic developments. Furthermore, we propose LongEmo, a novel memory-augmented agentic framework designed to tackle the immense challenges of long-range affective reasoning. LongEmo processes continuous video streams to construct an Event Memory Graph, explicitly modeling long-range dependencies and capturing emotional dynamics across discrete events. Given a question, the agent retrieves a query-relevant event stream from the graph, iteratively integrating multimodal memories and relational dependencies to deduce the final answer. Extensive evaluations of 17 representative methods reveal that they struggle significantly with emotion understanding and reasoning in long videos. In contrast, LongEmo achieves state-of-the-art performance, demonstrating the efficacy of its event-centric memory architecture.

1 INTRODUCTION

The introduction identifies a gap in long-range emotion reasoning: existing affective benchmarks and methods largely focus on short clips, while general long-video benchmarks emphasize physical events and plot facts. It introduces LongEmoBench and LongEmo to evaluate and support emotion reasoning across extended narratives.

1 INTRODUCTION

  • Existing affective methods and benchmarks largely focus on short, isolated clips, although emotions develop over time and depend on past experiences and unfolding events.General long-video benchmarks scale to hour-long inputs but primarily assess physical realities and plot facts rather than characters’ internal states.Source: §1 · p. 1
  • LongEmoBench evaluates emotion understanding at scene and episode levels, with 1,975 QA pairs across approximately 70 hours of video.Scene-level tasks assess affective comprehension within limited context; episode-level tasks require synthesizing temporally distant evidence for tracking and reasoning.Source: §1 · p. 2
  • LongEmo builds an Event Memory Graph of event-based emotional states and evidence, then retrieves a query-relevant event stream for progressive multimodal reasoning.The graph links memories according to temporal and causal logic, and the reasoning agent integrates within-event memories with relations across events.Source: §1 · p. 2
  • LongEmo achieves state-of-the-art performance, while standard MLLMs and general long-video methods struggle with complex emotion reasoning and long-range emotional dynamics.The authors report that LongEmo's performance supports the efficacy of its event-centric memory architecture.Source: §1 · p. 2

2 RELATED WORK

Prior affective benchmarks have expanded from emotion classification toward descriptions, explanations, transitions, and interactive scenarios, but generally remain limited to brief scenes. LongEmoBench and LongEmo target the long-video setting with episode-level evaluation and event-centric memory.

2 RELATED WORK

  • Affective benchmarks cover increasingly complex tasks, including emotion transitions, multi-turn and multi-party dialogue, and robustness to emotional hallucinations, but their video inputs remain brief.The described progression includes closed-set classification, open-vocabulary descriptions, and causal explanations.Source: §2 · p. 2
  • LongEmoBench extends affective evaluation to long videos, connecting fine-grained audiovisual cues with extended temporal contexts.Its overview distinguishes localized G1 scene-level interactions from G2 episode-level long-range emotional reasoning.Source: §2 · p. 3, Figure 1 · p. 3
  • Long-video methods use memory and active exploration, token compression, or streaming, but may disrupt emotional trajectories, discard fine-grained cues, or lose distant anchors.LongEmo is presented as an event-centric memory method designed to preserve continuous emotional dynamics.Source: §2 · p. 3

3 LONGEMOBENCH

LongEmoBench evaluates emotion understanding and reasoning across scene and episode contexts, using a hierarchical task taxonomy, human-in-the-loop construction, and task-specific scoring protocols.

3.1 TASK TAXONOMY

  • The G1 scene-level tier covers five interdependent tasks, from identifying contextual emotion to tracking transitions and trajectories, explaining causes, and predicting interpersonal influence.The tasks address emotional states, temporal change, retrospective motives and triggers, and prospective reactions.Source: §3.1 · p. 4
  • The G2 episode-level tier evaluates three tasks that connect distant cues: intensity comparison, prolonged emotional trajectories, and reasoning about outcomes, motives, or intentions.These tasks span full episodes or untrimmed videos and assess overarching emotional arcs.Source: §3.1 · p. 4

3.2 DATASET CONSTRUCTION AND STATISTICS

  • Annotators select long-form videos with interpersonal dynamics and emotional shifts, create fine-grained audiovisual G1 annotations, then connect distant events to form G2 queries.Specific-state tasks use the 26-class EMOTIC taxonomy; explanatory and continuous tasks use natural-language descriptions.Source: §3.2 · p. 4
  • LongEmoBench contains 1,354 videos totaling 69.8 hours and 1,975 queries; its G2 videos average 19.5 minutes and span varied real-world formats.The benchmark reports distributions for video duration, source domains, and eight task types.Source: §3.2 · p. 5

3.3 EVALUATION PROTOCOL

  • The protocol scores three categorical tasks with sample-averaged F1 and uses a rubric-based LLM judge for the remaining tasks, with reported human-LLM correlation of Pearson’s r = 0.872.Trajectory responses receive holistic scores from 0 to 4, while causal explanations receive scores from 0 to 3.Source: Label-based Evaluation · p. 5, Rubric-based Evaluation · p. 5

4 OUR METHOD

LongEmo models affective dynamics across discrete events by constructing event memories and linking them in a graph. For each question, it retrieves an ordered event stream and progressively inspects relevant evidence to reason over long-range video context.

4 OUR METHOD

  • LongEmo constructs event-centric memories from long video streams, then retrieves a temporally ordered event stream for progressive emotion reasoning.The framework has two stages: event-centric memory construction, followed by event-stream retrieval and reasoning.Source: §4 · p. 5

4.1 EVENT-CENTRIC MEMORY CONSTRUCTION

  • Each event node combines a textual summary, participant-level emotion records, and supporting multimodal evidence.The retained cues include facial expressions, vocal prosody, and dialogue, which can be reassessed under broader context.Source: §4.1 · p. 5
  • Sequential segment processing uses an entity registry and recent memories to create or update event nodes, while updates append transitions or retrospectively correct misread emotions.Appending preserves genuine emotional progression; retrospective correction revises an earlier interpretation when later context invalidates it.Source: §4.1 · pp. 5–6
  • The graph links events to relevant preceding events by combining query relevance with character continuity, then verifies candidate dependencies.This targets historical circumstances behind an event while filtering spurious dependencies.Source: §4.1 · p. 6

4.2 EVENT-STREAM RETRIEVAL AND REASONING

  • For a question, LongEmo retrieves seed events, expands through graph relations, and orders the resulting stream chronologically before selectively inspecting pivotal audiovisual evidence.The stream supplies lightweight global context; targeted multimodal inspection integrates local cues with the narrative trajectory to answer the query.Source: §4.2 · p. 7

5 EXPERIMENTS

Across LongEmoBench experiments, existing affective and long-video methods struggle with extended emotion reasoning, while LongEmo achieves the strongest reported result. Ablations and further analyses highlight the value of event-level context and the difficulty of tracking complex trajectories.

5 EXPERIMENTS

  • The G1 evaluation includes general-purpose and affective MLLMs alongside long-video architectures; G2 also evaluates agentic systems.The listed G1 models include Qwen3-VL-8B, Gemini-3-Flash, AffectGPT, LongVU, and VideoChat3; G2 adds VideoHV, LongVideoAgent, M3-Agent, and WorldMM.Source: §5 · pp. 7–8

5.1 MAIN RESULTS

  • AffectGPT and Emotion-LLaMA score 15.32% and 15.03%, trailing Qwen3-Omni-30B; the authors attribute this to specialization for static or short-clip recognition.The authors argue that this specialization compromises broad contextual reasoning for complex emotion reasoning in dynamic scenes.Source: §5.1 · p. 8
  • Emotion Cause is generally easier than continuous reasoning: Gemini-3-Flash scores 87.58% on Emotion Cause but 59.07% on Contextual Emotion and 57.89% on Emotion Transition.The authors associate the gap with difficulty capturing holistic emotional states and tracking changes across the full emotion lifecycle.Source: §5.1 · p. 8
  • General-scenario long-video models also struggle on G1 emotion tasks: leading VideoChat3-4B achieves 39.17%, with temporal compression and frame subsampling cited as discarding fine-grained affective cues.These architectures are optimized to retain high-level semantic actions rather than fleeting affective details.Source: §5.1 · p. 8
  • On extended G2 videos, long-video architectures score around 20%; the authors say these methods and agentic sampling fail to preserve continuous, fine-grained temporal cues.The G2 comparison includes long-video architectures and agentic systems, which exhibit the same stated limitation.Source: §5.1 · p. 8
  • LongEmo improves 14.78 points over the strongest baseline on G2, using an event-centric memory graph to anchor emotional shifts to narrative events.Table 3 reports G2 task results as percentages and defines Overall as the weighted average.Source: Table 3 · p. 8, §5.1 · p. 9

5.2 ABLATION STUDY

  • Removing event memory lowers Overall from 60.24% to 51.64%; removing event-stream retrieval lowers it to 57.62%, with ET at 54.41% and ER at 72.68%.The authors interpret these ablations as evidence for coherent event memories and relational context when tracking emotional evolution.Source: Figure 4 · p. 8, §5.2 · p. 9

5.3 MORE ANALYSIS

  • As trajectory stages increase, all methods become less able to capture complete trajectories; LongEmo errors most often involve evidence coverage and answer synthesis.The authors connect complexity to intermediate states being diluted or overwritten, and identify evidence acquisition and coverage (EC) and answer synthesis and compression (AS) as the most common failure types.Source: §5.3 · p. 9, Figure 5 · p. 9, Figure 6 · p. 9

6 CONCLUSION

The conclusion presents LongEmoBench and LongEmo as tools for studying emotion across scenes and episodes, and supports integrating local multimodal evidence with long-range context through structured event memories and relational retrieval.

6 CONCLUSION

  • LongEmoBench may underrepresent subtle and culturally diverse emotional behavior, and its emotion annotations can admit multiple reasonable interpretations.The authors caution that emotional predictions are not objective measurements of internal states or grounds for high-stakes decisions.Source: §6 · p. 9
  • The work may inform research on context-sensitive video systems, while downstream applications should be evaluated in their intended contexts with fairness, privacy, and potential misuse considered.Source: §6 · p. 9, ETHICS STATEMENT · p. 10
  • The authors recommend expanding data and task diversity, developing uncertainty-aware memory and retrieval, and incorporating diverse human judgments to improve reliability and generalization.These are identified as future-work directions.Source: §6 · p. 10

A DATASET EXAMPLES

The appendix illustrates LongEmoBench’s scene- and episode-level tasks, their answer formats, and the shared annotation vocabulary. Its video examples emphasize emotional evidence and developments distributed across interactions and time.

A DATASET EXAMPLES

  • LongEmoBench examples span five scene-level tasks and three episode-level tasks, illustrating the emotional evidence and answer forms each requires.The appendix also describes the shared format for questions and reference annotations.Source: A DATASET EXAMPLES · p. 15

A.1 SCENE-LEVEL EXAMPLES

  • Scene-level examples distinguish persistent or shared emotions, before-and-after transitions, ordered trajectories, emotion causes, and responses to specified influences.The tasks require integrating sustained states, event boundaries, or ordered emotional developments within a scene.Source: §A.1 · p. 15, Figure 7 · p. 15

A.2 EPISODE-LEVEL EXAMPLES

  • Episode-level examples ask models to compare intensity across people or moments, preserve intermediate trajectory stages, and reason across separated events for definite answers, causes, or motives.Trajectory examples preserve emotional order, persistence, and turning points rather than reducing a story to its beginning and end.Source: §A.2 · pp. 16, 18, Figure 8 · p. 16, Figure 9 · p. 17

A.3 DATA FORMAT

  • Each question is stored as a JSON record with ten top-level fields, including its source, granularity, task type, question, answer, optional details and rubric, and subtitles.The examples show structured answer details for trajectory stages and explanatory factors.Source: §A.3 · pp. 19–20, Table 4 · p. 19

B EMOTIC EMOTION CATEGORIES

  • Contextual Emotion, Emotion Transition, and Emotion Influence use the 26-category EMOTIC vocabulary, which permits co-occurring labels and standardizes evaluation targets.The vocabulary includes fine-grained states such as anticipation, engagement, and sympathy.Source: B EMOTIC EMOTION CATEGORIES · p. 22

C.1 VIDEO SOURCES

  • Video selection prioritizes rich emotional interactions, coherent narrative context, and audiovisual evidence that supports questions about trajectories, intensity comparisons, and reasoning across events.Materials include sitcoms and online interviews, reality shows, and gameplay.Source: §C.1 · pp. 22–23

C.2 G1 ANNOTATION

The annotation procedures define scene- and episode-level questions around emotion evidence, temporal scope, and answer requirements. Quality control uses independent reviews and removes items without reviewer agreement.

C.2 G1 ANNOTATION

  • For G1, model-assisted drafts of five task types are checked and revised by human annotators against the video, task assignment, question scope, and reference answer.Guidelines distinguish persistent emotion, category-changing transitions, chronological trajectories, supported causes, and actual responses to specified influences.Source: §C.2 · p. 23

C.3 G2 ANNOTATION

  • G2 questions are authored from scratch after annotators watch the complete video, with each question scoped to evidence distributed across time and a reference answer capturing the required comparison, trajectory, or reasoning.Annotators are instructed to use audiovisual evidence beyond subtitles where relevant and avoid unsupported psychological assumptions.Source: §C.3 · p. 25, Table 5 · p. 25

C.4 QUALITY CONTROL

  • Eight experienced AI practitioners review items after five hours of training; disagreements trigger re-annotation and renewed independent review, and items are retained only when both reviewers agree.Initial review agreement was 84.86% for G1 and 71.25% for G2; nominal Krippendorff’s α values were 0.133 and 0.118, respectively.Source: §C.4 · pp. 25–26

D EVALUATION PROTOCOL

The evaluation protocol combines set-based scoring for selected G1 emotion-label tasks with rubric-based LLM judging for the remaining tasks. Rubrics assess task-specific correctness, trajectory reconstruction, or causal explanations.

D.1 LABEL-BASED EVALUATION

  • G1 Contextual Emotion and Emotion Influence use F1 on normalized, deduplicated reference and predicted label sets; Emotion Transition scores before and after sets separately and averages them.The two transition sides are equally weighted, and providing only one side yields half that side’s F1 as the question score.Source: §D.1 · p. 26

D.2 RUBRIC-BASED EVALUATION

  • DeepSeek-V4-Flash serves as the LLM judge for remaining G1 and G2 tasks, using rubrics for result correctness, trajectory reconstruction, and causal explanation.The same rubric family is used across granularities wherever applicable.Source: §D.2 · p. 26
  • The protocol stores complete reference answers and optional structured details to let the judge assess correctness and completeness while accepting semantically equivalent formulations.Trajectory stages and explanatory factors can be represented as structured items; these items guide one holistic judgment.Source: §D.2 · pp. 26–27

D.2.1 SCORING RUBRICS

D.2.2 JUDGE MODEL PROMPT

  • The judge prompt directs evaluation against the question, reference answer, optional answer details, and scoring rubric, with one overall score based on correctness and completeness.It instructs the judge to accept semantically equivalent wording and not follow instructions embedded in the model answer.Source: §D.2.2 · p. 28

E IMPLEMENTATION DETAILS

This section specifies LongEmo’s model configuration, illustrates how its event memories and graph support a food-tasting comparison, and motivates temporal context as evidence for judging and tracing emotion.

E IMPLEMENTATION DETAILS

  • LongEmo uses Gemini-3.8-Flash for audiovisual perception, GPT-6 for planning and answer generation, and text-embedding-3-large for dense retrieval.Table 9 summarizes these configurations alongside those of agent-based baselines.Source: E IMPLEMENTATION DETAILS · p. 29, Table 9 · p. 29

F CASE STUDY

  • In the food-tasting case, E12 links a shift from skepticism about a Scotch egg to enjoyment with audiovisual evidence of chewing, nodding, and approval.Figure 14 shows the event memory; Figure 15 shows its local graph context and a chronological stream of five retrieved food-reaction memories.Source: F CASE STUDY · pp. 29–30, Figure 14 · p. 29, Figure 15 · p. 30
  • The model selects Party Rings biscuits, but the reference is fried chicken because retrieved memories do not establish the strongest enjoyment across the video.The example points to broad candidate coverage and accurate interpretation of local reactions as requirements for intensity comparison.Source: F CASE STUDY · p. 30

G.1 CONTEXT AND TEMPORAL DYNAMICS OF EMOTION

  • Earlier interactions can help resolve or revise judgments of a current emotion when immediate expression is ambiguous, suppressed, or potentially misleading.Contextual evidence should be combined with local expression rather than allowed to override it unconditionally.Source: §G.1 · p. 31
  • Tracking emotional change over time requires distinguishing a momentary judgment from a trajectory of persistence, change, or return to baseline.Changes in outward behavior may not align with changes in internal experience, making continuous temporal evidence relevant.Source: §G.1 · p. 31

G.2 IMPLICATIONS FOR BENCHMARK DESIGN

The benchmark asks whether models can use temporally distributed context to judge emotional states and reason about their development and causes. Its design tests whether answers depend on relevant evidence across time, not merely on isolated recognition or longer inputs.

G.2 IMPLICATIONS FOR BENCHMARK DESIGN

  • The central question is whether multimodal models can connect current emotional evidence to temporally distant information to determine states, development, and underlying causes.Recognizing isolated expressions alone does not demonstrate this capability.Source: §G.2 · pp. 31–32
  • LongEmoBench separates the answer a question requests from the supporting evidence needed to deduce it, allowing even categorical judgments to test contextual reasoning.The benchmark does not require a lengthy textual explanation as proof of that reasoning.Source: §G.2 · p. 32
  • Clip-level questions test bounded contextual states, transitions, and causes, while Episode-level questions require evidence from multiple distributed video segments.Episode-level tasks cover intensity comparison, emotional trajectories, and reasoning about attitudes or motives; questions answerable from one local segment are excluded.Source: §G.2 · p. 32
  • Across tasks, distributed information must contribute directly to an emotional judgment; factual retrieval and unsupported psychological explanations do not satisfy the benchmark’s design requirements.Long-range context must supply necessary evidence rather than merely increase input length.Source: §G.2 · p. 32

H LIMITATIONS AND ETHICAL CONSIDERATIONS

The paper bounds its claims by noting that video-based emotion judgments are interpretive and that the benchmark’s sources and task set cover only part of human affect. It also cautions against treating inferred emotions, motives, or intentions as authoritative judgments, especially in consequential settings.

H LIMITATIONS AND ETHICAL CONSIDERATIONS

  • LongEmoBench does not exhaust human affect: emotions may be ambiguous, mixed, culturally dependent, or only partially observable from behavior.Some questions admit multiple reasonable interpretations, so the benchmark assesses evidential support rather than directly observable ground truth.Source: H LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 32
  • The video collection favors narrative, open-domain settings with visible or narratively recoverable affect, potentially underrepresenting subtle, low-expressivity, culturally diverse, or non-narrative behavior.Its predefined Scene-Level and Episode-Level capabilities are not a complete measure of emotional intelligence.Source: H LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 32
  • LongEmo and LongEmoBench are intended for research, not high-stakes psychological assessment, diagnosis, surveillance, employment screening, or law enforcement.Predictions about internal states, motives, and intentions from behavior and context should not be treated as objective measurements.Source: H LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 32
  • Temporal co-occurrence or behavioral similarity alone cannot establish psychological causality, and generated explanations may still over-attribute unobservable motives or intentions.Downstream applications should preserve uncertainty and distinguish audiovisual evidence from inferred interpretations.Source: H LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 33
  • The authors frame evidence grounding, uncertainty awareness, and separation of observation from psychological inference as requirements for future affect-aware systems, not authoritative judgments about people.Human-centered videos may contain identifiable individuals and sensitive interactions, while source videos and pretrained models may carry demographic, cultural, and representational biases.Source: H LIMITATIONS AND ETHICAL CONSIDERATIONS · p. 33
Loading 2609.40079v1…