PaperlayerView exact revision on arXiv

Source-linked AI summary

Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes

Hao Li, Jinye Zhang, Bobo Li, Mong-Li Lee, Wynne Hsu, Zheng Wang, Hao Fei, Min Zhang

arXiv:2610.11410v1cs.AI

TL;DR

Computational affect analysis remains largely observational even though affective science describes emotion as a cognition-shaped process. TRACE formalizes and evaluates affective episodes across their conditions, feelings, regulation, and consequences; current models remain substantially below human performance, and TRACER improves full-chain scores over GPT-5 (thinking).

  • Problem

    Computational affect analysis remains largely observational, while affective science treats emotion as a dynamic process shaped by cognition and social interpretation.

    Source: §1 · p. 1

  • Method

    TRACE formalizes episodes as Condition, Affect, and Effect and evaluates five tasks; TRACER structures reasoning around explicit observations, character appraisals, and established conclusions.

    Source: §1 · p. 2

  • Results

    Current MLLM performance remains substantially below human performance, while TRACER raises the overall full-chain score from 60.79 to 63.59.

    Source: §6.4 · p. 13, §7 · p. 16

  • Takeaways & Limitations

    The shared Blueprint lets different affective reasoning capabilities be evaluated from a unified process perspective.

    Source: §1 · p. 2

    TRACER reasons over a fixed observation record that can omit fine-grained cues later needed for a particular construct.

    Source: §6.7 · p. 16

Abstract

from arXiv · show

Affective computing has progressed from categorical emotion recognition to open-ended affective analysis with large multimodal models. Yet affective science describes emotion as an unfolding process shaped by appraisal, regulation, and social interpretation, which remains underexplored computationally. We propose TRACE, a cognition-oriented framework that formalizes an affective episode through three interrelated stages: Condition, Affect, and Effect, integrating observable cues with cognitive factors such as internal stance and regulation of emotional display. Based on this formulation, TRACE-Bench evaluates multimodal models in real-world social scenes through five tasks spanning grounded affect recognition, regulation decoding, cause reasoning, effect reasoning, and full-chain reconstruction, with 3,746 structured question-answer pairs over 646 videos. A matched human-model comparison reveals a substantial performance gap, while affect-specialized models also generally lag behind general-purpose MLLMs. Model outputs show recurring failures, including treating displayed behavior as genuine feeling and fabricating unsupported events during long-chain generation. We further propose TRACER, a cognition-grounded structured reasoning method that couples each inference with explicit premises from factual observations, cognitive appraisals, and established upstream conclusions, forming a traceable graph of intermediate and target conclusions. TRACER outperforms all evaluated model baselines on each of the five tasks. Project page: https://cogaffc.github.io/TRACE

1 INTRODUCTION

Affective computing has moved toward open-ended analysis, but computational approaches still largely map observable signals to affective outcomes rather than modeling emotion as a cognition-shaped process. TRACE addresses this gap with a Condition–Affect–Effect framework, a five-task benchmark, and structured reasoning grounded in observations and appraisals.

Research gap

  • Existing computational approaches remain largely observational, leaving subjective appraisal, internal drivers, and strategic regulation underexplored.Affective research describes emotion as a dynamic chain in which events are appraised, felt states may be regulated before display, and displays can affect others.Source: §1 · p. 1
  • Prior efforts cover separate parts of the affective chain, often omitting internal stance, felt-to-displayed regulation, or systematic evaluation of downstream consequences.The resulting computational treatments lack a unified cognitive definition of affective episodes.Source: §1 · p. 2

Affective Blueprint

  • The Affective Blueprint connects Condition, Affect, and Effect, including internal drivers, felt and displayed states, regulation, and consequences for the subject or others.Cognition shapes each transition, from interpreting a trigger to regulating display and inferring from what is shown.Source: §1 · p. 2

TRACE-Bench

  • TRACE-Bench evaluates five complementary tasks, from grounded affect recognition and regulation decoding to cause and effect reasoning and full-chain reconstruction.The benchmark contains 3,746 structured question-answer pairs from 646 video clips.Source: §1 · p. 2

TRACER

  • TRACE introduces TRACER to address surface reading by structuring inference around video observations and what those observations mean to each character.Its reasoning uses four appraisal dimensions—relevance, implication, coping potential, and normative significance—and derives task-specific conclusions from explicit premises.Source: §1 · pp. 2–3

2 RELATED WORK

Related work evaluates affect through recognition, extraction, open-ended generation, and social reasoning, but existing targets generally remain localized or separately scored. TRACE-Bench centers these capabilities on one subject-centered affective episode.

2.1 Affective Recognition and Extraction

  • Recognition and extraction benchmarks predict affective states, identify cues, or recover explanatory fragments, but do not jointly evaluate the broader affective episode.These targets typically omit one or more of the external situation, internal driver, felt-to-manifestation relation, and subsequent effects.Source: §2.1 · p. 3

2.2 Generative Affective Understanding

  • Generative affective benchmarks broaden outputs to explanations and multi-scenario reasoning, yet score tasks independently and do not represent regulation of display or its effects on others.TRACE-Bench instead evaluates Condition, State, Manifestation, Regulation, and Effect as components of one episode.Source: §2.2 · p. 3

2.3 Cognitive and Social Reasoning

  • Social and theory-of-mind benchmarks assess intentions, beliefs, relations, and other social judgments, while TRACE-Bench centers those capabilities on a subject-centered affective episode.Its targets include triggering conditions, felt affect, manifestation, possible regulation, and resulting mental, affective, and physical effects.Source: §2.3 · pp. 3–4

3 PRELIMINARIES

The section grounds affect as a cognitive and social process, then formalizes it as a subject-centered Condition–Affect–Effect blueprint.

3 PRELIMINARIES

  • The preliminaries draw on cognitive-science and social-psychology theories to analyze affective processes and formalize them into stages and attributes.Section 3.1 provides the theoretical grounding, and Section 3.2 translates it into the Affective Blueprint.Source: §3 · p. 4

THEORY 1: Cognitive Appraisal Theory [17], [18], [47]

  • Appraisal links affective significance to the relation between external events and a subject’s beliefs, goals, concerns, and coping possibilities.The same event can produce different responses depending on the subject’s internal state.Source: THEORY 1: Cognitive Appraisal Theory… · p. 4

THEORY 2: Emotion Regulation Theory [22], [48]

  • Emotion regulation can reshape outward expression relative to felt affect, so observable behavior is not always a direct readout of feeling.A subject may suppress, amplify, substitute, or fabricate a display in response to social goals, relationship concerns, or situational expectations,.Source: THEORY 2: Emotion Regulation Theory… · p. 4, Figure 2 · p. 4

THEORY 3: Emotions as Social Information (EASI) [23]

  • EASI frames affective displays as social information that can alter others’ beliefs, feelings, and behavior.The described consequences include another person forming a belief, feeling concern, and offering comfort.Source: THEORY 3: Emotions as Social… · p. 4

3.2 The Affective Blueprint

  • The Affective Blueprint connects Condition, Affect, and Effect, combining external circumstances and internal stance with felt and displayed affect and downstream consequences.Condition identifies external reality and internal drivers; Affect includes state and manifestation; Effect records consequences for recipients.Source: §3.2 · p. 5
  • Regulation represents how a subject reshapes felt affect for display through a tactic, goal, and source-to-target relation.The Blueprint distinguishes felt and intended displayed states, including cases where sadness is concealed by a forced smile.Source: §3.2 · p. 5

4 TRACE-BENCH: BENCHMARKING AFFECTIVE CHAIN INFERENCE

TRACE-Bench operationalizes affective-chain inference as five tasks and constructs task-specific questions from verified video annotations. Its dataset contains 3,746 QA pairs from 646 videos, evaluated with field-level scoring and video-level bootstrap comparisons.

4.1 Task Formulation

  • Tasks 1–2 assess within-stage reasoning, Tasks 3–4 reason across stages, and Task 5 reconstructs the full chain.The tasks cover joint recognition of felt state and manifestation, regulation decoding, backward cause reasoning, forward effect reasoning, and complete Blueprint reconstruction.Source: §4.1 · pp. 6–7

4.2 Data Construction

  • Construction screens social-video clips for recognizable affective episodes, then uses human review to correct, complete, and validate candidate chains.Verified chains are converted into eligible task-specific QA pairs using predefined templates and receive final human validation.Source: §4.2 · p. 7
  • 3,746 structured QA pairs from 646 videos form TRACE-Bench, spanning home, workplace, personal, professional, and authority-based interactions.Task 4 includes 286–452 queries for each of the three Effect channels.Source: §4.2 · p. 7, Figure 4 · p. 8

4.3 Evaluation Protocol

  • Categorical fields use set-F1 or exact match, while open-ended descriptions receive semantic-adequacy and redundancy ratings.The redundancy discount is bounded by λ = 0.5.Source: §4.3 · pp. 7–8
  • Field scores are averaged through Blueprint constructs and stages, with equal Condition, Affect, and Effect weights for full-chain reconstruction.Model comparisons use video-level cluster bootstrap with 10,000 resamples and Holm correction against 14 baselines on each task.Source: §4.3 · p. 8

TURED REASONING

TRACER addresses surface readings of visible behavior and factual drift by grounding affective inferences in observations and character-centered appraisals. Its pipeline moves from factual records through appraisal readings to task-conditioned, premise-declared derivations.

TURED REASONING

  • TRACER constrains each inference to premises from factual observations, their affective meanings, and relevant Affective Blueprint constructs.This targets surface readings that collapse felt states and regulated displays, as well as unsupported downstream events in long-chain reasoning.Source: TURED REASONING · p. 8

5.1 Method Overview

  • The three-stage pipeline extracts observations, appraises them from relevant characters’ perspectives, then derives task-specific outputs through premise-grounded reasoning.The five tasks use stage-specific prompts with off-the-shelf MLLMs or LLMs and require no additional training.Source: §5.1 · pp. 8–9, Figure 5 · p. 9

5.2 Factual Observation Extraction

  • The observation stage records temporally ordered, identifiable scene facts before assigning affective meaning, keeping later inferences anchored to an independent factual source.Task context selects and groups relevant observations, while the full record remains available for citations.Source: §5.2 · p. 9

5.3 Cognition-Grounded Affective Analysis

  • Character-centered appraisal turns selected facts into evidence-linked propositions, using four dimensions to interpret their meaning rather than assigning scores or categories.The dimensions cover relevance, implication, coping potential, and normative significance; recipient-side readings are added when the query includes an Effect.Source: §5.3 · p. 9

5.4 Blueprint-Guided Structured Derivation

  • Task-conditioned reasoning topology orders the constructs and dependencies needed for each query, while premise-declared derivation cites observations, appraisals, supplied constructs, or earlier conclusions for each inference.Supplied values remain fixed, and queried answers are extracted from the derived queried nodes.Source: §5.4 · p. 10

6 EXPERIMENTS

The experiments compare multimodal model families with TRACER across TRACE-Bench, test the value of video evidence and structured reasoning, and diagnose task-specific errors. TRACER outperforms every model baseline on all five tasks, while full-chain reconstruction and several component-level judgments remain challenging.

6.1 Experimental Setup

  • The evaluation compares open-source, affect-specialized, and closed-source models, using model-specific video inputs and a shared task-question format.Baseline inputs include available subtitles; only Qwen3-Omni-30B receives audio, while GPT-5 uses eight frames and Qwen3-VL-32B uses a 60-frame budget.Source: §6.1 · p. 10

6.2 Main Results on TRACE-Bench

  • TRACER outperforms every model baseline on all five tasks; its average margin over GPT-5 (thinking) is 5.80 points (95% CI 4.42 to 7.17).The largest gains are in T2 regulation decoding (7.97 points) and T3 cause diagnosis (7.44 points); full-chain reconstruction shows the smallest improvement.Source: §6.2 · pp. 10–11, Table 2 · p. 11

6.3 Evidence Ablation

  • Joint subtitles and frames outperform question-only input on all five tasks for both tested models; neither channel alone matches joint input.Channel preferences vary by task, and increasing GPT-5’s frame count from eight to 16 yields small gains on four tasks but a slight T2 decrease.Source: §6.3 · pp. 11–12, Figure 7 · p. 12

6.4 Within-Task Performance Breakdown

  • For T1, State scores exceed Manifestation scores across six baselines, but predictions still confuse related categories, intensity, and mixed polarity.For T2, tactic prediction often mistakes unregulated cases for active regulation: false-alarm rates range from 76% to 96% across GPT-5 (thinking), Qwen3-VL, and TRACER.Source: §6.4 · p. 12, Table 3 · p. 12, Figure 9 · p. 12
  • Internal Driver scores trail External Reality across models; TRACER leads on both, while its T4 and T5 gains are uneven across channels and constructs.TRACER’s T4 Physical Effect score rises from 46.9 to 58.0 versus GPT-5 (thinking); its T5 full-chain score rises from 60.79 to 63.59, led by Effect, from 47.74 to 53.84.Source: §6.4 · pp. 12–13, Figure 10 · p. 13, Table 5 · p. 13

6.5 Diagnostic Evaluation of Failure Modes

  • In T5, TRACER reduces aggregate fabrication from 11.0% to 4.7%, and supplied appraisal records improve correct State attribution for both tested models.The largest fabrication reduction is in Effect, from 17.3% to 3.3%. Appraisal records raise correct State attribution from 50% to 60% for GPT-5 and from 40% to 51% for Qwen3-VL; for Qwen3-VL, surface reading falls to 2%, while other errors or omissions rise to 47%.Source: §6.5.1 · pp. 13–14, §6.5.2 · p. 14

6.6 Component Ablation

  • Appraisal-based CoT raises the average from 58.87 to 60.61, but PDD improves all five tasks and lifts the average from 58.86 to 64.18.Adding RT alone yields a modest 0.89-point average gain and does not improve every task; appraisal-based CoT still trails RT + PDD by 3.6 points.Source: §6.6 · p. 15, Table 7 · p. 15

6.7 Limitations and Future Work

  • TRACE-Bench does not evaluate dependencies between successive episodes or changes in regulation strategy across episodes; TRACER’s fixed observation record can omit later-relevant fine-grained cues.The paper identifies adaptive evidence acquisition and reduced computational overhead as future directions.Source: §6.7 · pp. 15–16

7 CONCLUSION

TRACE formalizes affective episodes as a cognition-shaped process and evaluates them across five tasks. The conclusion reports a substantial human–model gap and improved performance from TRACER across all five tasks.

7 CONCLUSION

  • TRACE frames affect as Condition–Affect–Effect and benchmarks five tasks; current MLLMs remain below humans, while TRACER improves all five tasks and reduces unsupported T5 content.TRACER’s largest gains are in regulation decoding and cause reasoning.Source: §7 · p. 16

Appendix

  • The appendix provides implementation and experimental details, the scoring protocol, and data-construction and annotation details.Its figures, tables, and equations use the prefix “S”.Source: Appendix · p. 18

A.1 Inference Settings

  • Inference gives evaluated methods the same task instructions and subtitles with frames sampled across each clip; only Qwen3-Omni-30B receives audio.TRACER reuses one observation record per video; T5 derives Condition, Affect, and Effect in separate calls.Source: §A.1 · p. 18

A.2 Observation Extraction

  • Observation records use video frames sampled at 1 fps, up to 256 frames, and timestamped subtitles to describe 8–12 moments per clip.The record excludes emotion labels, causal statements, and selection by relevance.Source: §A.2 · p. 18

A.3 Appraisal Prompt

  • Each appraisal request yields an evaluative proposition for four dimensions, and later derivation selects which propositions to cite as premises.The request includes the observation record, character, and group of items to interpret.Source: §A.3 · p. 19

A.4 Field-Level Results

The appendix extends selected main-paper comparisons and specifies how field and task scores are aggregated.

A.4 Field-Level Results

  • Tables S1–S4 extend selected Section 6.4 comparisons to all evaluated models.The supplied text does not report their score values.Source: §A.4 · p. 19
  • Field scores average over items querying that field, while task scores are aggregated within each item before averaging across items.Failed outputs score zero; asterisks and daggers flag parsing rates below 90% and 50%, respectively.Source: §A.4 · p. 19
  • Parsing rates and means over parsed outputs are reported separately in Appendix B.3.Source: §A.4 · p. 19

A.5 Ablation and Model-Family Comparisons

This appendix describes an item-sampling procedure and a family-normalized heatmap for model comparisons, alongside the scope of two T5 tables.

A.5 Ablation and Model-Family Comparisons

  • The ablation samples 150 items per task, allocating them proportionally across five equal-width GPT-5 (thinking) score bands.Bands span the 0–1 scale from [0, 0.2) through [0.8, 1].Source: §A.5 · pp. 19–20
  • Tables S4 and S5 cover T5 full-chain reconstruction scores and stage-specific fabrication rates, respectively.The supplied captions do not provide the table values.Source: Table S4 · p. 20, Table S5 · p. 20
  • The heatmap double-centers mean scores for three baseline families across tasks, excluding TRACER.Positive values indicate relative strengths; the Mean row reports uncentered family scores.Source: §A.5 · p. 20

A.6.1 Factual Drift

The appendix audits factual drift and surface reading, describes human solvability and judge calibration, and supplements the scoring protocol with field-level criteria.

A.6.1 Factual Drift

  • The audit checks atomic statements in External Reality, Manifestation, and Physical Effect against video evidence, distinguishing supported, misplaced, fabricated, and unverifiable claims.Human reviewers agreed with primary assessments on fabricated-content judgments in 570 of 630 field-slot comparisons (90.5%, κ = 0.59).Source: §A.6.1 · p. 20
  • 6.3 percentage points: GPT-5 (thinking) minus TRACER fabrication-rate difference; smaller differences for +RT and +RT + PDD (w/o App.) were not significant.Unverifiable rates remained similar across configurations.Source: §A.6.1 · pp. 20–21

A.6.2 Surface Reading

  • Human reviewers agreed on 91 of 98 descriptions; for open-source models, reduced surface reading came with more other errors or omissions, not fully correct State attribution.Source: §A.6.2 · p. 21
  • With appraisal records, both models interpret the guest’s laughter as softening disdain rather than expressing amusement.The records connect her preference for a more worthwhile conversation with the social cost of criticizing it.Source: §A.6.2 · p. 22

A.7 Human Solvability and Judge Calibration

  • Two participants independently answered two questions per task from 250 distinct videos, selected using reference annotations and video metadata without consulting model results.Sampling was stratified by task-specific annotation characteristics.Source: §A.7 · p. 22
  • In a blinded sample of 100 open-text fields, the positive mean difference indicates that the automated judge scores below the human rater.Source: §A.7 · pp. 22–23

APPENDIX B SCORING PROTOCOL

  • Appendix B supplements the main-paper metrics with field scores, task aggregation, and treatment of unparsed outputs.Source: Appendix B · p. 23

B.1 Field-Level Scoring

  • Field scoring separates semantic agreement with the reference from irrelevant repetition or filler; answer length itself is not penalized.Semantic adequacy also considers the character, focal episode, and requested information.Source: §B.1 · p. 23

B.2 Task-Level Aggregation

Task scores combine the fields requested by each task, while the reference determines which components apply. The appendix also specifies how regulation outputs and parsing failures affect scoring.

B.2 Task-Level Aggregation

  • Task-level scores combine fields according to requested outputs, including Regulation Evidence in Task 2; Task 5 excludes the given affective object from its State score.The equations instantiate the main paper’s scoring hierarchy when all components are annotated.Source: §B.2 · pp. 23–24
  • Reference-unannotated components are excluded and their remaining weights renormalized, whereas prediction-missing components retain their weight and receive zero.Item scores are averaged within tasks, and the five task means contribute equally to the overall score.Source: §B.2 · p. 24
  • For Task 2, the supplied source state is not re-recognized; without regulation, only tactic is scored, and Regulation Evidence is a task-specific output.For regulated items, evidence receives equal weight with other applicable fields; evidence and goal are omitted when there is no regulation.Source: §B.2 · p. 24
  • A Task 2 target is scored only for Substitution or Fabrication when its reference category differs from the supplied source category.Two predicted categories cannot match a single reference category.Source: §B.2 · p. 24

B.3 Parsing and Score Reporting

  • Unparsable outputs receive zero, and reported scores retain these failures; Table S11 presents parsing rates below 90% and both all-output and parsable-only means.The reported value is bold in Table S11; the table caption states that these bold values are reported in the main paper’s Table 2.Source: §B.3 · p. 24, Table S11 · p. 24

APPENDIX C DATA CONSTRUCTION AND ANNOTATION

The appendix describes human revision and re-annotation procedures, clarifies field judgments, and supplies task-specific prompts and answer formats. The templates progress from grounded affect and regulation through causes, downstream effects, and full-chain reconstruction.

C.1 Annotation and Human Revision

  • The revision analysis pairs 1,125 retained MLLM-proposed chains with raw human records before AI polishing and counts wording or meaning changes as edits.Comparisons are limited to fields present in both records, and formatting and empty-value conventions are standardized before differences are counted.Source: §C.1 · p. 24

C.2 Annotation Consistency

  • Two annotators independently re-annotated 50 chains, with closed-field agreement measured by Cohen’s κ and open-text agreement by strict content-match proportion.The figure caption warns that only κ corrects for chance, so the two agreement measures are not directly comparable.Source: §C.2 · pp. 24–25, Figure S9 · p. 25

C.3 Field Annotation Guidance

  • Intensity depends on the situation, concerns, and response—not merely calm display or control—while mixed polarity requires simultaneous positive and negative feelings.A mismatch between state and manifestation, or a change across moments, does not by itself count as mixed polarity.Source: §C.3 · p. 25
  • Effects include connected utterances or actions responding to the affective episode, even beyond the first response, but exclude reactions to a new event.Figure S8 contrasts a continuing physical effect with a later surprise attributed to a new event rather than the preceding affect.Source: §C.3 · p. 25, Figure S8 · p. 25

C.4 Task and Answer Formats

  • Figure S10 shows task and answer templates whose field names follow the main paper’s Table 1; optional channels and fields appear only when annotated.Source: Figure S10 · p. 26

T1 Grounded Affect Recognition

  • Task 1 asks models to infer internal affective state and observable manifestation from a subject’s scene and observed channels.Its answer format includes state category, fine description, polarity, intensity, and facial, bodily, vocal, and verbal manifestations.Source: T1 Grounded Affect Recognition · p. 26

T2 Regulation Decoding

  • Task 2 supplies the subject’s state and asks whether and how its outward manifestation is regulated, including tactic, goal, target category, and video evidence.Source: T2 Regulation Decoding · p. 26

T3 Affective Cause Reasoning

Loading 2610.11410v1…