PaperlayerView exact revision on arXiv

Source-linked AI summary

From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding

Jia Li, Yichao He, Yangchen Yu, Qiankun Li, Xinyi Li, Baiyi Ye, Zhenzhen Hu, Richang Hong, Erik Cambria

arXiv:2610.11918v1cs.MMcs.AIcs.HC

TL;DR

Existing multimodal emotion datasets and metrics provide limited supervision and evaluation for the cognitive appraisal behind emotional responses, despite MLLMs’ ability to produce explanations. The paper introduces a perception-to-appraisal dataset, model, and benchmark; CogEmo-MoE achieves strong benchmark results, with cross-domain experiments suggesting transfer beyond its training domain.

  • Problem

    Existing multimodal emotion datasets mainly provide labels or perception-level descriptions, while conventional metrics largely assess final predictions rather than cognition-level reasoning.

    Source: §1 · p. 2

  • Method

    The paper pairs CogEmo-40K’s perception-to-appraisal supervision with CogEmo-MoE’s sparse appraisal-specific adaptation and CogEmo-Bench’s AEQS evaluation.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    On CogEmo-Bench, CogEmo-MoE achieves 70.60 AEQS, 71.43% accuracy, and 57.17% Macro-F1; cross-domain experiments suggest cognition-grounded supervision transfers beyond the training domain.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The results support appraisal reasoning as an intermediate representation beyond surface-level cue-to-label emotion recognition.

    Source: §7 · p. 9

    CogEmo-40K uses existing MER2025 data, and its appraisals are evidence-grounded interpretations rather than definitive accounts of internal psychological states; high-stakes use warrants human oversight.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Recent multimodal large language models (MLLMs) increasingly incorporate explainable reasoning for emotion understanding. However, reasoning based mainly on observable affective cues can reduce emotion understanding to superficial cue-label associations, giving rise to the Clever Hans effect. Such shortcuts become unreliable when affective cues are implicit, conflicting across modalities, linguistically misleading, or obscured by redundant details. In contrast, human emotions are shaped by how individuals interpret and evaluate surrounding events beyond observable cues. Inspired by appraisal theories of emotion, we formulate multimodal emotion understanding as a progression from perception to cognitive appraisal, and introduce a dataset, a model, and a benchmark to support this novel paradigm. CogEmo-40K is a large-scale instruction-tuning dataset constructed through a perception-to-appraisal pipeline to elicit evidence-grounded reasoning across six cognitive appraisal dimensions underlying emotion. CogEmo-MoE is a compact sparse MLLM that introduces interleaved MoE blocks for appraisal-specific adaptation, enabling effective appraisal reasoning at a substantially smaller scale than typical emotion MLLMs. CogEmo-Bench introduces an Appraisal Evidence Quality Score (AEQS) to assess cognitive-affective understanding across six complementary appraisal dimensions, addressing the limitation of conventional emotion metrics that evaluate what emotion is predicted but not why it arises. Extensive experiments show that our paradigm not only leads CogEmo-Bench, but also exhibits strong cross-domain generalization. Our findings suggest that perception-to-appraisal reasoning can move beyond surface-level cue-label associations toward more reliable multimodal emotion understanding and closer cognitive alignment between MLLMs and humans.

1 INTRODUCTION

Emotion understanding requires more than recognizing observable affective cues: people interpret events through appraisals that help explain why emotions arise. The paper introduces a perception-to-appraisal paradigm, supporting dataset, compact model, and benchmark for this reasoning.

1 INTRODUCTION

  • Existing MLLMs can give plausible explanations while still relying on cue–emotion associations, and existing datasets and metrics provide limited supervision or evaluation of cognition-level reasoning.This gap matters when multimodal evidence is ambiguous, conflicting, or context-dependent.Source: §1 · p. 2
  • The paper models cognitive appraisal as an intermediate process across six dimensions: goal congruence, coping potential, accountability, future expectancy, arousal, and action tendency.Appraisal concerns what a situation means to the individual, beyond what observable behavior reveals.Source: §1 · pp. 1–2
  • CogEmo-MoE improves AEQS over GPT-5.2 from 56.30 →70.60 and accuracy over Qwen3-Omni from 63.10% →71.43%.The paper also introduces CogEmo-40K for cognition-grounded supervision and CogEmo-Bench for appraisal reasoning and emotion recognition.Source: §1 · p. 2
  • The perception-to-appraisal paradigm shifts emotion understanding from direct cue-to-label prediction toward reasoning about how perceptual evidence is interpreted before emotion arises.The paper frames superficial cue associations as a Clever Hans effect: seemingly intelligent behavior that relies on hidden cues, biases, or shortcuts.Source: §1 · pp. 1–2

2 COGEMO-40K: COGNITION-GROUNDED MULTIMODAL EMOTION REASONING DATASET

CogEmo-40K provides cognition-grounded multimodal emotion reasoning examples through a perception-to-appraisal pipeline and human-guided alignment. The process separates observable evidence from interpretation, then checks appraisal and emotion consistency against human judgments.

2.1 DATASET OVERVIEW

  • The dataset annotates 115,595 MER2025 source samples and refines approximately 40K instances; its held-out CogEmo-Bench-Eval subset contains over 6,000 human-verified appraisal references across six dimensions.Each instance follows multimodal observation → cognitive appraisal → emotion-state reasoning.Source: §2.1 · p. 3

2.2 TWO-STAGE PERCEPTION-TO-APPRAISAL DATA GENERATION

  • Annotation generation separates perception from appraisal to reduce entanglement between observable descriptions and cognitive interpretation, while discouraging shortcut cue–label associations.The perception stage describes scene, face, speaker audio, and transcript evidence without emotion labels or latent psychological interpretations.Source: §2.2 · p. 3
  • The appraisal stage interprets evidence across six dimensions before deriving an emotion, grounding each appraisal in perception-level descriptions.This creates an explicit trajectory from multimodal observation through cognitive interpretation to emotion state.Source: §2.2 · pp. 3–4

2.3 HUMAN-GUIDED COGNITIVE ALIGNMENT

  • Five researchers rated sampled perceptual descriptions above 4.0 on average for accuracy, completeness, and objectivity on a five-point scale.The sample comprised 1,000 generated descriptions.Source: §2.3 · p. 4
  • Human-guided alignment checks consistency across perception, appraisal, and emotion, and revises uncertain or inconsistent examples rather than discarding them outright.Volunteers compare appraisals with human interpretations across six dimensions and verify the final emotion label; experts review the refinements.Source: §2.3 · p. 4

3 COGEMO-MOE: MODEL DESIGN

CogEmo-MoE combines multimodal projection with sparse experts specialized for cognitive appraisal. Its progressive training moves from multimodal alignment and perception enhancement to cognition-level adaptation.

3.1 ARCHITECTURE OVERVIEW

  • CogEmo-MoE uses Qwen-1.8B with CLIP/CLAP encoders, projecting video and audio features into language space alongside instruction embeddings.Its sparse experts are initialized from corresponding feed-forward network weights and specialized for cognitive appraisal reasoning.Source: §3.1 · p. 4

3.2 SPARSE COGNITIVE EXPERT ADAPTATION

  • Interleaved MoE layers use a lightweight router to select experts from each hidden state, activating only the top-k experts for each input.Each selected expert contributes through its routing weight.Source: §3.2 · p. 5

3.3 PROGRESSIVE PERCEPTION-TO-COGNITION TRAINING

  • Training progresses from projector-only alignment on MERR-Coarse, to joint backbone-projector perception enhancement, then expert-only cognition adaptation using CogEmo-40K.The three-stage training data are summarized in Table 1.Source: §3.3 · p. 5, Table 1 · p. 5

4 COGEMO-BENCH: COGNITION-GROUNDED EMOTION UNDERSTANDING BENCHMARK

CogEmo-Bench evaluates both cognitive appraisal reasoning and conventional emotion recognition. Its AEQS measures alignment between appraisal evidence and human-verified references across six dimensions.

4.1 COGEMO-BENCH OVERVIEW

  • The benchmark compares extracted appraisals with human-verified references using AEQS, while standard classification metrics evaluate final emotion predictions.An Appraisal Extractor normalizes free-form responses into six appraisal dimensions without judging correctness or adding evidence.Source: §4.1 · p. 5

4.2 EVALUATION METRICS

  • AEQS uses fixed-encoder similarity between extracted model evidence and human-verified references to assess appraisal evidence, complementing metrics that evaluate only final predictions.Higher AEQS indicates stronger semantic alignment with human-verified cognitive references.Source: §4.2 · p. 6
  • A one-sided empty evidence pair scores 0, both-empty pairs are excluded, and valid sample scores are averaged across the six appraisal dimensions.The dimension-level sample set includes valid samples for that dimension.Source: §4.2 · p. 6
  • Final emotion predictions are additionally evaluated with Accuracy and Macro-F1.Source: §4.2 · p. 6

5 RESULTS AND OBSERVATIONS

The results show that CogEmo-MoE pairs strong appraisal reasoning with emotion recognition, while analyses highlight the value of structured appraisal profiles and cognition-grounded supervision.

Obs. 1

  • Strong emotion prediction does not ensure cognitive understanding; cognition-oriented supervision improves both appraisal reasoning and emotion discrimination in CogEmo-MoE.VideoChat-Flash predicts emotions strongly despite limited explicit appraisal reasoning, while AffectGPT and Emotion-LLaMA show limited transfer to cognition-level understanding.Source: §5 · p. 6

Obs. 2

  • Cognition-oriented reasoning transfers across emotion spaces and consistently improves recognition, whereas fixed-label strength or multi-candidate scoring may not establish robust judgments.The paper notes that counting a prediction as correct whenever the target appears among multiple generated candidates may overestimate performance.Source: §5 · p. 6

Main results

  • 56.30→70.60 AEQS over GPT-5.2; 63.10%→71.43% accuracy and 52.19%→57.17% Macro-F1 over Qwen3-Omni, with compact-scale balance.CogEmo-MoE achieves a favorable balance between appraisal reasoning and emotion recognition despite its compact model scale.Source: §5 · p. 6, Table 2 · p. 7

Cognitive Language Profiles

  • Emotion categories show distinct joint appraisal profiles: anger features conflict and confrontation, while sadness features loss and withdrawal.Fear emphasizes threat, low coping, high arousal, and escape; anxiety is associated with conflict, uncertain coping, elevated arousal, and avoidance. Figure 4 encodes phrase prevalence, not appraisal intensity.Source: §5 · pp. 7–8, Figure 4 · p. 7

Emotion Alignment of Cognition-Grounded Supervision

  • On shared samples under an eight-way partition, CogEmo-40K yields clearer emotion-label structure and stronger per-class and overall classification than MERCaption+.The comparison uses five-fold cross-validation with the same XLM-RoBERTa classifier after removing explicit emotion-conclusion sentences.Source: Figure 5 · p. 8, §5 · p. 8

MoE ablations

  • Interval placement raises accuracy 64.29%→71.43% over First-Half; four-expert Top-2 routing raises accuracy 65.97%→71.43% and Macro-F1 52.28%→57.17%.Top-2 routing improves both emotion recognition and cognitive appraisal quality; simply increasing experts with Top-1 routing does not improve performance.Source: Table 3 · p. 8, Table 4 · p. 8, §5 · p. 8

6 RELATED WORK

Prior multimodal emotion work often emphasizes labels, observable cues, or plausible explanations, while appraisal theory offers an intermediate account of how events are evaluated before emotion judgments.

6.1 FROM MULTIMODAL PERCEPTION TO EMOTION REASONING

  • Current multimodal emotion supervision and evaluation emphasize final labels, observable cues, or textual plausibility, so fluent explanations alone do not establish reasoning about how emotion arises.The section traces progress from multimodal perception and contextual models to explanation-generating MLLMs and notes that rationales can be unstable and benchmark formats can admit shortcuts.Source: §6.1 · p. 9

6.2 COGNITIVE APPRAISAL AS AN INTERMEDIATE REPRESENTATION

  • CogEmo operationalizes appraisal as an intermediate representation between observation and emotion through dataset supervision, a model of the transition, and a benchmark evaluation.Appraisal theory frames emotion in terms of how individuals evaluate events with respect to goals, coping, responsibility, expected outcomes, and action readiness.Source: §6.2 · p. 9

7 CONCLUSION

The conclusion presents perception-to-appraisal reasoning as an alternative to surface cue-to-label mapping, supported by a dataset, model, benchmark, and reported benchmark and transfer results.

Conclusion

  • CogEmo-MoE achieves 70.60 AEQS, 71.43% accuracy, and 57.17% Macro-F1 on CogEmo-Bench, with analyses showing distinct appraisal patterns and transfer beyond the training domain.The authors interpret these results as supporting appraisal reasoning as an intermediate representation beyond surface-level cue-to-label emotion recognition.Source: §7 · p. 9

AI USE STATEMENT

The statements describe AI-assisted editing, reproducibility materials, ethical cautions, and the organization of appendices.

AI USE STATEMENT

  • AI tools were used to improve manuscript clarity, grammar, and readability; authors reviewed and revised the edits and take responsibility for the final content.Source: AI USE STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

  • The paper reports detailed data, model, and evaluation procedures, with prompts, verification protocols, training settings, and additional experimental details in appendices.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

  • CogEmo-40K uses existing MER2025 data, and its appraisals are evidence-grounded interpretations rather than definitive accounts of internal states; high-stakes use warrants human oversight.The statement also flags potential inherited linguistic, cultural, and domain biases.Source: ETHICS STATEMENT · p. 10

APPENDIX CONTENTS

  • The appendices are organized around limitations and social impact, dataset details, implementation, benchmark details, and additional experimental results.Source: APPENDIX CONTENTS · p. 16

A LIMITATIONS AND SOCIAL IMPACT

The section identifies limits in the appraisal dimensions, dataset coverage, and benchmark evaluation, while describing the data-generation and human-verification pipeline used to build CogEmo-40K.

A LIMITATIONS AND SOCIAL IMPACT

  • The six appraisal dimensions capture important aspects of emotion formation but do not exhaust cognitive emotion processes; dataset coverage and the fixed benchmark evaluator also remain bounded.Model-generated appraisals are evidence-grounded reasoning, not definitive assessments of an individual's psychological state, especially in high-stakes applications.Source: A LIMITATIONS AND SOCIAL IMPACT · p. 16

B.1 SOURCE DATASET: MER2025

  • CogEmo-40K uses unlabeled MER2025 multimodal source samples also used for MERCaption, then creates new cognition-grounded annotations rather than reusing MERCaption's generated descriptions.The annotations undergo multi-round filtering and human-guided alignment.Source: §B.1 · p. 17

B.2 TWO-STAGE DATA GENERATION PROMPTS

  • The two-stage generation pipeline first produces objective, emotion-free perceptual descriptions, then derives appraisal-based emotion interpretations from those descriptions.Stage 1 requires observable scene, facial, and audio information; Stage 2 integrates evidence with all six appraisal dimensions and ends with an emotion label.Source: §B.2 · pp. 17–18

B.3 HUMAN VERIFICATION AND QUALITY FILTERING

  • Human quality control audits perceptual descriptions and verifies appraisal judgments and emotion labels against the original video and audio.Stage 1 average scores for accuracy, completeness, and objectivity all exceed 4.0; Stage 2 directly retains instances when the label agrees with human judgment and at least four of six dimensions are consistent, correcting failed instances before producing approximately 40K final examples.Source: §B.3 · pp. 18–20

B.4 DATASET STATISTICS AND COMPARISON

The supplementary sections characterize CogEmo-40K's annotation length, emotion and appraisal-language distributions, describe model training and evaluation procedures, and report CogEmo-MoE's benchmark comparison.

B.4.1 TEXT LENGTH COMPARISON

  • CogEmo-40K annotations are substantially longer than MERR and MAFW annotations but more concise than MERCaption annotations.The comparison is based on word count; the authors relate the pattern to the dataset's focus on structured appraisal reasoning rather than purely descriptive content.Source: §B.4.1 · pp. 20–21, Figure 6 · p. 20

B.4.2 EMOTION DISTRIBUTION

  • CogEmo-Bench-Eval preserves neutral as the dominant category while assigning relatively larger proportions to fear and disgust, less frequent and often harder-to-distinguish categories.The distribution is intended to preserve natural imbalance while strengthening coverage of difficult, low-frequency categories.Source: Figure 7 · p. 21, §B.4.2 · p. 21

B.4.3 COGNITIVE LANGUAGE PROFILES

  • Appraisal-language profiles show recurring emotion-specific tendencies, including anger with goal conflict and external attribution, and fear with threat, limited coping, and withdrawal.The profiles summarize frequent semantic patterns in the annotations rather than reproducing the raw phrases.Source: §B.4.3 · p. 21, Table 8 · p. 22

C.1 DETAILS OF COGEMO-MOE MODEL

  • CogEmo-MoE is a roughly 3.1B-parameter multimodal model built on Qwen-1.8B with CLIP visual and CLAP audio encoders, trained progressively for alignment, perception, and appraisal specialization.Its training targets move from coarse descriptions to perception-level descriptions and then cognition-level reasoning.Source: §C.1 · p. 23, §C.1.1 · p. 23, Table 9 · p. 23

C.1.1 TRAINING PROMPTS FOR EACH STAGE

  • The three training prompts progress from concise appearance and sound descriptions to detailed emotion-free perception, then six-dimension appraisal reasoning with a final emotion label.The final stage uses the appraisal chain-of-thought format produced by the dataset generation pipeline.Source: §C.1.1 · pp. 23–24

C.2 MODEL SCALE AND PERFORMANCE COMPARISON

  • 14.30, 8.33, and 4.98 points: CogEmo-MoE leads the strongest baselines on AEQS, accuracy, and Macro-F1, respectively, on CogEmo-Bench.The model has 3.1B parameters, while most compared multimodal emotion LLMs use backbones of at least 7B parameters.Source: Table 10 · p. 24, §C.2 · p. 24

D DETAILS OF COGEMO-BENCH

  • CogEmo-Bench is a held-out evaluation set with over 6,000 human-verified appraisal references across six dimensions and the same eight-class emotion space as CogEmo-40K.Models receive supported modalities among video, audio, and subtitles; general-purpose MLLMs are prompted for appraisal reasoning, while emotion-specialized models retain their original prompting protocols.Source: D DETAILS OF COGEMO-BENCH · p. 25, §D.1 · p. 25, §D.2 · p. 26

D.2 EVALUATION PROMPT OF DIFFERENT MODELS

  • A fixed Appraisal Extractor organizes each model's explicitly stated reasoning into six appraisal dimensions and an emotion label, leaving absent dimensions empty without adding evidence.The same extractor and prompt are used across evaluated models.Source: §D.2 · pp. 26–27

D.3 METRIC COMPUTATION DETAILS

The section defines a unified evaluation pipeline for appraisal evidence and emotion prediction, then describes a diagnostic test of how CogEmo-40K reasoning relates to emotion labels.

D.3 METRIC COMPUTATION DETAILS

  • AEQS compares model and human appraisal evidence by semantic similarity across six dimensions, while Accuracy and Macro-F1 evaluate final emotion predictions.A one-sided evidence mismatch scores zero; dimensions empty on both sides are excluded from that dimension's comparison.Source: §D.3 · p. 27
  • All models share the same extractor, embedding model, emotion label space, and metric procedure to support consistent appraisal and emotion-recognition comparisons.The evaluation set contains human-verified references for six appraisal dimensions and final labels in an eight-class emotion space.Source: §D.3 · p. 27

E.1 COGNITIVE REASONING–EMOTION ALIGNMENT

  • CogEmo-40K reasoning representations show strong emotion separability in a five-fold classification diagnostic, while feature-space cluster separability further supports the annotations' quality.The diagnostic uses 11,770 samples from the CogEmo-40K and MERCaption+ intersection; explicit final emotion-conclusion sentences are stripped to reduce label leakage.Source: §E.1 · p. 28, Table 12 · p. 28

E.2 CROSS-DOMAIN GENERALIZATION EXPERIMENTS

The cross-domain comparison finds that CogEmo-40K improves on MERCaption on most benchmarks, while three qualitative cases illustrate appraisal-based distinctions between similar or misleading affective cues and emotions.

E.2 CROSS-DOMAIN GENERALIZATION EXPERIMENTS

  • The controlled comparison uses the same 115,595 source samples, AffectGPT framework, and MERBench protocol to compare appraisal annotations with MERCaption’s affective descriptions across nine datasets.The appraisal annotations in this comparison are generated without human refinement.Source: §E.2 · pp. 28–29
  • CogEmo-40K improves on MERCaption on most benchmarks, including MER2023 by 1.89 points and MELD by 2.66 points, but trails on CMU-MOSEI and CH-SIMS v2.MER2023 rises from 72.12 to 74.01 and MELD from 54.69 to 57.35; CMU-MOSEI scores are 69.75 versus 70.97, and CH-SIMS v2 trails by 0.12 points.Source: Table 14 · p. 29, Table 13 · p. 29, §E.2 · p. 29

E.3 MERBENCH EVALUATION PROTOCOL

  • MERBench covers nine datasets across fine-grained emotion recognition, basic emotion recognition, and sentiment analysis, with shared preprocessing, prompting, and metric computation.Its models produce free-form analyses that are mapped to benchmark label spaces for scoring.Source: §E.3 · p. 30, Table 15 · p. 30

E.4.1 CASE 1: DISTINGUISHING CONTEMPT FROM ANGER

  • For contempt versus anger, CogEmo-MoE identifies external blame and composed coping, while GPT-5.2 reads confrontational cues as anger.The analysis describes external blame and low arousal as distinguishing contempt from anger’s high arousal and goal obstruction.Source: §E.4.1 · p. 31

E.4.2 CASE 2: OVERCOMING SURFACE-LEVEL TEXTUAL MISLEADING

  • For the misleading reassurance wording, CogEmo-MoE interprets ongoing threat and limited coping as fear, whereas GPT-5.2 predicts sadness from the textual cues.The case analysis says appraisal-based descriptions may help distinguish interpretations when textual cues are ambiguous.Source: §E.4.2 · p. 32

E.4.3 CASE 3: CORRECTLY IDENTIFYING NEUTRAL AFFECT

  • For a polite apology in a formal interaction, CogEmo-MoE predicts neutral affect, while GPT-5.2 projects anxiety onto the exchange.The analysis points to coping and low arousal as inconsistent with anxiety in this case.Source: §E.4.3 · p. 33

E.4.4 SUMMARY OF CASE STUDY

  • The three selected cases illustrate appraisal’s discriminative value in subtle distinctions, misleading text, and neutral contexts; the authors note that main experiments validate statistical superiority.The authors describe these cases as illustrative qualitative examples and direct readers to Table 2 for statistical validation.Source: §E.4.4 · p. 34, Table 16 · p. 34
Loading 2610.11918v1…