PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning

Cheng Ye, Weidong Chen, Zhaobo Qi, Beier Zhu, Zhendong Mao

arXiv:2609.36782v1cs.CV

TL;DR

MLLMs have difficulty distinguishing semantically close emotions that depend on fine-grained visual evidence. DAN combines hierarchical scene- and object-level reasoning with contrastive pruning at inference time; it reports improved benchmark performance, including a +10.47% emotion accuracy improvement on WebEmo25 with Qwen3-VL-8B-Instruct.

  • Problem

    MLLMs struggle to reliably distinguish semantically proximal emotions using fine-grained visual evidence, which can involve subtle expressions, postures, and interactions.

    Source: §1 · pp. 1–3

  • Method

    DAN is a training-free inference-time framework that combines HERC’s scene- and object-level cues with soft-gated reasoning and CDVP’s discriminative visual pruning for ambiguous candidates.

    Source: §1 · pp. 2–3

  • Results

    +10.47% emotion accuracy improvement with Qwen3-VL-8B-Instruct on WebEmo25, which contains 25 fine-grained emotion categories.

    Source: §1 · p. 3

  • Takeaways & Limitations

    DAN improves emotion-recognition accuracy on several benchmarks without additional training or annotation.

    Source: §1 · p. 3, §4 · p. 10

Abstract

from arXiv · show

While multimodal large language models (MLLMs) have demonstrated exceptional capabilities in objective understanding tasks, their performance in affective reasoning still falls significantly short of human standards. We attribute it to a central capability gap: MLLMs are difficult to reliably distinguish semantically proximal emotions based on fine-grained visual evidence, which could be decoupled as two limitations: 1) Insufficient Attribution. The global reasoning paradigm of conventional MLLMs severely dilutes fine-grained emotion cues, where subtle emotional states are usually implicitly encoded, thereby generating emotional misjudgments in complex scenarios. 2) Insufficient Discrimination. Existing methods could only identify regions generally associated with emotions, which fails to distinguish discriminative regions between semantically similar emotions, leading to ambiguous emotion judgements. To overcome these limitations, we present a training-free inference-time optimization framework, named Decoding Affective Nuances (DAN). Specifically, we propose a Hierarchical Emotional Reasoning Chain (HERC) that enhances the insufficient attribution by harmonizing fine-grained scene/object-level cues and performing a soft-gated reasoning. Furthermore, to discriminate between semantically proximal emotions, we design a Contrastive Discriminative Visual Pruning (CDVP), which isolates discriminative visual tokens to reason the final emotion category by computing the absolute discrepancy between the attention distributions of similar emotions. Performances on several benchmarks demonstrate that DAN significantly improves discrimination for affective nuances without consuming additional training resources, especially achieving +10.47% improvements with Qwen3-VL-8B-Instruct on WebEmo25 dataset that contains 25 fine-grained emotion categories.

1 INTRODUCTION

MLLMs struggle to distinguish semantically close emotions that depend on subtle visual cues. DAN combines hierarchical cue reasoning with contrastive visual pruning, improving benchmark performance without additional training or annotations.

1 INTRODUCTION

  • MLLMs struggle to distinguish semantically proximal emotions when small visual differences in expressions, postures, and interactions determine the label.The introduction characterizes this gap through insufficient attribution and insufficient discrimination.Source: §1 · pp. 1, 3
  • Global visual processing can dilute subtle affective cues, while existing pruning methods may find emotion-related regions without distinguishing which candidate each region supports.The introduction reports that over 59% of baseline failure cases fall in low confidence-variance intervals, with errors concentrated among semantically proximal emotions.Source: §1 · p. 2, Figure 1 · p. 2
  • DAN is a training-free inference-time framework that uses HERC for scene- and object-level cues and soft-gated reasoning, then CDVP for discriminative visual-token pruning and final emotion reselection.CDVP compares candidate-specific attention maps for ambiguous emotions.Source: §1 · pp. 2–3
  • Experiments on several public benchmarks report that DAN outperforms state-of-the-art methods without additional training or annotations.The paper presents the approach as a resource-efficient framework for emotion recognition.Source: §1 · p. 3
  • +10.47% emotion accuracy improvement with Qwen3-VL-8B-Instruct on WebEmo25 demonstrates DAN’s benchmark gains.WebEmo25 contains 25 fine-grained emotion categories.Source: §1 · p. 3

2 METHODOLOGY

DAN processes images and prompts through a preliminary emotion-ranking stage followed by conditional discriminative reselection. HERC combines coarse-to-fine affective cues with soft-gated reasoning, while CDVP applies to ambiguous samples.

2.1 PRELIMINARY

  • DAN takes an image and textual prompt to produce an emotion description, with HERC generating preliminary rankings and CDVP selecting ambiguous cases for discriminative reselection.The two components form the framework’s successive inference stages.Source: §2.1 · p. 3

2.2 HIERARCHICAL EMOTION REASONING CHAIN

  • HERC mines scene-level and object-level affective clues, then combines coarse polarity and fine-grained emotion scores through adaptive soft-gated reasoning.Using soft polarity scores rather than hard intermediate labels is intended to mitigate error propagation and provide a stable preliminary distribution for pruning.Source: §2.2 · pp. 3–4

2.3 CONTRASTIVE DISCRIMINATIVE VISUAL PRUNING

  • CDVP selects a bounded set of top-ranked emotions whose scores fall within a variance threshold of the top score, sending ambiguous samples to further discriminative reselection.Samples with a singleton ambiguous set retain HERC’s highest-scoring emotion; the truncation threshold k balances computational overhead.Source: §2.3 · pp. 4–5

1 Initialize:S∗←s1

DAN uses contrastive prompts to isolate evidence distinguishing ambiguous emotions, then prunes nondiscriminative visual tokens before final emotion selection.

1 Initialize:S∗←s1

  • Contrastive prompts make the model explain visual evidence for one ambiguous emotion over its competitors, shifting attention toward discriminative regions.The process begins by isolating samples with significant emotional ambiguity.Source: §1 · p. 5
  • Visual-token importance is scored by mean absolute attention discrepancy between each ambiguous emotion and its competitors.Tokens that distinguish a category from the others receive higher scores.Source: §1 · p. 5
  • The method drops low-ranking tokens from the differential attention map, then uses the pruned image and a discriminative prompt to select the final emotion.The pruning mask guides the model toward discriminative visual regions.Source: §1 · p. 5
  • Contrastive discriminative visual pruning focuses the model on regions that can distinguish ambiguous emotions and may correct initial misjudgments.Source: §1 · p. 6

3 EXPERIMENT

DAN improves average emotion-recognition accuracy over zero-shot, Zero-shot-CoT, and SEPM baselines across four public benchmarks. Ablations and visualizations examine its components, while supporting complementary cue mining, soft-gated reasoning, and contrastive pruning.

3.1 EXPERIMENTAL SETUP

  • Experiments use EmoSet, WebEmo, Emotion6, and Abstract, with emotion accuracy (Acc) as the metric; four MLLMs are evaluated.The benchmarks span basic-emotion labels and WebEmo’s 25-category fine-grained setting.Source: §3.1 · p. 6

3.2 MAIN COMPARISON

  • +10.7%/+11.4%/+17.5%/+47.4% are DAN’s average-accuracy gains over the four zero-shot MLLMs on WebEmo25; gains over SEPM are +3.8%/+5.0%/+6.2%/+6.6%.The paper also reports average-accuracy gains over Zero-shot of +8.8%/+13.8%/+10.0%/+14.4% and over Zero-shot-CoT of +8.7%/+11.6%/+11.5%/+11.5%, in the same four-model order.Source: §3.2 · pp. 6–7

3.3 ABLATION STUDY

  • Scene- and object-level cues have dataset-dependent effects, but their integration improves consistently across scenarios; CDVP consistently improves accuracy across datasets.Scene-level cues can hurt on Emotion6 but help on Abstract8, where emotions are often implicit in global environments.Source: §3.3 · p. 7, Table 2 · p. 7
  • Soft gating avoids direct label removal or ungated fine-grained prediction: cascaded score products mitigate error accumulation and preserve a smooth distribution for CDVP.No-gating is vulnerable to local visual noise, while hard-gating can discard the correct answer when polarity is ambiguous.Source: §3.3 · p. 7, Table 3 · p. 7
  • Last-3 Average achieves the best performance among four attention-map strategies; using only the last layer loses semantic information and reduces inference stability.The strategies are Global Average, First-3 Average, Last Layer, and Last-3 Average.Source: Table 4 · p. 7, §3.3 · p. 8
  • CDVP gains come from contrastive prompting and discriminative pruning, not merely reranking or narrowing choices; contrastive-related pruning performs best across datasets.The strategy comparison finds random pruning can remove emotion-related tokens, query-related pruning can retain irrelevant redundancy, and FoE pruning misses subtle distinctions between similar emotions.Source: §3.3 · p. 8, Table 5 · p. 8, Table 6 · p. 8
  • Performance first rises and then falls as α or β increases; the selected settings are α = 0.125 and β = 0.3.Excessive α admits low-confidence emotions treated as noise, while excessive β can remove critical visual cues.Source: §3.3 · pp. 8–9, Figure 3 · p. 8

3.4 DOES DAN ALLEVIATE INSUFFICIENT ATTRIBUTION AND DISCRIMINATION?

  • HERC focuses attention on fine-grained emotion-related regions, while CDVP distinguishes ‘awe’ from ‘amusement’ using their contrasting visual attention and selects ‘amusement’.In the example, ‘awe’ attends mainly to sky and light, whereas ‘amusement’ attends more to the ferris wheel and playground.Source: §3.4 · p. 9

4 CONCLUSION

DAN combines hierarchical emotion reasoning with contrastive visual pruning to improve MLLM emotion understanding during inference. The paper reports accuracy improvements without additional training or annotation.

4 CONCLUSION

  • DAN combines HERC’s coarse-to-fine clue mining and soft-gated reasoning with CDVP’s contrastive prompting, differential pruning, and final emotion reselection.CDVP selects ambiguous candidates and focuses the MLLM on discriminative visual regions associated with them.Source: §4 · pp. 9–10
  • The framework targets inference-time emotion understanding, particularly ambiguous scenarios involving semantically proximal emotions.Source: §4 · p. 9
  • DAN improves emotion-recognition accuracy without additional training or annotation, with experiments across several benchmarks supporting the framework and its components.Source: §4 · p. 10

AI USE STATEMENT

The authors disclose using generative AI for language polishing and parts of the experiment code, but not for the motivation or reported results and figures.

AI USE STATEMENT

  • Generative AI tools were used to polish the manuscript language.Source: AI USE STATEMENT · p. 10
  • AI tools assisted with writing and debugging portions of the experiment code, which the authors tested and verified by running the reported experiments.Source: AI USE STATEMENT · p. 10
  • The authors state that AI was not used for the motivation or production of reported results and figures, and accept responsibility for the final content.Source: AI USE STATEMENT · p. 10

A.1 RELATED WORK

Related work covers supervised emotion-understanding approaches and training-free inference methods. The paper positions its aim against the annotation and training costs of fine-tuning.

A.1 RELATED WORK

  • Some emotion MLLM approaches build affective datasets and fine-tune models, while AffectGPT and Emotion-LLaMA add specialized dataset, encoder, or fusion designs.The paper notes the annotation and training costs of these approaches.Source: §A.1 · p. 14
  • The paper aims to enhance MLLM emotion understanding at inference without additional annotations or training costs.Source: §A.1 · p. 14
  • Training-free approaches use prompts, in-context strategies, token pruning, or multi-agent collaboration to guide frozen models at inference.Examples include MM-PEAR-CoT, EmoGist, SEPM, and MERMAID.Source: §A.1 · p. 14

A.2 EXPERIMENTAL SETUP

The evaluation uses four public emotion-recognition benchmarks and emotion accuracy, testing DAN with four MLLMs under specified inference-time settings.

A.2 EXPERIMENTAL SETUP

  • Emotion accuracy (Acc) is the evaluation metric across EmoSet, WebEmo, Emotion6, and Abstract.The benchmarks include datasets with 6, 7, 8, and 25 emotion categories, as well as abstract paintings.Source: §A.2 · p. 14
  • Experiments use Qwen2.5-VL-7B-Instruct, Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, and InternVL3.5-8B.Unless otherwise specified, ablations use Qwen3-VL-8B-Instruct.Source: §A.2 · p. 15
  • The inference-time, training-free experiments run on one NVIDIA A800 GPU with 80GB of memory.The variance threshold is α = 0.125, the dropping ratio is β = 0.3, and the truncation threshold is k = 2.Source: §A.2 · p. 15

A.3 DETAILS OF BASELINES

The section describes direct zero-shot and chain-of-thought baselines, alongside SEPM, a training-free method that prunes emotion-related visual tokens.

A.3 DETAILS OF BASELINES

  • The zero-shot baseline asks the MLLM to choose the emotion description that best represents the image.The prompt requests a direct numbered choice from the emotion set.Source: §A.3 · p. 15
  • The Zero-shot-CoT baseline adds a step-by-step reasoning instruction before asking for an emotion choice.It otherwise asks for a direct numbered choice from the emotion set.Source: §A.3 · p. 15
  • SEPM sharpens emotion perception through training-free pruning of emotion-related visual tokens.Its indiscriminate focus on emotion-related regions can weaken discrimination between semantically similar categories.Source: §A.3 · p. 15

A.4 WHY WE CHOOSE CONFIDENCE VARIANCE AS A STANDARD?

The analysis motivates confidence variance as a measure of ambiguity for deciding when to apply CDVP, and explains why candidate-conditioned attention discrepancies target discriminative rather than shared visual evidence.

A.4 WHY WE CHOOSE CONFIDENCE VARIANCE AS A STANDARD?

  • As top-two confidence variance increases, emotion accuracy improves, supporting variance as an ambiguity measure and a gate for CDVP.Lower variance indicates greater difficulty distinguishing semantically proximal emotions.Source: §A.4 · p. 15, Figure A.1 · p. 15
  • For most benchmarks, k = 2 covers most ground-truth labels; moving from k = 2 to k = 3 yields negligible Recall@k gains.WebEmo25 remains an exception: its Recall@k reaches only 42.11% even at k = 3; the authors associate this with fine-grained labels and annotation ambiguity.Source: §A.4 · pp. 15–16, Table A.1 · p. 15

A.5 WHY ARE TOKENS THAT EXHIBIT SIGNIFICANT DIFFERENCES BETWEEN ATTENTION MAPS OF TWO EMOTIONS SELECTED FOR PRUNING?

  • Tokens with similar attention for two candidate emotions provide shared evidence, while asymmetric attention can better distinguish the candidates.The paper illustrates shared evidence with a face relevant to both sadness and disappointment.Source: §A.5 · p. 16
  • CDVP retains high-discrepancy tokens and prunes low-discrepancy ones, treating discrepancy as a candidate-conditioned score rather than causal proof.Ablations report gains over adding no pruning and over random, query-related, and FoE-related pruning.Source: §A.5 · p. 16

A.6 ADDITIONAL EXPERIMENTAL ANALYSIS

The additional analyses examine cue contributions, prompt sensitivity, fine-grained classification, CDVP triggering, and inference cost; prompt templates specify the staged and contrastive queries.

A.6 ADDITIONAL EXPERIMENTAL ANALYSIS

  • Scene-level cues contribute more than object-level cues to coarse-grained polarity assessment, while overemphasizing localized details may trigger hallucinations.The authors describe scene cues as more stable and reliable for polarity judgments.Source: §A.6 · p. 17, Table A.2 · p. 17
  • Accuracy fluctuations across entire, module-based, and sequential prompt injection strategies are reported as negligible across datasets.The authors attribute the consistency to the method's architectural design rather than prompt engineering.Source: §A.6 · p. 17, Table A.3 · p. 17
  • DAN surpasses prior methods on CUB-200-2011 and FGVC-Aircraft, and matches the best model on Stanford Cars and Stanford Dogs.The authors present this as evidence of scalability beyond emotion recognition to fine-grained image classification.Source: §A.6 · p. 17, Table A.4 · p. 17
  • CDVP trigger rates decrease as model size increases and are higher on WebEmo25 than on Emotion6, EmoSet8, and WebEmo7.The authors also report a substantially high trigger rate on Abstract8, which they associate with the difficulty of interpreting abstract paintings.Source: §A.6 · pp. 17–18, Table A.5 · p. 18
  • CDVP increases average inference time per sample, but the time remains almost within one second across all datasets.The authors describe this efficiency as user-friendly for practical applications.Source: §A.6 · p. 18, Figure A.3 · p. 18

B DETAILS OF PROMPT TEMPLATES

  • The clue-mining prompt requests scene-level cues such as lighting and colors, followed by object-level cues such as facial and body expressions.The coarse prompt asks for positive or negative polarity, and the fine-grained prompt asks for a category choice based on the preceding analysis.Source: B DETAILS OF PROMPT TEMPLATES · p. 18
  • The contrastive prompt asks why a candidate emotion fits rather than its competitors and requests visual cues supporting that distinction.The discriminative prompt then asks the model to choose from the options using the discriminative visual image.Source: B DETAILS OF PROMPT TEMPLATES · pp. 18–19
Loading 2609.36782v1…