PaperlayerView exact revision on arXiv

Source-linked AI summary

CASE: Confusion-Aware Skill Evolution for Frozen Multimodal Emotion Models

Siyu Wu, Zeyu Liang, Linghao Wang, Qiyun Zheng, Yi Yao, Ling Lo, Hongxia Xie

arXiv:2610.32481v1cs.AI

TL;DR

MLLMs still systematically confuse semantically adjacent emotions despite their ability to reason over multimodal evidence. CASE uses a sparse library of pair-specific skills with frozen backbones, and experiments report consistent effectiveness across emotion tasks and multiple backbones, including transfer without target-dataset evolution or backbone updates.

  • Problem

    MLLMs can identify relevant affective evidence yet still struggle to distinguish semantically similar emotions, whose confusions cluster around neighboring pairs.

    Source: Introduction · p. 1

  • Method

    CASE selectively retrieves and evolves pair-specific EmotionSkills in a sparse external library while keeping the MLLM backbone frozen.

    Source: Introduction · p. 2

  • Results

    Experiments across three emotion tasks and multiple backbones demonstrate consistent effectiveness and generality, with DFEW-evolved skills transferring to CA-MER and EMER without target-dataset evolution or backbone updates.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    External skill evolution is a promising alternative to parameter-based adaptation for multimodal emotion understanding.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have substantially advanced Multimodal Emotion Recognition (MER) by enabling fine-grained emotion understanding through reasoning over multimodal evidence. However, they still exhibit systematic confusion between semantically adjacent emotions with highly overlapping multimodal cues. Existing approaches typically address these errors through global parameter adaptation, making confusion correction costly and inflexible. We instead observe that emotion confusion is highly localized and therefore better addressed through reusable pair-specific expertise. Building on this insight, we propose Confusion-Aware Skill Evolution (CASE), the first agent-skill framework for MER. CASE maintains a sparse, self-evolving library of pair-specific EmotionSkills that externalize affective expertise from model parameters, enabling the learned expertise to be continuously refined and readily transferred across diverse frozen MLLM backbones. During inference, CASE dynamically retrieves the appropriate EmotionSkill to reinforce boundary-aware reasoning while keeping the MLLM backbone frozen. Extensive experiments on CA-MER, DFEW, and EMER demonstrate the effectiveness and transferability of CASE across both general-purpose and emotion-specialized MLLMs. Notably, CASE improves the average accuracy of Qwen3-VL-8B on the CA-MER benchmark from 46.80% to 59.13%, while achieving consistent gains across MLLMs.

Introduction

MLLMs can reason over multimodal evidence but still confuse semantically similar emotions. CASE addresses this localized confusion with reusable pair-specific skills rather than global parameter adaptation.

Introduction

  • Emotion errors concentrate on neighboring pairs, motivating explicit reinforcement of discriminative boundaries rather than treating all categories equally.Existing approaches primarily rely on global parameter adaptation or task-specific fine-tuning, which overlooks localized confusion and does not accumulate reusable pair-specific expertise.Source: Introduction · pp. 1–2
  • CASE maintains a sparse, self-evolving library of pair-specific EmotionSkills that are retrieved during inference while the MLLM backbone remains frozen.The skills externalize affective expertise from model parameters and can be reused across frozen backbones.Source: Introduction · p. 2
  • CASE localizes ambiguous predictions, evolves skills from recoverable errors, and expands its library only when recurrent confusion patterns emerge.Its skills are pair-specific discrimination protocols for systematically confused adjacent emotions.Source: Introduction · p. 2
  • Skills evolved only on DFEW transfer to CA-MER and EMER without target-dataset evolution or backbone updates.The evaluation spans closed-set in-the-wild recognition, audio-visual conflict-aware recognition, and open-vocabulary emotion reasoning.Source: Figure 1 · p. 1, Introduction · p. 2
  • CASE reframes MER from global multiclass classification toward selective discrimination on ambiguous fine-grained emotion boundaries.This external skill approach supports evolving emotion taxonomies without continual model adaptation.Source: Introduction · p. 2

Related Work

Prior MLLM emotion work expands label spaces and evidence modeling, while non-parametric learning methods store and reuse experience in varied forms. CASE applies reusable, selectively activated skills to recurrent confusion between emotion categories.

Multimodal Emotion Recognition with MLLMs

  • Existing MLLM emotion methods expand affective label spaces, improve multimodal evidence modeling, support open-vocabulary labels, and address incongruent evidence, but rarely target recurrent category confusions.The cited approaches include emotion-specific architectures, instruction tuning, retrieval augmentation, adaptive fusion, and modality selection.Source: Multimodal Emotion Recognition with… · p. 2

Non-Parametric Learning through Memory and Skills

  • Non-parametric approaches reuse experience as retrieved examples, verbal reflections, refined outputs, evolved prompts, or reusable instructions, workflows, toolboxes, and skills.Some persistent skill libraries support continual refinement and verifier-guided evolution.Source: Non-Parametric Learning through Memory… · p. 2
  • CASE adapts skill-library learning to MER: each EmotionSkill captures cues for one confusable pair, activates from top-2 predictions, and remains only if it improves that local boundary.The underlying MLLM stays frozen while external pair-specific knowledge refines the decision boundary.Source: Non-Parametric Learning through Memory… · p. 2, Multimodal Emotion Recognition with… · p. 2

Method

CASE treats fine-grained MER as local discrimination between candidate emotion pairs, using a sparse external skill library while keeping the prompt-conditioned MLLM frozen. Its lifecycle selectively routes ambiguous cases, attributes feedback to a pair, and validates skill changes before deployment.

Method

  • CASE models each input as audio, video, and optional linguistic context, and treats the recognizer as a frozen prompt-conditioned MLLM interface.The output schema is determined by the prompt, and emotion-valued output fields take values from the label space.Source: Problem Formulation · p. 2
  • Because the MLLM often identifies two plausible emotions but applies the wrong distinguishing criterion, CASE compares candidate pairs instead of reconsidering all labels.The approach decomposes recognition into local discrimination and stores skills for empirically observed confusion pairs, keeping the library sparse.Source: Problem Formulation · pp. 2–3
  • CASE’s sequential evolution procedure takes a frozen MLLM, editor, initializer, skill library, and evolution stream, and returns an evolved skill library.Algorithm 1 distinguishes primary recoverability-guided local evolution from secondary expansion for uncovered pairs.Source: Problem Formulation · p. 3
  • CASE selectively reopens an ambiguous boundary: clear predictions are preserved, while routed cases receive pairwise inference when a matching skill exists.A pair-specific protocol guides re-examination of facial, behavioral, temporal, and contextual cues and comparison of evidence for both emotions.Source: Phase 1: Selective Confusion-Pair… · p. 3
  • For covered pairs, CASE records diagnostic evidence, skill version, round-one and round-two predictions, and ground-truth feedback for trajectory attribution.Only attributable trajectories provide pair-level learning signals; candidate-recall errors are excluded from skill editing.Source: Phase 2: Recoverability-Aware Skill… · pp. 3–4
  • When failures accumulate, an editor proposes a local textual patch that extracts reusable distinctions, preserves rules supported by successful anchors, and avoids sample-specific or label-forcing expressions.The patch is based on a bounded set of failures for the pair.Source: Phase 2: Recoverability-Aware Skill… · p. 4
  • A candidate revision is activated only if it improves pair-specific predictions on a fixed validation subset without collapsing toward either label.Otherwise, CASE retains the existing skill; only the external library is updated, not the MLLM.Source: Problem Formulation · p. 3, Phase 3: Behaviorally Validated Update… · p. 4
  • Uncovered pairs contribute expansion evidence only when the ground-truth emotion belongs to the pair; recurrent candidates require validation support and no label collapse before activation.Rejected candidates remain inactive, so expansion targets recurrent, behaviorally supported boundaries rather than all possible pairs.Source: Phase 3: Behaviorally Validated Update… · p. 4

Experiments

Experiments evaluate CASE across frozen backbones and three emotion-recognition settings, reporting gains on CA-MER, DFEW, and open-set EMER. Additional analyses examine selective routing and pairwise repair.

Experiments

  • Evaluation spans CA-MER audio-visual conflict subsets, DFEW in-the-wild facial expression recognition, and fine-grained multimodal reasoning on EMER.The reported metrics are accuracy for CA-MER and EMER, and UAR and WAR for DFEW.Source: Experimental Setup · p. 5
  • In the sad–angry example, a pair-specific EmotionSkill guides focused re-evaluation that corrects the backbone prediction without reconsidering the full label space.Figure 3 depicts the pairwise repair.Source: Overall Effectiveness and Generality… · p. 5, Figure 3 · p. 5
  • 51.58%: Emotion-Qwen with CASE-Retrieval achieves this accuracy on open-set EMER, and CASE improves all three frozen backbones without EMER-specific adaptation.CASE-Retrieval consistently outperforms CASE-Full on EMER.Source: Figure 4 · p. 5, Overall Effectiveness and Generality… · p. 5
  • 60.88 UAR and 68.69 WAR: CASE with Gemini-3.5-Flash achieves these DFEW results, supporting effectiveness in the in-distribution setting.The paper reports that CASE consistently benefits different frozen backbones on DFEW.Source: Overall Effectiveness and Generality… · p. 5
  • On routed samples covered by a skill, the round-1 error rate is 52.0% versus 36.7% for non-routed samples, and intervention raises accuracy from 48.0% to 55.3%.The accuracy gain is 7.3 percentage points.Source: Selective Local Repair (RQ2) · p. 5
  • 46.80% to 59.13%: CASE-Full improves Qwen3-VL-8B on CA-MER, with Table 1 covering CA-MER and DFEW results.The supplied result also reports an improvement for Gemini-3.5-Flash, but does not provide its complete value pair.Source: Overall Effectiveness and Generality… · p. 5, Table 1 · p. 6

Model UAR WAR

CASE improves the CA-MER skill library through trajectory-driven evolution and validated expansion, while selective routing preserves predictions for non-routed and uncovered cases. Pair-level results show gains across most examined confusion pairs.

Selective Local Repair (RQ2)

  • All configurations achieve positive overall Repair–Harm Gaps, with detailed results reported in Supplementary Section D.The supplied passage does not give the gap values.Source: Model UAR WAR · p. 6
  • Non-routed samples retain their round-1 predictions at 63.3% accuracy, and samples routed to uncovered pairs retain their predictions at 69.6%.CASE therefore intervenes mainly on difficult, covered cases rather than changing predictions in these other groups.Source: Model UAR WAR · p. 6

Which confusion boundaries are repaired?

  • Both CASE variants improve most of the examined confusion pairs, especially sad–worried and angry–worried.Figure 6 reports PairAcc changes for the five most frequent first-round confusion pairs.Source: Model UAR WAR · p. 6

Trajectory-Driven Skill Evolution (RQ3)

  • Trajectory-driven evolution improves the static EmotionSkill library’s overall accuracy from 57.00% to 58.27%, with the external library improving from prediction trajectories while the backbone remains frozen.Source: Trajectory-Driven Skill Evolution (RQ3) · p. 6

Validated Deployment and Sparse Expansion (RQ4)

  • Among 114 candidate revisions, CASE accepts 10 and rejects 102 without validation gains and two that cause label collapse.The editor proposes revisions; the validation gate determines which enter the active library.Source: Validated Deployment and Sparse… · p. 6
  • Four new pairs pass the bootstrap gate, expanding coverage from 17 to 21 pairs and improving CA-MER accuracy from 58.27% to 59.13%.The gate is intended to admit behaviorally supported updates rather than maximize accepted revisions.Source: Validated Deployment and Sparse… · p. 7

Conclusion

CASE localizes ambiguous predictions to recurrent emotion-pair boundaries and selectively invokes pair-specific EmotionSkills while keeping MLLMs frozen. Experiments across three emotion tasks and multiple backbones demonstrate consistent effectiveness and generality.

Conclusion

  • CASE localizes ambiguous predictions to recurrent emotion-pair boundaries and selectively invokes pair-specific EmotionSkills for focused discrimination.The framework avoids globally updating model parameters.Source: Conclusion · p. 7
  • Experiments across three emotion tasks and multiple backbones demonstrate CASE's consistent effectiveness and generality.The paper presents external skill evolution as a promising alternative to parameter-based adaptation for multimodal emotion understanding.Source: Conclusion · p. 7
  • Figure 6 tracks PairAcc changes on the five most frequent first-round confusion pairs under CASE-Full and CASE-Retrieval.Source: Figure 6 · p. 7

Related papers

Loading 2610.32481v1…