PaperlayerView exact revision on arXiv

Source-linked AI summary

Emotion-Qwen: A Unified Framework for Emotion and Vision Understanding

Dawei Huang, Qing Li, Chuan Yan, Zebang Cheng, Zihao Han, Yurong Huang, Xiang Li, Bin Li, Xiaohui Wang, Zheng Lian, Zhi-Qi Cheng, Xiaojiang Peng

arXiv:2505.06685v3cs.MMcs.CV

TL;DR

Emotion-Qwen addresses the difficulty of fine-grained video emotion understanding while preserving general vision-language reasoning, which existing emotion-focused LMMs often lose. It combines facial emotion capture, dynamically routed expert compression, staged pre-training, and the bilingual VER dataset. The framework achieves state-of-the-art emotion results while remaining competitive on general VL tasks.

Abstract

from arXiv · show

Accurate emotion understanding in videos necessitates effectively recognizing and interpreting emotional states by integrating visual, textual, auditory, and contextual cues. Although recent Large Multimodal Models (LMMs) have exhibited significant progress in general vision-language (VL) tasks, their performance often deteriorates in emotion-specific scenarios, exhibiting catastrophic forgetting when fine-tuned on emotion-centric tasks. To overcome these limitations, we propose Emotion-Qwen, a unified multimodal framework designed to simultaneously enable robust emotion understanding and preserve general VL reasoning capabilities. Emotion-Qwen introduces a novel Hybrid Compressor based on a Mixture-of-Experts (MoE) architecture, dynamically routing inputs to optimally balance emotion-specific processing and general multimodal reasoning. We further propose a carefully structured three-stage pre-training pipeline, leveraging extensive general and emotion-focused datasets to strengthen multimodal representation robustness and model adaptability. Additionally, we develop the Video Emotion Reasoning (VER) dataset, a large-scale bilingual resource containing over 40K video clips annotated with detailed context-aware emotional descriptions, significantly facilitating research on fine-grained emotional reasoning. Extensive experiments confirm that Emotion-Qwen achieves state-of-the-art performance across multiple emotion recognition and reasoning benchmarks, while maintaining highly competitive results in general VL tasks.

Introduction

Emotion-Qwen targets the gap between coarse emotion recognition and general multimodal reasoning by combining emotion-specific processing with general vision-language understanding. It also introduces VER to support fine-grained, context-aware emotional reasoning.

Introduction

  • Emotion understanding requires integrating facial, vocal, linguistic, and contextual visual cues, but unimodal and existing multimodal approaches struggle with dynamic, contextually nuanced emotions.The challenge is interpreting complex emotional states rather than merely classifying isolated signals.Source: Introduction · p. 1
  • Existing emotion-focused LMMs often provide only coarse basic-emotion classification, and emotion fine-tuning can cause catastrophic forgetting of general vision-language capabilities.These limitations hinder deeper interpretability, context-aware reasoning, and robust multimodal generalization.Source: Introduction · p. 1
  • Emotion-Qwen combines a Facial Emotion Capture module with an attention-aware Mixture-of-Experts Hybrid Compressor to balance nuanced emotion understanding and general vision-language reasoning.The FEC module extracts expressive facial features, while the compressor dynamically routes inputs between emotion-specialized and general-purpose experts.Source: Introduction · p. 1
  • VER contains over 40K bilingual video clips with more than 80K detailed annotations focused on contextual and causal emotional cues beyond traditional classification.The dataset is designed to support advanced fine-grained emotional reasoning.Source: Introduction · p. 2
  • Emotion-Qwen reports state-of-the-art performance across emotion-centric and general vision-language benchmarks, including 87.3 on MMBench, 87.9 on zero-shot TextVQA, and 78.31 UAR on DFEW.Additional reported results include 8.25/8.16 Clue/Label Overlap on EMER and 85.49 accuracy on EmoSet.Source: Introduction · p. 2

Related Work

Prior LMMs improve general multimodal reasoning but remain weak on fine-grained emotional interpretation and may lose general VL abilities after emotion-focused tuning. Emotion-Qwen addresses these gaps with VER, a dataset capturing dynamic, contextual, and cross-situational emotional reasoning.

Related Work

  • Existing LMMs support unified reasoning across visual, textual, and auditory inputs but struggle with nuanced emotions and causal explanations in complex videos.Emotion-focused fine-tuning can further compromise general vision-language capabilities through catastrophic forgetting.Source: Related Work · p. 2
  • Traditional multimodal emotion recognition datasets mainly provide limited single-label annotations, leaving contextual information and deeper reasoning insufficiently represented.VER is proposed to capture dynamic expressions, narrative contexts, and cross-situational inference.Source: Related Work · pp. 2–3
  • Recent emotion-oriented architectures and training strategies improve reasoning but often suffer from task-specific overfitting, while richer resources such as EMER remain limited to isolated contexts.These limitations motivate a broader video resource for context-aware emotional reasoning.Source: Related Work · pp. 2–3
  • Figure 2 presents Emotion-Qwen’s pipeline, combining facial emotion capture with a Hybrid Compressor that dynamically integrates emotion-specific and general multimodal features.The architecture includes a FEC module, vision encoder, Hybrid Compressor, and LLM backbone.Source: Figure 2 · p. 3

Methodology

Emotion-Qwen combines facial-expression selection, visual encoding, expert-routed compression, and staged training to support both emotion reasoning and general vision-language processing. Its methodology also introduces the bilingual VER dataset, whose annotations are model-synthesized and human-verified.

Overview of Emotion-Qwen

  • Emotion-Qwen processes video through Facial Emotion Capture, a CLIP ViT vision encoder, a Hybrid Compressor, and a Qwen2.5 language-model backbone.The compressor aligns compressed visual tokens with textual features before multimodal generation.Source: Overview of Emotion-Qwen · p. 3

Facial Emotion Capture Module

  • FEC detects faces, estimates landmarks and emotion probabilities, selects high-confidence emotional frames, masks salient facial regions, and preserves temporal order before encoding.The enriched sequence emphasizes expressive facial content while retaining the original video’s temporal coherence.Source: Facial Emotion Capture Module · p. 3

Hybrid Compressor with Expert Routing

  • The Hybrid Compressor routes visual embeddings through Emotion and General Experts, then uses attention-based gating to combine their outputs adaptively.This design preserves task-relevant information, reduces redundancy, and improves multimodal representation alignment for emotion and general VL tasks.Source: Hybrid Compressor with Expert Routing · pp. 3–4

Video Emotion Reasoning Dataset

  • VER builds on filtered MAFW and MER2024 samples, synthesizes fine-grained annotations from multiple models, and retains only annotations rated at least 3 by professional analysts.The dataset emphasizes facial expressions, body language, contextual settings, and detailed emotional reasoning.Source: Video Emotion Reasoning Dataset · p. 4, Figure 3 · p. 4

Pre-training for Emotion-Qwen

  • Three pre-training stages separately warm up the General and Emotion Experts before unlocking the language model for diverse vision-language instruction tuning.Stage 1 uses general image-text data, Stage 2 uses culturally diverse facial-emotion data, and Stage 3 trains complex multimodal patterns.Source: Pre-training for Emotion-Qwen · p. 4, Table 1 · p. 4

Emotional Instruction Fine-tuning

  • Emotional instruction fine-tuning freezes the Vision Encoder and Hybrid Compressor while applying multiple LoRA instances to the LLM with emotion-focused prompts.VER, DFEW, and EmoViT provide task-specific supervision for diverse multimodal emotion reasoning tasks.Source: Emotional Instruction Fine-tuning · pp. 4–5, Table 2 · p. 5

Experiments

Emotion-Qwen is evaluated across general vision-language, emotion-centric, and ablation settings, with results emphasizing balanced performance and adaptive specialization. Its strongest reported results come from fine-tuning with VER and from the Hybrid Compressor and multi-LoRA training strategy.

Zero-shot Evaluation

  • 87.3 on MMBench, 77.2 on ScienceQA, and 87.9 on TextVQA show competitive zero-shot general vision-language performance against leading open-source and proprietary LMMs.The reported comparisons include GPT-4V, Qwen2-VL, and DeepSeek-VL.Source: Zero-shot Evaluation · p. 5

Instruction Fine-tuning Evaluation

  • 85.47 on MER2024 SEMI, 79.67 on NOISE, 78.31 WAR on DFEW, and 85.49 accuracy on EmoSet establish leading fine-tuned emotion-benchmark results.The DFEW WAR result exceeds MMA-DFER’s 77.51, while the EmoSet result surpasses EmoViT’s prior SOTA.Source: Instruction Fine-tuning Evaluation · p. 6, Table 4 · p. 6
  • 8.25 Clue Overlap and 8.16 Label Overlap on EMER set new records and outperform Emotion-LLaMA on emotion reasoning.These results are reported after instruction fine-tuning with VER and other specialized emotion datasets.Source: Instruction Fine-tuning Evaluation · p. 6

Ablation Study

  • The FEC module improves EmoSet by 2.29%, DFEW by 1.92%, and MER2024 by 2.36%, while its effect on general VL performance is negligible.MMBench shows a slight gain when FEC is excluded, consistent with its scene-level rather than facial-expression emphasis.Source: Ablation Study · p. 6, Table 5 · p. 6
  • The Hybrid Compressor outperforms alternative projector configurations and surpasses the MLP projector by over 8.44% accuracy on TextVQA.It achieves the best reported accuracy-efficiency balance despite modest parameter and computational overhead.Source: Ablation Study · pp. 6–7, Table 6 · p. 6
  • Average gating weights favor the General Expert on general VL tasks and the Emotion Expert on emotion-specific tasks, supporting task-aware routing.The routing analysis is reported in Table 7.Source: Ablation Study · p. 7, Table 7 · p. 7
  • Multiple LoRA adapters outperform full fine-tuning and a single adapter across evaluated tasks while matching single-LoRA training time and GPU memory use.Full fine-tuning performs relatively worse, which the authors associate with limited instruction-tuning scale and distributional mismatch.Source: Ablation Study · p. 7, Table 8 · p. 7, Table 9 · p. 7
  • Performance remains influenced by the LLM backbone, although the framework integrates competitively with LLaMA 3 across evaluated benchmarks.The authors identify larger backbone models as future work.Source: Ablation Study · p. 7, Table 10 · p. 7

Conclusion

Emotion-Qwen combines facial-expression extraction, dual-expert multimodal compression, and the bilingual VER dataset for fine-grained emotional reasoning. It reports state-of-the-art emotion results while retaining strong general vision-language performance, with audio integration and broader cross-modal generalization left for future work.

Conclusion

  • Emotion-Qwen combines the FEC module and dual-expert Hybrid Compressor to balance expressive facial-cue extraction with general vision-language reasoning.The framework is designed to address limitations in emotional LMMs.Source: Conclusion · p. 7
  • The bilingual VER dataset supports fine-grained, context-aware emotional reasoning at large scale.The paper describes VER as a resource tailored to multimodal emotional reasoning.Source: Conclusion · p. 7
  • Emotion-Qwen achieves state-of-the-art performance across emotion-centric benchmarks while maintaining superior results on general vision-language tasks.The authors also state that model weights and code are publicly available.Source: Conclusion · p. 7
  • Future work will incorporate audio modalities and improve cross-modal generalization.Source: Conclusion · p. 7

Related papers

Loading 2505.06685v3…