PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis

Qize Yang, Detao Bai, Yi-Xing Peng, Xihan Wei

arXiv:2501.09502v1cs.CV

TL;DR

Video MLLMs struggle to integrate audio, capture subtle facial expressions, and learn from detailed multimodal emotion data. Omni-Emotion addresses these gaps with reviewed datasets and aligned audio and facial encoders, achieving state-of-the-art emotion recognition and reasoning while retaining practical limitations.

  • Problem

    Video MLLMs struggle to recognize emotions because they inadequately integrate audio and subtle facial cues, while detailed multimodal emotion instruction datasets remain limited.

    Source: §1 · pp. 1–2

  • Method

    Omni-Emotion constructs SRE and HRE datasets and aligns Whisper-large-v3 audio and FaceXFormer facial features with a video MLLM for instruction tuning.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Omni-Emotion achieves state-of-the-art performance in emotion recognition and reasoning, including 68.80/53.81 UAR on DFEW/MAFW, 65.9 EMER-OV, and 8.22 EMER clue overlap.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The datasets and unified audio-facial modeling support multimodal emotion analysis across recognition, open-vocabulary, and reasoning tasks.

    Source: §1 · p. 2, §6 · p. 8, §7 · p. 9

    The datasets’ video diversity remains below real-world applications, mixed-emotion accuracy remains limited, and model size and input length hinder practical deployment.

    Source: §5.5 · p. 8

Abstract

from arXiv · show

Understanding emotions accurately is essential for fields like human-computer interaction. Due to the complexity of emotions and their multi-modal nature (e.g., emotions are influenced by facial expressions and audio), researchers have turned to using multi-modal models to understand human emotions rather than single-modality. However, current video multi-modal large language models (MLLMs) encounter difficulties in effectively integrating audio and identifying subtle facial micro-expressions. Furthermore, the lack of detailed emotion analysis datasets also limits the development of multimodal emotion analysis. To address these issues, we introduce a self-reviewed dataset and a human-reviewed dataset, comprising 24,137 coarse-grained samples and 3,500 manually annotated samples with detailed emotion annotations, respectively. These datasets allow models to learn from diverse scenarios and better generalize to real-world applications. Moreover, in addition to the audio modeling, we propose to explicitly integrate facial encoding models into the existing advanced Video MLLM, enabling the MLLM to effectively unify audio and the subtle facial cues for emotion understanding. By aligning these features within a unified space and employing instruction tuning in our proposed datasets, our Omni-Emotion achieves state-of-the-art performance in both emotion recognition and reasoning tasks.

1. Introduction

Omni-Emotion addresses video MLLMs’ difficulty integrating audio, fine-grained facial information, and detailed multimodal emotion annotations. It combines reviewed datasets with explicit audio and facial encoders, achieving state-of-the-art results across emotion analysis tasks.

Motivation

  • Video MLLMs struggle to recognize emotions because they lack reasoning abilities and do not effectively capture audio or fine-grained facial information.These limitations produce sub-optimal performance in real-world scenarios.Source: §1 · pp. 1–2
  • Emotion analysis data remains difficult to curate because labeling standards vary, expressions have different definitions, and cultural contexts influence emotional expression.The paper identifies detailed, diverse multimodal instruction data as a missing resource.Source: §1 · p. 2

Contributions

  • The paper curates 24,137 self-reviewed videos and 3,500 manually verified videos with detailed multimodal emotion annotations.The annotations cover general visual cues, facial attributes and expressions, audio, explainable reasons, and open-vocabulary labels.Source: §1 · p. 2

Method

  • Omni-Emotion aligns Whisper-large-v3 audio and FaceXFormer facial features with a video MLLM embedding space and jointly trains encoder projectors with the LLM decoder.The approach uses instruction tuning on the curated datasets to process multiple modalities.Source: §1 · p. 2

Contributions

  • 68.80/53.81 UAR on DFEW and MAFW, 65.9 average EMER-OV score, and 8.22 EMER clue overlap establish state-of-the-art performance across emotion tasks.The model outperforms existing methods by a clear margin.Source: §1 · p. 2

2. Related works

Prior emotion systems and video MLLMs often lack integrated audio, detailed facial modeling, quality assurance, or sufficient reasoning-oriented annotations. Omni-Emotion extends video MLLMs with audio and fine-grained facial cues while using reviewed multimodal data.

Multimodal Large Language Models

  • Existing video MLLMs have difficulty understanding audio cues and subtle facial expressions because they lack specialized multimodal emotion training.Omni-Emotion extends a video MLLM with a strong audio encoder and fine-grained facial cues.Source: §2 · pp. 2–3

Emotion Analysis

  • Emotion datasets commonly use categorical labels, while EMER provides detailed reasons at only 332 samples and Emotion-LLaMA’s inferred descriptions lack thorough quality assessment.These limitations motivate larger, reviewed annotations with richer emotional explanations.Source: §2 · p. 2
  • Earlier multimodal methods focus on feature interaction or modality completion, whereas emotion-oriented MLLMs still struggle with precise emotional analysis and reasoning.Related systems integrate vision or audio but lack effective facial modeling for fine-grained facial information.Source: §2 · pp. 2–3

3. Self-Reviewed Emotion Dataset

The self-reviewed dataset pipeline enriches existing emotion datasets with global visual, facial, audio, and reasoning annotations, then filters them against ground-truth labels. It produces the 24,137-sample SRE dataset and a separately human-reviewed 3,500-sample HRE dataset.

3. Self-Reviewed Emotion Dataset

  • The dataset construction leverages existing DFEW, MAFW, MER24, CAER, AFEW-VA, RAVDESS, and FERV39K datasets, using global information, detailed facial features, and audio cues to extract emotional clues.Source: §3 · p. 3
  • The pipeline extracts global video descriptions at 1 FPS, tracked facial details, and audio captions, ASR, and emotion recognition before GPT-3.5 consistency verification.Inconsistent or uncertain intermediate descriptions are discarded.Source: §3 · p. 3

3.1. Self-Reviewed Emotion Dataset Curation

  • GPT-3.5 scores agreement between explainable emotion reasons and original labels from 0 to 10, with scores below 5 treated as misaligned.The score distributions are examined on DFEW and MAFW.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
  • 24,137 samples form the SRE dataset after GPT-based alignment filtering, extending categorical emotion labels to multimodal analysis while maintaining high agreement.Samples scoring below 5 on the 0–10 alignment scale are removed, and indoor RAVDESS clips are excluded.Source: §3.1 · pp. 3–4

3.2. Human-Reviewed Emotion Dataset Curation

  • The HRE dataset contains 3,500 manually reviewed samples selected as 700 non-overlapping samples from each source training set.Together, SRE and HRE include samples from diverse sources.Source: §3.2 · p. 4

4. Omni-Emotion MLLM

Omni-Emotion extends a video MLLM with audio and fine-grained facial encoders, then fuses these features with general visual information for emotion analysis. The model uses staged alignment and multimodal instruction tuning to unify the modalities while retaining general visual knowledge.

4. Omni-Emotion MLLM

  • Omni-Emotion extends a video MLLM with audio and facial encoders to capture auditory clues and fine-grained facial information.The proposed architecture adds audio and facial encoders to an existing video MLLM.Source: §4 · p. 4, Figure 5 · p. 5

4.1. Extending Video MLLM to Audio Modality

  • Whisper-large-v3 audio representations are aligned with language features through an audio encoder and projector whose output matches the LLM hidden size.Audio is resampled to 16 kHz, converted to mel spectrograms, pooled, and projected before entering the LLM.Source: §4.1 · p. 4

4.2. Detail Facial Modeling

  • FaceXFormer supplies fine-grained facial representations because general visual encoders also capture background and non-person cues that are less relevant to emotion analysis.Its multi-scale facial features are fused, flattened, and passed through a projector matching the LLM hidden size.Source: §4.2 · p. 5

4.3. Visual Feature Fusion Module

  • The visual fusion module evaluates frame-level concatenation, frame-level cross-attention, and whole-clip concatenation to combine facial and general visual features.Cross-attention uses facial features as queries and visual features as keys and values, while whole-clip concatenation has the least effect on the original video MLLM.Source: Figure 6 · p. 5, §4.3 · pp. 5–6

4.4. Training Detail

  • Training proceeds through audio alignment, facial alignment, and multimodal instruction tuning, with modality tokens and frame-sequence parameters supporting multimodal and temporal modeling.The final phase uses general visual, facial, and audio features while training the projectors and LLM decoder.Source: §4.4 · p. 6

5. Experiments

Across open-vocabulary, reasoning, modality-analysis, and in-the-wild recognition settings, Omni-Emotion is evaluated against existing methods and shows strong results. The section also reports dataset and deployment limitations.

5. Experiments

  • The experiments cover open-vocabulary recognition, multimodal emotion reasoning, and additional emotion-recognition evaluation across distinct settings.EMER-OV contains open-vocabulary labels, and EMER descriptions include facial micro-expressions, speech tone, and contextual video information.Source: §5 · p. 6, §5.1 · p. 6, §5.2 · p. 7

5.1. Open-Vocabulary Emotion Recognition

  • In EMER-OV, Omni-Emotion significantly outperforms AffectGPT, especially in recall, while producing a richer set of emotional descriptors.The open-vocabulary evaluation uses 332 samples and compares grouped-label recall, precision, and average score.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2. Multimodal Emotion Reasoning

  • Omni-Emotion achieves the highest scores on both emotion-cue overlap and summarized-emotional-state overlap in EMER reasoning evaluation.The evaluation uses ChatGPT scoring under the same setting as Emotion-LLaMA.Source: §5.2 · p. 7, Table 2 · p. 7

5.3. Analysis

  • Adding audio significantly improves recall and accuracy, while vision, audio, and face together achieve the best average performance despite slightly lower accuracy than vision plus audio.The analysis identifies fine-grained facial cues as important for multimodal open-vocabulary emotion recognition.Source: §5.3 · p. 7

5.4. Emotion Recognition Evaluation.

  • Omni-Emotion achieves state-of-the-art performance on DFEW and MAFW, with a notable margin on the more challenging MAFW dataset.DFEW and MAFW contain 7 and 11 emotion categories, respectively, and evaluation uses UAR and WAR.Source: §5.4 · pp. 7–8, Table 5 · p. 8

5.5. Limitations

  • The datasets remain less diverse than real-world video, mixed-emotion accuracy is insufficient, and model size and input-token length challenge practical deployment.The authors note that front-facing-camera data may have a different distribution from existing sources.Source: §5.5 · p. 8

6. Conclusion

Omni-Emotion addresses insufficient detailed training data and weak audio and facial modeling in current MLLMs by combining reviewed datasets with aligned audio and facial encoders.

  • Omni-Emotion uses 24,137 self-reviewed samples and 3,500 manually annotated samples with detailed emotion annotations to support learning and generalization.Source: §6 · p. 8
  • The method incorporates audio and facial encoders into existing Video MLLMs, aligns their features in a unified space, and applies instruction tuning.Source: §6 · p. 8
  • Omni-Emotion achieves state-of-the-art results in emotion recognition and reasoning.Source: §6 · p. 8

7. More Qualitative Results Details of Datasets

Qualitative comparisons show that Omni-Emotion captures overall visual context, subtle facial expressions, and audio, producing scores closer to ground-truth details than Emotion-LLaMA.

  • Omni-Emotion captures overall visual information, subtle facial expressions, and audio within videos, with scores significantly higher than Emotion-LLaMA.Source: §7 · p. 9

8. Instructions Details of Training

Training uses separate audio, facial, and multimodal instruction-tuning data, with examples targeting audio understanding, speech transcription, facial emotion recognition, and integrated emotion analysis.

Audio feature alignment

  • Audio alignment uses 1.5M instruction-tuning examples covering integrated emotional analysis from actions, facial expressions, audio, and visual context.Source: §8 · p. 9
  • Audio instructions include captioning, summarization, and speech transcription tasks.Source: §8 · p. 9

Facial feature alignment

  • Facial alignment uses about 130K examples from seven emotion datasets to train the facial projector.Source: §8 · p. 9
  • Facial instructions ask models to identify the primary emotion from candidate emotion categories.Source: §8 · p. 9

Multimodal instruction tuning

  • Multimodal instruction tuning uses HRE and SRE to generate about 100K training examples.Source: §8 · p. 9

9. Annotation comparison with other datasets

Compared with EMER and MERR, the proposed dataset is more aligned with EMER, more detailed than MERR, and combines self-review with human review for scalable quality control.

  • The proposed annotations are more aligned with EMER and more detailed than MERR.Source: §9 · p. 9
  • Self-review against original ground-truth labels and human review provide quality control while supporting a larger annotation scale.Source: §9 · p. 9

Related papers

Loading 2501.09502v1…