PaperlayerView exact revision on arXiv

Source-linked AI summary

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, Alexander Hauptmann

arXiv:2406.11161v2cs.AIcs.MM

TL;DR

Emotion perception is multimodal, yet existing MLLMs have difficulty processing audio and recognizing facial micro-expressions. The paper introduces the MERR dataset and Emotion-LLaMA, which align audio, visual, and textual features for instruction-tuned emotion recognition and reasoning. Emotion-LLaMA reports strong performance across EMER, MER2023, MER2024, and DFEW evaluations.

  • Problem

    Existing emotion-recognition methods are limited by single-modality inputs, while MLLMs struggle with audio integration and subtle facial micro-expressions.

  • Method

    The paper constructs MERR and develops Emotion-LLaMA, using emotion-specific audio and visual encoders, shared feature alignment, and instruction tuning.

  • Results

    Emotion-LLaMA outperforms other MLLMs across evaluations, including 7.83 Clue Overlap and 6.25 Label Overlap on EMER and 0.9036 F1 on MER2023-SEMI.

  • Takeaways & Limitations

    The results support Emotion-LLaMA’s reported generalization across multimodal emotion recognition and reasoning tasks.

  • Takeaways & Limitations

    The open-source MERR release excludes source videos, which require direct application to original providers and compliance with EULAs.

Abstract

from arXiv · show

Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling. However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on MER2023-SEMI challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset.

1 Introduction

Real-world emotion perception is inherently multimodal, but existing single-modality methods and MLLMs struggle with audio integration, micro-expressions, and emotion-specific reasoning. Emotion-LLaMA addresses these gaps with the MERR dataset and instruction-tuned multimodal modeling, achieving strong results across benchmarks.

  • Real-world emotional data integrates text, audio, and images, limiting the coverage of single-modality approaches.
  • Existing multimodal methods emphasize feature interaction and modality completion, while knowledge-level interaction for emotional reasoning remains under-explored.
  • Audio processing and subtle facial micro-expression recognition remain key challenges for MLLMs such as GPT-4V.
  • Emotion-LLaMA combines the MERR dataset, emotion-specific encoders, and instruction tuning to improve multimodal emotional recognition and reasoning.
  • 7.83 Clue Overlap and 6.25 Label Overlap were achieved on EMER, alongside F1 scores of 0.9036 on MER2023-SEMI and 0.8452 on MER2024-NOISE.

2 Related Work

Related work covers multimodal large language models and instruction tuning, highlighting their growing capabilities alongside persistent limitations in emotional understanding.

  • Multimodal Large Language Models (MLLMs): MLLM research commonly adapts pretrained models such as CLIP, Q-Former, and ImageBind for general-domain applications.
  • Multimodal Large Language Models (MLLMs): GPT-4V still struggles with audio emotional cues and facial micro-expressions because specialized multimodal emotion training and knowledge are limited.
  • Instruction Tuning: Instruction tuning has been applied across NLP and vision-language tasks to improve language-model zero-shot and few-shot capabilities.
  • Instruction Tuning: LLaVA converts image-text pairs into instruction-following data, while EmoVIT generates visual emotion instruction data from paired annotations.

3 Methodology

Emotion-LLaMA constructs multimodal emotion annotations and integrates audio, visual, and textual features into an instruction-tuned LLaMA model. Its pipeline selects emotionally informative frames, extracts complementary modality-specific cues, projects them into a shared token space, and trains recognition and reasoning tasks.

  • The methodology comprises MERR dataset construction, the Emotion-LLaMA architecture, and multimodal training procedures.
  • 3.1 MERR Dataset Construction: MERR selects a peak-expression frame using OpenFace Action Unit intensities, then combines visual, contextual, and audio descriptions into comprehensive emotional annotations.
  • 3.1 MERR Dataset Construction: 28,618 samples receive coarse-grained LLaMA-3 annotations, while 4,487 expert-selected samples receive fine-grained annotations spanning broader emotional categories.
  • 3.2 Multimodal Emotion-LLaMA Model: HuBERT encodes audio, while MAE, VideoMAE, and EVA provide local facial, temporal dynamic, and global contextual visual features.
  • 3.2 Multimodal Emotion-LLaMA Model: Linear projections transform audio and visual features into language-space tokens that are fused with text through Emotion-LLaMA’s inner cross-attention mechanism.
  • 3.3 Training Procedures: Training first aligns multimodal feature tokens using coarse-grained MERR samples, then applies fine-grained multimodal instruction tuning for emotion recognition and reasoning.

4 Experiments

Experiments evaluate Emotion-LLaMA across emotion reasoning, recognition, challenge benchmarks, qualitative cases, and ablations. Results consistently report strong performance, while ablations examine the contributions of instruction data and sample selection.

  • Evaluation Setup: Emotion-LLaMA is evaluated on MER2023, MER2024, DFEW, and EMER, using MERR pre-training followed by target-dataset fine-tuning.The evaluation covers both multimodal emotion recognition and emotion reasoning.
  • Multimodal Emotion Recognition: 59.37% WAR is achieved in zero-shot DFEW evaluation, and fine-tuning gives Emotion-LLaMA the highest UAR and WAR scores.The model is also described as maintaining more balanced predictions across emotion categories than other MLLMs.
  • Challenge Evaluation: Emotion-LLaMA achieves the highest F1 score across modalities on MER2023 by mapping audio and visual features into textual space.The paper attributes the improvement to integrating audio, visual, and textual information for capturing emotional-expression nuances.
  • Challenge Evaluation: 8.52% improvement in average accuracy and recall over GPT-4V is reported for MER-OV, with Emotion-LLaMA achieving the highest zero-shot score among participating large multimodal models.MER-OV permits diverse labels rather than restricting recognition to a small fixed label set.
  • Ablation Evaluation: 7.41 clue overlap and 5.56 label overlap are obtained with MERR coarse-grained instructions, while fine-grained instructions add gains of 0.42 and 0.69, respectively.The raw concatenation baseline yields the poorest performance, and sample selection experiments report the best performance after tuning on automatically annotated MERR data.

5 Ethics and Conclusion

The paper reports Emotion-LLaMA as a multimodal model for real-world emotion recognition and reasoning, while restricting MERR access through provider permissions and usage agreements.

  • Conclusion: Table 8 reports ablation results for different training strategies.
  • Ethics: MERR access is restricted to academic research under signed usage agreements and requires direct application to original data providers.The open-source release excludes source videos and contains emotion-description JSON files only.
  • Conclusion: Emotion-LLaMA achieved state-of-the-art performance through instruction tuning on comprehensive datasets including MER2023 and DFEW.
  • Acknowledgments: The study’s listed support includes national, institutional, industry, and U.S. government research funding.

A MERR Dataset Details

MERR covers nine emotion categories, including the challenging categories doubt and contempt, and uses rich multimodal descriptions to represent emotional states and context.

  • Emotion Categories: MERR contains nine categories: neutral, happy, angry, worried, surprise, sad, fear, doubt, and contempt.Doubt and contempt receive particular attention because they are often overlooked or difficult to distinguish.
  • Emotion Categories: The inclusion of doubt and contempt broadens MERR beyond the emotion categories commonly addressed in many datasets.
  • Multimodal Descriptions: MERR descriptions combine facial expressions, body language, vocal cues, and situational factors beyond simple categorical labels.These descriptions are intended to provide a comprehensive account of emotional state and context.

A.2 Data Filtering and Pseudo-Labeling

MERR is constructed through filtering, AU-based pseudo-labeling, multimodal description generation, refinement, and expert review to produce emotion-focused annotations.

  • Filtering and Pseudo-Labeling: MERR filtering identifies video segments with strong emotional expressions and assigns initial labels from facial cues.
  • Filtering and Pseudo-Labeling: OpenFace extracts and aligns faces, measures Action Unit presence and intensity, and applies AU combinations to assign rule-based emotion pseudo-labels.For example, AU05 plus AU26 is associated with surprise, while AU04 plus AU15 indicates sadness.
  • Multimodal Annotation: The annotation process collects visual expression, visual context, audio tone, and multimodal integration descriptions.Qwen-Audio is used to analyze and describe emotional content in speech.
  • Visual Expression Description: The emotional peak frame is selected by summing AU amplitudes across frames and choosing the frame with the highest total after checking pseudo-label consistency.The selected frame’s AUs are mapped to textual descriptions of facial movements.
  • Visual Objective Description: MiniGPT-v2 describes the visual scene from the emotional peak frame, including the environment, actions, and object interactions.
  • Dataset Refinement: Initial multimodal descriptions support pre-training alignment, while filtering removes contradictory or duplicate samples and leaves 4,487 refined samples for expert review.The initial coarse-grained stage uses 28,618 pseudo-labeled samples; four domain experts review the refined descriptions.

A.4 Data Statistics and Comparisons

MERR emphasizes short emotional clips, diverse and detailed multimodal annotations, and reasoning-oriented supervision, with human evaluation indicating generally strong annotation quality.

  • Video Statistics: Most MERR videos last 2–4 seconds, balancing emotional context with manageable processing and annotation requirements.
  • Dataset Comparison: MERR provides 28,618 coarse-grained and 4,487 fine-grained samples across nine categories, including doubt and contempt.
  • Dataset Comparison: MERR combines visual expressions, visual context, audio tone, and textual information with emotion-reasoning annotations.These annotations record rationale alongside labels to support explainable emotion recognition.
  • Annotation Quality: Five volunteers evaluated 180 fine-grained descriptions using five criteria scored from 0 to 5.The criteria cover visual, audio, and textual description accuracy plus reasoning-process and reasoning-result correctness.
  • Annotation Quality: 4.258 was the average human-evaluation score for MERR annotations.The Neutral category received the lowest score because its facial and audio cues are weaker and harder to annotate automatically.
  • Facial Features: Figure 4 analyzes AU frequency overall and the five most frequent AUs within each of nine expression categories.
  • Dataset Comparison: Figure 5 compares label distributions across DFEW, MER2023, EMER, and MERR datasets.
  • Instruction Data: The instruction-following format organizes visual, audio, and textual inputs with target instructions and outputs for Emotion-LLaMA training and evaluation.A worked example combines an excited voice, widened eyes, and a wide-open mouth to reason toward surprise.

B Training and Implementation Details

Emotion-LLaMA is trained to recognize and reason about emotions from multimodal inputs through instruction-guided, multitask learning. Its implementation combines modality-specific encoding, shared feature projection, coarse-to-fine training, and dataset-aware evaluation metrics.

  • Instruction Tuning: Instruction sets use task identifiers [emotion] and [reason] to prompt classification and multimodal emotional analysis.The instructions vary in phrasing to encourage robust understanding of each task.
  • Task Design: Emotion recognition and reasoning are trained in parallel through a multitask learning approach on multimodal inputs.Recognition predicts emotion labels, while reasoning analyzes multimodal cues and explains predicted labels.
  • Training Strategy: Coarse-to-fine training first uses coarse-grained annotations, then fine-grained descriptions to learn increasingly nuanced emotional representations.The strategy is intended to support detailed expression capture and improve generalization while reducing overfitting.
  • Multimodal Architecture: Modality-specific encoders capture complementary facial, temporal, and scene-level visual cues before multimodal integration.The visual encoders are MAE, VideoMAE, and EVA; MAE focuses on local facial details, while VideoMAE models temporal dynamics.
  • Multimodal Architecture: Encoder outputs are linearly projected into a shared embedding dimension and combined with learned task-specific special tokens.This unified representation supports cross-modal emotion reasoning within the language model.
  • Evaluation: DFEW evaluation uses WAR and UAR, while MER2023 uses WAF to account for class imbalance and balance performance across emotion categories.WAR emphasizes majority classes, UAR emphasizes minority classes, and WAF combines precision and recall through weighted category F-scores.

C.1.3 EMER Dataset

EMER evaluation assesses whether Emotion-LLaMA generates coherent emotional reasoning aligned with ground-truth descriptions. ChatGPT-based scoring measures clue overlap, contradictions, relevance, clarity, and coherence rather than relying only on classification metrics.

  • Evaluation Protocol: EMER evaluates the quality and coherence of Emotion-LLaMA’s generated emotional reasoning against ground-truth descriptions.The evaluation uses gpt-3.5-turbo-16k-0613 to score predicted reasoning.
  • Evaluation Protocol: ChatGPT scoring considers overlap with ground-truth reasoning, contradictory or irrelevant information, and explanation clarity and coherence.These guidelines produce a quantitative score for each generated reasoning.
  • Evaluation Rationale: The EMER approach measures whether explanations articulate relevant emotional cues, extending evaluation beyond accuracy or F-score.It is designed to provide a more nuanced assessment of emotional reasoning.

C.2.1 Confusion Matrices

Confusion matrices show strong recognition of common emotions but weaker performance on underrepresented categories. Class imbalance contributes to confusion between less frequent emotions and more prevalent categories.

  • Common Emotions: Emotion-LLaMA achieves high accuracy on “angry,” “happy,” and “sad,” with most samples correctly classified and minimal confusion among them.These common emotions have relatively clear facial, vocal, and linguistic cues.
  • Rare Emotions: “Disgust” and “fear” are harder to recognize because they have substantially fewer training samples than common emotions.The confusion matrices show higher confusion and lower performance for these categories.
  • Rare Emotions: A notable proportion of “disgust” and “fear” samples are misclassified as “angry” or “sad,” reflecting overlapping or ambiguous cues.The observed errors indicate difficulty distinguishing these less frequent emotional categories.
  • Limitations and Directions: Addressing class imbalance remains a crucial challenge requiring further methods to improve performance on less frequent emotions.Suggested directions include minority-class augmentation, class-weighted losses, and adaptive learning strategies.

C.2.2 Ablation Study of Hyperparameters

The ablation study examines learning-rate and training-data effects on Emotion-LLaMA. Results indicate that hyperparameter choice and data quantity materially affect performance, with diminishing gains near the full dataset.

  • Learning Rate: Learning-rate selection significantly affects Emotion-LLaMA’s convergence and generalization performance.The study seeks a range balancing convergence speed against overfitting or underfitting.
  • Learning Rate: Too high a learning rate can destabilize training, whereas too low a rate can slow convergence and increase potential overfitting.The ablation compares model performance across different learning rates.
  • Data Proportion: Increasing the training-data proportion generally improves performance by supporting more comprehensive and robust emotion representations.The ablation compares subsets ranging from smaller fractions to the full dataset.
  • Data Proportion: Performance gains gradually diminish as the data proportion approaches the full dataset, indicating a possible saturation point.This trade-off informs data requirements and computational-resource allocation.

C.2.3 MER2024 Challenge

The MER2024 Challenge evaluates multimodal emotion recognition under semi-supervised, noisy, and open-vocabulary settings. Emotion-LLaMA performs strongly in realistic noise conditions and shows greater generalization in open-vocabulary recognition.

  • Challenge Tracks: MER2024 includes MER-SEMI, MER-NOISE, and MER-OV tracks targeting unlabeled-data learning, noise robustness, and open-vocabulary emotion recognition.MER-SEMI provides over 110,000 unlabeled samples, while MER-NOISE simulates audio noise and image blur.
  • MER-NOISE: 84.52% WAF placed Emotion-LLaMA first in MER-NOISE, exceeding the second- and third-place teams by 1.47% and 1.65%.The system also used Emotion-LLaMA predictions as pseudo-labels to improve Conv-Attention.
  • MER-NOISE: MER-NOISE tests robustness to audio additive noise and image blur, which are prevalent in practical settings.The track encourages data augmentation and other methods to improve resilience under degraded modality conditions.
  • MER-OV: Emotion-LLaMA demonstrates greater generalization capabilities than other multimodal large language models on the MER-OV track.MER-OV uses diverse labels to represent subjective and ambiguous emotional states beyond a few discrete categories.

C.2.4 Detailed Scoring Cases

Detailed EMER scoring cases compare Emotion-LLaMA with other MLLMs on multimodal emotion reasoning. The examples emphasize its ability to combine subtle facial, vocal, and linguistic cues into coherent explanations.

  • Evaluation Setup: Emotion-LLaMA’s EMER reasoning cases are evaluated against ground-truth emotional descriptions using prompts designed to assess quality and coherence.Tables 19 and 20 compare its outputs with those of other multimodal large language models.
  • Case Analysis: Emotion-LLaMA identifies subtle facial expressions, raised-voice cues, and dialogue content to infer doubt or uncertainty coherently.The model’s reasoning aligns well with the ground-truth description in the cited case.
  • Case Analysis: The model combines emphatic tone and conversational context with visual cues to infer the underlying emotion accurately.This illustrates multimodal cue integration in the scoring examples.
  • Comparative Results: Other MLLMs struggle to produce accurate and coherent reasoning, sometimes focusing on irrelevant details or missing emotional nuances.The comparison highlights differences in explanation quality as well as emotion interpretation.

C.3.1 General Task Performance

The Emotion-LLaMA demo exposes multimodal emotion recognition, reasoning, and broader visual-language tasks through an interactive interface. It illustrates applications ranging from face detection and question answering to real-time emotion explanations.

  • General Tasks: Figure 12 demonstrates face detection and question answering as general tasks beyond core emotion understanding.Face localization can identify regions of interest for subsequent emotion recognition, while question answering uses visual and textual reasoning.
  • Emotion Tasks: Figure 13 showcases emotion recognition and reasoning from combinations of visual, audio, and textual inputs.The model generates natural-language explanations by considering facial expressions, vocal cues, and linguistic content.
  • Comparative Evaluation: Users can compare Emotion-LLaMA outputs with baseline models or human annotations to assess accuracy, coherence, and alignment with human judgments.The side-by-side comparison supports evaluation of model behavior in real-world scenarios.
  • Potential Applications: The demo highlights potential use in affective computing and mental-health assessment tools.The described applications involve recognizing and reasoning about emotions from multimodal data.
  • Demo Interface: The interactive demo accepts images, videos, and text and returns real-time emotion predictions and explanations.Figure 11 presents the web-based interface for multimodal emotion recognition and reasoning.
  • Scope and Claims: The paper’s introduction and abstract claims are judged to match the reported contributions and scope.The checklist justification specifically notes robust generalization by models instruction-tuned on the emotion-related dataset.
  • Limitations: The paper reports a limitation in instruction-following data generation: a small subset of samples contains mismatched emotional cues.The checklist associates this limitation with Section 3.1 and Figure 4.
Loading 2406.11161v2…