PaperlayerView exact revision on arXiv

Source-linked AI summary

EmoLLM: Multimodal Emotional Understanding Meets Large Language Models

Qu Yang, Mang Ye, Bo Du

arXiv:2406.16442v2cs.CV

TL;DR

MLLMs remain limited in interpreting nuanced multimodal emotions, motivating a benchmark and model designed for emotional understanding. EmoBench provides diverse multimodal training and evaluation instructions, while EmoLLM combines multi-perspective visual processing with guided reasoning. EmoLLM achieves a 12.1% average improvement across multiple foundation models on EmoBench, although the benchmark answers and model performance remain vulnerable to limited annotation diversity, hallucination, and prompt sensitivity.

  • Problem

    MLLMs struggle to interpret nuanced multimodal emotions, especially when visual, auditory, and textual cues must be integrated.

    Source: §1 · p. 1

  • Method

    EmoBench supplies large-scale emotional instructions, while EmoLLM uses Multi-perspective Visual Projection and EmoPrompt for multimodal emotional understanding.

    Source: §1 · p. 2

  • Results

    12.1% average improvement over baselines is reported across multiple foundation models on EmoBench.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The work supports deeper and more nuanced emotional understanding in MLLMs through benchmark-based enhancement and evaluation.

    Source: §1 · p. 2, §6 · p. 9

    EmoBench answers may lack diversity because they were generated by GPT-4 and automated scripts, while hallucination and prompt sensitivity may affect EmoLLM.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multi-modal large language models (MLLMs) have achieved remarkable performance on objective multimodal perception tasks, but their ability to interpret subjective, emotionally nuanced multimodal content remains largely unexplored. Thus, it impedes their ability to effectively understand and react to the intricate emotions expressed by humans through multimodal media. To bridge this gap, we introduce EmoBench, the first comprehensive benchmark designed specifically to evaluate the emotional capabilities of MLLMs across five popular emotional tasks, using a diverse dataset of 287k images and videos paired with corresponding textual instructions. Meanwhile, we propose EmoLLM, a novel model for multimodal emotional understanding, incorporating with two core techniques. 1) Multi-perspective Visual Projection, it captures diverse emotional cues from visual data from multiple perspectives. 2) EmoPrompt, it guides MLLMs to reason about emotions in the correct direction. Experimental results demonstrate that EmoLLM significantly elevates multimodal emotional understanding performance, with an average improvement of 12.1% across multiple foundation models on EmoBench. Our work contributes to the advancement of MLLMs by facilitating a deeper and more nuanced comprehension of intricate human emotions, paving the way for the development of artificial emotional intelligence capabilities with wide-ranging applications in areas such as human-computer interaction, mental health support, and empathetic AI systems. Code, data, and model will be released.

1 Introduction

Existing MLLMs perform poorly on nuanced multimodal emotions, motivating EmoBench and EmoLLM. EmoBench supplies large-scale training and evaluation data, while EmoLLM combines multi-perspective visual features with guided emotional reasoning.

Motivation

  • MLLMs remain unsatisfactory at interpreting nuanced multimodal emotions, especially fear and anger, partly because emotional training datasets are scarce.Text-based emotional analysis also struggles with multimodal tasks requiring visual, auditory, and textual cues.Source: §1 · p. 1

EmoBench

  • EmoBench provides approximately 287,000 multimodal instructions spanning universal emotional and application tasks for training and evaluation.Its application tasks include hate, sarcasm, and humor detection.Source: §1 · p. 2, Figure 1 · p. 2

EmoLLM

  • EmoLLM uses Multi-perspective Visual Projection to jointly model object content and relationships, extracting features suited to emotional tasks.The approach considers multiple viewpoints over visual information and represents object relationships as graphs.Source: §1 · p. 2
  • EmoPrompt guides reasoning with GPT-4V-validated task examples because inaccurate initial observations can undermine subsequent emotional inference.The prompt selects a stored example whenever guidance is required.Source: §1 · p. 2

Results

  • 12.1% average improvement over baseline models is reported across multiple foundation models on EmoBench.The experiments are described as extensive evaluations demonstrating substantial improvement.Source: §1 · p. 2

2 Related Works

Prior work provides multimodal emotion, intention, and language resources, but MLLMs still underperform on complex emotional reasoning. The paper positions EmoBench as an emotional instruction-tuning dataset addressing this limitation.

2.1 Multi-modality Emotional Tasks and Methods

  • Multimodal emotion datasets progressed from scripted audiovisual interactions toward more naturalistic and diverse sources, while intention datasets span forums, social media, and television clips.Examples include IEMOCAP, CMU-MOSEI, MELD, Emotic, GoEmotions, and MIntRec.Source: §2.1 · p. 3
  • Existing multimodal emotion methods use cross-modal attention, modality-invariant and modality-specific representations, or adversarial learning for fusion and prediction.The surveyed methods include MulT, MISA, and MFSA.Source: §2.1 · p. 3
  • A GPT-based approach converts multimodal emotion tasks into text recognition but relies on MLLM preprocessing and is described as unsuitable for practical applications.This limitation motivates a more direct multimodal approach.Source: §2.1 · p. 3

2.2 Multi-modality Large Language Models

  • MLLMs extend LLMs with visual and audio inputs for tasks such as captioning and VQA, yet remain weak on complex emotions and reasoning-heavy categories.The paper attributes this weakness primarily to insufficient relevant data and specialized models.Source: §2.2 · p. 3
  • Figure 2 depicts EmoBench as a diverse-content benchmark whose generated instructions support improving multimodal emotional understanding.Its pipeline includes template definition, template-set generation, and instruction generation.Source: Figure 2 · p. 4

3 EmoBench

EmoBench is constructed from diverse emotion and intention datasets and organized into universal and application-oriented tasks. A GPT-assisted pipeline converts these data into multimodal question-answering instructions.

3 EmoBench

  • EmoBench sub-samples from eight datasets and organizes the resulting examples into conversations using generative models and automated scripts.The benchmark is introduced as a large-scale resource centered on emotional dimensions.Source: §3 · p. 4

3.1 Data Preparation and Task Definition

  • Universal emotional tasks cover multimodal emotion and intention understanding, including both closed-set labels and open-set category prediction.Open-set evaluation removes the requirement for a predefined label list.Source: §3.1 · p. 4
  • Emotional application tasks target socially relevant humor, hate, and sarcasm detection.These subtasks are selected for frequent or important industry and social-media applications.Source: §3.1 · p. 4

3.2 Instruction Construction

  • Universal-task instructions use generated question bases, task label sets, multimodal data placeholders, and ground-truth labels in a visual-question-answering format.Application subtasks instead use binary questions such as whether the data contains sarcasm.Source: §3.2 · p. 4
  • Figure 3 shows EmoLLM processing user queries and multimodal inputs through an LLM and modality-specific encoders, with multi-perspective projection combining visual representations.The projection extracts features from visual tokens and connects cluster centers in a graph.Source: Figure 3 · p. 5

4 Methodology

EmoLLM combines multimodal encoding with Multi-perspective Visual Projection and EmoPrompt reasoning to support emotional understanding. Its methodology represents visual content through object-level and relational cues, then guides emotion reasoning with GPT-4V-generated demonstrations.

4.1 Model Overview

  • EmoLLM uses modality encoders, Multi-perspective Visual Projection, and EmoPrompt as its three major methodological components.The architecture adds visual and audio encoders, graph-based visual representations, and GPT-4V-generated reasoning prompts.Source: §4.1 · p. 5

4.2 Modality Encoding

  • The modality encoding stage uses CLIP-VIT-L/14 for images and video frames, WHISPER-BASE with an MLP for audio, and LLaMA2-7B for text.The encoded modality features are transformed into a shared dimension for multimodal processing.Source: §4.2 · pp. 5–6

4.3 Multi-perspective Visual Projection

  • MVP combines content-based object information with relation-based connections to capture local and global emotional cues in visual inputs.It clusters visual tokens, constructs a graph over cluster centers, applies graph convolutions, and combines the resulting relation representation with content features.Source: §4.3 · pp. 6–7

4.4 EmoPrompt Reasoning

  • EmoPrompt uses visual data and labels with GPT-4V to generate logical chains that guide EmoLLM toward correct emotional reasoning.Hundreds of task-specific reasoning examples are collected as demonstrations for the model’s reasoning process.Source: Figure 4 · p. 7, §4.4 · pp. 7–8

4.3 Multi-perspective Visual Projection

  • MVP clusters visual tokens with DPC-KNN, expands video tokens across critical events, and aggregates multiscale features before projection.For videos, frame representations are clustered to identify events and preserve temporal understanding.Source: §4.3 · pp. 6–7

4.4 EmoPrompt Reasoning

  • EmoPrompt addresses the dependence of step-by-step emotion inference on potentially inaccurate initial observations by supplying demonstrations of correct reasoning.The demonstrations are produced from emotional-task samples paired with ground-truth labels.Source: §4.4 · pp. 7–8

5 Experiments

Experiments evaluate EmoLLM against 7B-scale baselines and state-of-the-art MLLMs, then analyze its visual projection, prompt quantity, and training-strategy choices. EmoLLM shows superior emotional understanding, while ablations identify settings that improve performance and balance efficiency.

5.1 Experimental Setup

  • Experiments use CLIP and WHISPER encoders with a 7B Vicuna-v1.5 foundation model, and final results are averaged across three random seeds.Training uses EmoBench data, five epochs, batch size 16, AdamW, cosine scheduling, and LoRA on four RTX 4090 GPUs.Source: §5.1 · p. 8

5.2 Main Results

  • EmoLLM achieves superior performance to same-scale 7B baselines across six EmoBench emotional subtasks.The evaluation covers close-set and open-set emotion classification, intention recognition, and three emotional application tasks.Source: §5.2 · p. 8
  • EmoLLM outperforms GPT-4V, Gemini-1.0, and Gemini-1.5 on close-set and open-set emotion classification while using fewer parameters.This comparison uses an image-only EmoBench subset containing six emotion categories.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 Ablation Studies

  • 64.06% accuracy is achieved when the Multi-perspective Visual Projection hyperparameter τ is set to 0.1.Performance declines as τ increases from 0.1 toward 0.5, indicating sensitivity to this setting.Source: §5.3 · p. 8, Figure 5 · p. 9
  • 64.06% accuracy is reached when all EmoPrompts are used, while gains diminish beyond 600 prompts.The result suggests a range that balances computational efficiency and emotional understanding.Source: §5.3 · p. 9, Figure 5 · p. 9
  • Sequential training—objective-task fine-tuning followed by emotional-task fine-tuning—produces the best emotional-understanding results.The compared strategies are emo, mixed objective-and-emotional training, and sequential training.Source: §5.3 · p. 9, Table 4 · p. 9

6 Conclusion

The paper introduces EmoBench and EmoLLM to improve and evaluate multimodal emotional understanding. It reports substantial gains over baselines while identifying limited answer diversity and LLM vulnerabilities as limitations.

6 Conclusion

  • EmoBench evaluates and enhances MLLM emotional understanding, while EmoLLM combines Multi-perspective Visual Projection with EmoPrompt.The benchmark and model target emotional understanding in multimodal data.Source: §6 · p. 9
  • 12.1% average improvement over baselines is demonstrated across multiple foundation models on EmoBench.The conclusion attributes this result to extensive EmoBench experiments.Source: §6 · p. 9
  • EmoBench answers may lack diversity because they were generated by GPT-4 and automated scripts rather than human annotators.The paper suggests combining automation with manual labeling to increase diversity.Source: §6 · p. 9
  • LLM hallucination and prompt sensitivity may affect EmoLLM’s performance.These vulnerabilities are identified as an inherent limitation of LLMs used by the model.Source: §6 · p. 9

Related papers

Loading 2406.16442v2…