Source-linked AI summary
EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysis
Zhiwei Liu, Kailai Yang, Tianlin Zhang, Qianqian Xie, Sophia Ananiadou
TL;DR
Affective-analysis models often cover classification while omitting regression, and downstream datasets lack comprehensive emotional annotations. The paper introduces instruction-tuned EmoLLMs with AAID and evaluates them on the 14-task AEB benchmark, where they outperform open-source LLMs and exceed ChatGPT and GPT-4 in most tasks. The authors conclude that EmoLLMs can serve as comprehensive affective annotation tools, while noting their current English-only scope and possible real-world bias.
Problem
Existing affective LLMs overlook sentiment-strength and emotion-intensity regression, while downstream datasets lack comprehensive emotion-related annotations.
Method
The paper fine-tunes open-source LLMs with the 234K-sample AAID dataset and evaluates them on AEB, a benchmark containing 14 affective analysis tasks.
Results
EmoLLMs outperform all other open-source LLMs and exceed ChatGPT and GPT-4 in 7 regression tasks and 4 classification tasks.
Takeaways & Limitations
EmoLLMs can serve as comprehensive affective annotation tools across data from different platforms and sources.
Takeaways & Limitations
Current EmoLLMs are limited to English text and may exhibit biases when applied to real-world textual content.
Abstract
from arXiv · showhide
Sentiment analysis and emotion detection are important research topics in natural language processing (NLP) and benefit many downstream tasks. With the widespread application of LLMs, researchers have started exploring the application of LLMs based on instruction-tuning in the field of sentiment analysis. However, these models only focus on single aspects of affective classification tasks (e.g. sentimental polarity or categorical emotions), and overlook the regression tasks (e.g. sentiment strength or emotion intensity), which leads to poor performance in downstream tasks. The main reason is the lack of comprehensive affective instruction tuning datasets and evaluation benchmarks, which cover various affective classification and regression tasks. Moreover, although emotional information is useful for downstream tasks, existing downstream datasets lack high-quality and comprehensive affective annotations. In this paper, we propose EmoLLMs, the first series of open-sourced instruction-following LLMs for comprehensive affective analysis based on fine-tuning various LLMs with instruction data, the first multi-task affective analysis instruction dataset (AAID) with 234K data samples based on various classification and regression tasks to support LLM instruction tuning, and a comprehensive affective evaluation benchmark (AEB) with 14 tasks from various sources and domains to test the generalization ability of LLMs. We propose a series of EmoLLMs by fine-tuning LLMs with AAID to solve various affective instruction tasks. We compare our model with a variety of LLMs on AEB, where our models outperform all other open-sourced LLMs, and surpass ChatGPT and GPT-4 in most tasks, which shows that the series of EmoLLMs achieve the ChatGPT-level and GPT-4-level generalization capabilities on affective analysis tasks, and demonstrates our models can be used as affective annotation tools.
1 INTRODUCTION
The paper addresses incomplete affective analysis by introducing instruction-tuned models, a multi-task dataset, and a broad evaluation benchmark. EmoLLMs outperform other open-source models and exceed ChatGPT and GPT-4 across most evaluated tasks.
- 1 INTRODUCTION: Existing LLM-based affective models mainly address classification, overlooking sentiment strength and emotion intensity regression.The paper identifies this as a limitation of prior affective analysis systems.
- 1 INTRODUCTION: Existing downstream datasets often lack comprehensive emotion labels, leading researchers to rely on tools or transfer learning that cover only one affective aspect.The paper cites VADER, TextBlob, and transferred models as examples of limited annotation coverage.
- 1 INTRODUCTION: The authors construct AAID with 234K samples and AEB with 14 affective analysis tasks spanning regression and classification.AAID supports instruction tuning, while AEB evaluates generalization across datasets from varied platforms and sources.
- 1 INTRODUCTION: The paper proposes EmoLLMs as open-source instruction-following models for comprehensive multi-task affective analysis.The models are fine-tuned with AAID to handle classification and regression tasks.
- 1 INTRODUCTION: EmoLLMs exceed ChatGPT and GPT-4 in 7 regression tasks and 4 classification tasks on AEB.The authors report comparable capabilities with ChatGPT and GPT-4 across most affective analysis tasks.
2 RELATED WORK
Prior work includes affective-analysis tools, PLM-based models, and open-source instruction-following LLMs. The paper positions EmoLLMs as the first open-source LLM series designed for comprehensive multi-task affective analysis, including regression tasks.
- 2 RELATED WORK: Existing affective tools such as VADER and TextBlob are convenient but have limited effectiveness in sentiment analysis.Prior studies also combine PLMs with other architectures for short and simple text sentiment analysis.
- 2 RELATED WORK: Prior PLM- and LLM-based systems focus on individual affective aspects and lack sentiment-strength and emotion-intensity prediction.The related work specifically identifies missing affective regression capabilities.
- 2 RELATED WORK: Closed-source ChatGPT and GPT-4 motivate efforts to develop open-source instruction-following LLMs such as LLaMA, OPT, BLOOM, and Falcon.The paper situates EmoLLMs among domain-specific instruction-tuned systems.
- 2 RELATED WORK: EmoLLMs are presented as the first open-source LLM series for comprehensive multitask affective analysis.This scope distinguishes the work from systems targeting individual affective-analysis aspects.
3 METHODS
The paper frames affective analysis as a unified generative instruction-tuning problem spanning classification and regression tasks. It builds AAID and AEB to train and evaluate models across affective tasks and domains.
- 3.1 Problem Formulation: EmoLLMs treat affective analysis as a generative task covering sentiment polarity and strength, emotion classification, and intensity prediction.Each task uses context-target pairs containing a task description, target text, query, and answer, and the merged dataset optimizes conditional language modeling.
- 3.2 Instruction Tuning Data Building: AAID is built from SemEval-2018 Task 1: Affect in Tweets and covers five subtasks spanning emotion intensity, sentiment strength, sentiment classification, and emotion classification.The subtasks include emotion intensity regression, ordinal emotion-intensity classification, valence regression, ordinal valence classification, and emotion classification.
- 3.2 Instruction Tuning Data Building: AAID converts raw affective examples into instruction-tuning samples using task-specific templates, with outputs adjusted for classification, sentiment strength, emotion classification, or emotion intensity.Ten different task instructions are used for each task to augment the training and validation sets.
- 3.3 AEB: Affective Evaluation Benchmark Building: AEB evaluates models with task prompts across regression, sentiment classification, and emotion classification settings, using dataset-specific task formats.The benchmark statistics distinguish regression intensity ranges from sentiment and emotion classification category counts.
4 EVALUATION
The evaluation compares EmoLLMs with fine-tuned PLMs, zero-/few-shot LLMs, and other instruction-tuned models across AEB-1 and AEB-2. EmoLLaMA achieves the strongest overall affective-analysis performance, including on tasks and domains beyond its training data.
- AEB-1 results: EmoLLaMA-chat-13B outperforms all other LLMs and surpasses the AEB-1 top-ranked system in the first four tasks.It improves over the top-ranked system by 3.2% on EI-reg and 6.8% on EI-oc.
- AEB-1 results: 3.2% and 6.8% are EmoLLaMA-chat-13B’s improvements over the top-ranked system on EI-reg and EI-oc, respectively.The reported scores are 0.831 versus 0.799 for EI-reg and 0.763 versus 0.695 for EI-oc.
- AEB-1 results: EmoLLaMA-chat-13B provides more comprehensive affective performance than the other instruction-tuned EmoLLMs, although EmoT5 leads E-c and EmoOPT narrowly leads some regression tasks.EmoT5 scores 0.568 versus EmoLLaMA’s 0.545 on E-c macro-F1, while EmoOPT scores 0.887 versus 0.886 on V-reg.
- AEB-2 results: On AEB-2, EmoLLaMA outperforms ChatGPT, GPT-4, and non-fine-tuned LLMs in most regression tasks, exceeding GPT-4 by over 10% on the first four regression tasks.It performs less well than ChatGPT and GPT-4 on SST and Emobank-Arousal.
- AEB-2 results: EmoLLMs transfer well beyond Twitter-trained data: they perform strongly across other platforms and domains, while VADER and TextBlob perform significantly worse.On AEB-2, only TDT and V-Tweet are sourced from Twitter; the remaining datasets come from different platforms and domains.
5 CONCLUSION
The paper introduces EmoLLMs, AAID, and AEB for comprehensive affective analysis, reporting strong performance and ChatGPT- and GPT-4-level generalization on unseen tasks.
- EmoLLMs achieve SOTA performance against other open-sourced LLMs and show ChatGPT- and GPT-4-level generalization across unseen affective analysis tasks.
- EmoLLMs comprise comprehensive affective analysis models and annotation tools, alongside the AAID dataset and AEB benchmark.
- Instruction tuning can greatly enhance LLM performance in a specific domain, enabling EmoLLMs to surpass ChatGPT and GPT-4 in most tasks.
6 DISCUSSIONS
EmoLLMs support practical applications by automatically providing multiple emotional signals, but their current scope is constrained by dataset coverage and English-only textual input.
- EmoLLMs can automatically provide multiple emotional signals for applications including misinformation detection and mental-health analysis.
- Internet and social-media datasets differ from other textual content in expression, format, and style, creating potential real-world biases.
- Current EmoLLMs are limited to English text and lack content from other languages and modalities.