PaperlayerView exact revision on arXiv

Source-linked AI summary

Sentiment Analysis in the Era of Large Language Models: A Reality Check

Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, Lidong Bing

arXiv:2305.15005v1cs.CL

TL;DR

The paper examines how well large language models perform across sentiment analysis tasks, how they compare with specialized small models, and whether current evaluation practices remain suitable. Through systematic evaluation and varied prompting, it finds that LLMs handle simpler tasks well but struggle with complex structured sentiment analysis, while outperforming SLMs in few-shot settings; it also introduces SENTIEVAL for broader evaluation.

  • Problem

    Existing studies cover limited sentiment analysis tasks, datasets, models, and settings, leaving the true capacity of LLMs for sentiment analysis unclear.

  • Method

    The paper systematically evaluates LLMs across varied sentiment analysis tasks and compares them with specialized models using zero-shot and few-shot prompting strategies.

  • Results

    LLMs perform well on simpler zero-shot tasks and consistently outperform SLMs in few-shot learning, but lag behind specialized models on complex tasks requiring structured sentiment information.

  • Takeaways & Limitations

    SENTIEVAL provides a more comprehensive and realistic framework for evaluating LLM sentiment analysis abilities across a wider range of tasks.

  • Takeaways & Limitations

    Current evaluations often focus on specific tasks or datasets and use inconsistent prompts, limiting the reliability and breadth of conclusions about LLM sentiment analysis abilities.

Abstract

from arXiv · show

Sentiment analysis (SA) has been a long-standing research area in natural language processing. It can offer rich insights into human sentiments and opinions and has thus seen considerable interest from both academia and industry. With the advent of large language models (LLMs) such as ChatGPT, there is a great potential for their employment on SA problems. However, the extent to which existing LLMs can be leveraged for different sentiment analysis tasks remains unclear. This paper aims to provide a comprehensive investigation into the capabilities of LLMs in performing various sentiment analysis tasks, from conventional sentiment classification to aspect-based sentiment analysis and multifaceted analysis of subjective texts. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets. Our study reveals that while LLMs demonstrate satisfactory performance in simpler tasks, they lag behind in more complex tasks requiring deeper understanding or structured sentiment information. However, LLMs significantly outperform SLMs in few-shot learning settings, suggesting their potential when annotation resources are limited. We also highlight the limitations of current evaluation practices in assessing LLMs' SA abilities and propose a novel benchmark, \textsc{SentiEval}, for a more comprehensive and realistic evaluation. Data and code during our investigations are available at \url{https://github.com/DAMO-NLP-SG/LLM-Sentiment}.

1 Introduction

This study conducts a reality check of LLMs for sentiment analysis across diverse tasks, comparing them with specialized smaller models and reassessing evaluation practices. LLMs perform satisfactorily on simpler tasks and excel in few-shot settings, but remain weaker on complex structured tasks and face context-length constraints.

  • 1 Introduction: The study evaluates LLMs across sentiment classification, aspect-based sentiment analysis, and multifaceted subjective-text analysis.It considers 13 sentiment analysis tasks across 26 datasets.
  • 1 Introduction: LLMs are compared with smaller domain-specialized models in both zero-shot and few-shot settings.The comparison includes Flan-T5, Flan-UL2, ChatGPT, text-davinci-003, and T5 baselines.
  • 1 Introduction: LLMs perform satisfactorily on simple zero-shot tasks but lag behind specialized models on complex tasks requiring deep understanding or structured sentiment information.Performance improves with larger models, but a gap remains.
  • 1 Introduction: In few-shot learning, LLMs consistently outperform smaller models when annotated data are limited.This supports their potential use when annotation resources are scarce.
  • 1 Introduction: Few-shot use of LLMs remains constrained by the context length available for demonstrations.The paper identifies this limitation as requiring attention for effective utilization.
  • 1 Introduction: The proposed SENTIEVAL benchmark broadens evaluation across SA tasks, varied instructions, and more realistic natural-language settings.It addresses task, dataset, and prompt inconsistencies in existing evaluations.

2 Background

Sentiment analysis covers broad computational study of sentiment, opinion, emotion, and subjective feeling, with applications ranging from reviews to social media. Existing LLM evaluations are fragmented, motivating a systematic reality check across tasks, datasets, models, and settings.

  • 2 Background: Sentiment analysis studies opinions, sentiments, emotions, and other subjective feelings through computational methods.Its applications include product-review analysis and understanding social-media posts.
  • 2 Background: Sentiment analysis includes sentiment classification, fine-grained aspect-based analysis, and multifaceted subjective-text tasks.Examples include hate-speech, irony, and comparative-opinion detection.
  • 2 Background: LLMs use large-scale pretraining and techniques such as instruction tuning and reinforcement learning from human feedback.These models can perform tasks without supervised training in zero-shot or few-shot settings.
  • 2 Background: Prior LLM sentiment-analysis studies use limited task coverage and differing datasets or experimental designs, leaving overall capability unclear.Existing work includes zero-shot comparisons, ChatGPT studies, and LLM-generated weak labels.

3 Investigated Tasks and Datasets

The evaluation spans sentiment classification, aspect-based sentiment analysis, and multifaceted subjective-text analysis using varied datasets and metrics. It includes conventional classification, structured aspect-level extraction, and specialized phenomena such as implicit sentiment, hate speech, and stance.

  • 3 Investigated Tasks and Datasets: The study categorizes sentiment-analysis tasks into sentiment classification, aspect-based sentiment analysis, and multifaceted subjective-text analysis.The evaluation samples at most 500 test examples per dataset for balance.
  • 3.1 Sentiment Classification: Sentiment classification assigns predefined sentiment classes to documents, sentences, or aspects using accuracy.Datasets include movie reviews, business reviews, and social-media posts with binary to five-class labels.
  • 3 Investigated Tasks and Datasets: The investigated datasets span domains such as laptops, restaurants, reviews, tweets, and computer science comparisons.Task-specific metrics include accuracy, macro-averaged F1, and Micro-F1.
  • 3.2 Aspect-based Sentiment Analysis: The compound ABSA tasks extract increasingly complete sentiment structures, from aspect-polarity pairs to aspect-opinion-sentiment triplets and category-augmented quadruples.UABSA, ASTE, and ASQP use exact-match Micro-F1 evaluation.
  • 3.3 Multifaceted Analysis of Subjective Text: MAST covers specialized subjective phenomena including implicit sentiment, hate speech, irony, stance, and comparative opinion mining.These tasks extend sentiment analysis beyond positive-versus-negative classification.

4 Evaluations

The evaluation compares directly inferred LLMs with domain-trained SLMs across zero-shot and few-shot sentiment tasks, examining performance, prompt sensitivity, and evaluation validity. LLMs perform well on simpler tasks and in few-shot settings, but remain weaker on structured sentiment extraction and show task-dependent prompt and context effects.

  • Models and Baselines: The study evaluates open-source Flan-T5 and Flan-UL2, GPT-3.5 models, and domain-trained T5 baselines in zero-shot and few-shot settings.The SLM baseline is trained on domain-specific data for each dataset, while LLMs perform downstream inference without specific training.
  • Zero-shot Results: ChatGPT reaches 97% of T5’s prediction on SC tasks and 83% on MAST tasks on average, matching specialized models on simpler zero-shot problems.For harder tasks, ChatGPT scores 52.4 versus 65.6 accuracy on Yelp-5 and 72.80 versus 80.35 on comparative opinion mining.
  • Zero-shot Results: Scaling is not consistently beneficial: Flan-UL2 sometimes matches or exceeds larger models, possibly because of its model size and large-scale instruction tuning.Larger models generally outperform smaller ones in some comparisons, but parameter count alone does not guarantee better results.
  • Zero-shot Results: LLMs struggle with fine-grained structured sentiment: text-003 reaches only around 54% of a fine-tuned T5 model’s performance on ABSA despite being more than 200 times larger.Flan-T5 and Flan-UL2 achieve no notable performance across the ABSA datasets, while text-003 and ChatGPT remain below fine-tuned SLMs.
  • Analysis of Sensitivity on Prompt Design: Prompt choice has less effect on many SC datasets but can substantially change performance on structured ABSA tasks and produce SC outliers.The analysis varies performance across five prompts per task using ChatGPT-generated prompts.
  • Human Evaluation: Human evaluation finds better results than automated evaluation, but even relaxed formatting requirements leave fine-grained sentiment performance unsatisfactory.Generative outputs can be invalid when they do not follow the required structured format, especially for ABSA.
  • Few-shot Results: Across 1-shot, 5-shot, and 10-shot settings, ChatGPT and Flan-UL2 outperform T5 in almost all cases, especially on structured ABSA tasks.Even at 10 shots, ChatGPT provides a baseline requiring T5 to use nearly five to ten times more data for comparable performance; larger LLM shot counts can hit context limits.
  • Few-shot Results: Additional shots help LLMs greatly on ABSA but provide limited or negative gains on some simpler, stance, and comparative tasks as contexts grow.The authors attribute some declines to overly long contexts that may mislead LLMs, making extra examples task-dependent rather than universally beneficial.

5 SENTIEVAL Benchmark

The paper identifies weaknesses in current sentiment-analysis evaluations and proposes SENTIEVAL, a unified benchmark using varied natural-language instructions and fixed prompts for broader, more reliable assessment.

  • Rethinking SA Capability Evaluation: Current evaluations often focus on narrow task or dataset subsets, limiting coverage of LLM sentiment-analysis capabilities.
  • Rethinking SA Capability Evaluation: Prompt variation can substantially affect ChatGPT performance, complicating fair and stable comparisons across models and studies.
  • SENTIEVAL: Construction: SENTIEVAL unifies diverse sentiment-analysis tasks to assess overall proficiency rather than isolated capabilities.
  • SENTIEVAL: Construction: SENTIEVAL uses natural-language instructions in varied styles and diverse fixed prompts to better reflect interaction and stabilize comparisons.
  • SENTIEVAL: Re-evaluate: On SENTIEVAL, models are evaluated by exact-match label scores, while task-specific results are also reported for investigation.
  • SENTIEVAL: Re-evaluate: ChatGPT establishes a strong baseline, but LLMs still have substantial room to improve, particularly on ABSA and MAST tasks.

6 Discussions

The discussion finds that LLMs are effective for simple sentiment classification and few-shot settings, but weaker on structured sentiment tasks and constrained by several unresolved challenges. It also argues that model scale alone does not guarantee better performance and that evaluation practices need broader coverage.

  • LLMs can effectively handle binary or trinary sentiment classification, even zero-shot, with limited sensitivity to prompt design.Their performance can match or surpass fine-tuned smaller language models.
  • When annotation resources are scarce, LLMs outperform SLMs in few-shot learning, although restricted context length limits document-level applications.The discussion notes that SLMs may be more suitable for some document-level tasks.
  • LLMs lag behind SLMs on structured sentiment outputs such as aspect-based sentiment analysis, with performance varying substantially across prompts.This weakness appears in both automatic and human evaluations.
  • Larger models do not always perform better: Flan-UL2 can be comparable to GPT-3.5 models despite being much smaller.The authors suggest that instruction-tuning a reasonably sized model may suffice for practical sentiment analysis.
  • SA Challenges for LLMs: Remaining challenges include interpreting sarcasm, irony, humor, and culturally specific expressions, extracting fine-grained sentiment, and adapting to evolving language.Examples include ironic expressions on Chinese social media, structured aspect-level information, and rapidly changing social-media idioms.

7 Conclusions

The paper systematically evaluates LLMs across sentiment analysis tasks and concludes that they perform well on simpler tasks but struggle with more complex ones. LLMs consistently outperform SLMs in few-shot settings, while SENTIEVAL addresses limitations in existing evaluation practices.

  • The study systematically evaluates LLM capabilities across a range of sentiment analysis tasks.
  • LLMs perform well on simpler tasks in zero-shot settings but struggle with more complex sentiment analysis tasks.
  • LLMs consistently outperform SLMs in few-shot learning, indicating potential when annotation resources are scarce.
  • SENTIEVAL is introduced as a more comprehensive and realistic benchmark in response to limitations of current evaluation practices.
  • Traditional sentiment analysis tasks can reach near-human performance, but comprehensive understanding of subjective feelings remains unresolved.

A.1 Prompts for Each SA Task

This appendix presents one-shot prompts for the investigated sentiment analysis tasks, specifying task instructions, allowed labels, output formats, and demonstration examples. The prompts cover sentiment classification and aspect-based sentiment tasks across multiple datasets.

  • The appendix provides one-shot prompts for each investigated sentiment analysis task.The prompt materials identify the task and dataset before presenting instructions and examples.
  • Sentiment Classification: Sentiment classification prompts require assigning predefined labels and returning only the label.Examples cover binary, trinary, and five-class sentiment settings across movie, review, and social-media datasets.
  • Sentiment Classification: The demonstrations pair input sentences with sentiment labels to illustrate the required classification behavior.Examples include positive, negative, neutral, very positive, and very negative labels.
  • Aspect-Based Sentiment: Aspect sentiment prompts ask for sentiment toward a specified aspect, using negative, neutral, or positive labels.The examples target aspects such as Office, webcam, touchpad, word processor, classical furniture, curry flavor, and garlic knots.
  • Aspect-Based Sentiment: Unified aspect-based prompts require returning all aspect–sentiment pairs as a Python list of tuples.The prompt instructs the model to return an empty list when no aspect-sentiment pairs are present.
Loading 2305.15005v1…