PaperlayerView exact revision on arXiv

Source-linked AI summary

Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study

Zengzhi Wang, Qiming Xie, Yi Feng, Zixiang Ding, Zinong Yang, Rui Xia

arXiv:2304.04339v2cs.CLcs.AI

TL;DR

The paper asks whether ChatGPT can serve as a universal sentiment analyzer despite limited knowledge of its capability boundaries. It evaluates ChatGPT across sentiment-analysis tasks, datasets, settings, baselines, prompting techniques, and human assessment, finding strong but uneven performance that can rival fine-tuned BERT while trailing domain-specific SOTA models.

  • Problem

    The paper investigates ChatGPT’s capability boundaries on understanding opinions, sentiments, and emotions, including polarity-shift and open-domain sentiment analysis.

  • Method

    The study evaluates ChatGPT across 7 sentiment-analysis tasks and 17 benchmark datasets in three settings, comparing it with fine-tuned BERT and SOTA models while testing prompting techniques and human evaluation.

  • Results

    ChatGPT can rival fine-tuned BERT and sometimes SOTA models, handles polarity shifts and open-domain scenarios well, but remains uneven across tasks and domains.

  • Takeaways & Limitations

    The findings support ChatGPT as a potentially universal, well-behaved sentiment analyzer relative to training specialized systems for each domain or dataset.

  • Takeaways & Limitations

    The evaluation is limited by possible data leakage, non-exhaustive prompt engineering, and primary evaluation of ChatGPT without equally powerful comparison models.

Abstract

from arXiv · show

Recently, ChatGPT has drawn great attention from both the research community and the public. We are particularly interested in whether it can serve as a universal sentiment analyzer. To this end, in this work, we provide a preliminary evaluation of ChatGPT on the understanding of \emph{opinions}, \emph{sentiments}, and \emph{emotions} contained in the text. Specifically, we evaluate it in three settings, including \emph{standard} evaluation, \emph{polarity shift} evaluation and \emph{open-domain} evaluation. We conduct an evaluation on 7 representative sentiment analysis tasks covering 17 benchmark datasets and compare ChatGPT with fine-tuned BERT and corresponding state-of-the-art (SOTA) models on them. We also attempt several popular prompting techniques to elicit the ability further. Moreover, we conduct human evaluation and present some qualitative case studies to gain a deep comprehension of its sentiment analysis capabilities.

1 Introduction

This preliminary study evaluates whether ChatGPT can function as a universal sentiment analyzer across sentiment, opinion, and emotion tasks. ChatGPT shows strong but uneven capabilities, handling polarity shifts and open-domain settings well while remaining weaker on some extraction tasks and domains.

  • Evaluation scope: The study evaluates ChatGPT on 7 sentiment analysis tasks spanning 17 benchmark datasets across standard, polarity shift, and open-domain settings.It compares ChatGPT with fine-tuned BERT and task-specific state-of-the-art models while testing prompting techniques.
  • Standard evaluation: ChatGPT shows impressive zero-shot performance on sentiment classification, rivaling fine-tuned BERT but trailing domain-specific fully supervised SOTA models.
  • Standard evaluation: ChatGPT achieves reasonable zero-shot performance on comparative sentence identification but struggles with comparative element extraction.
  • Standard evaluation: ChatGPT performs significantly better than fully supervised baselines on emotion cause extraction but worse on emotion-cause pair extraction.
  • Standard evaluation: Human evaluation suggests ChatGPT’s predictions on E2E-ABSA and CEE are more desirable than automatic metrics indicate, despite weaker information-extraction scores.
  • Polarity shift and open-domain evaluation: ChatGPT handles polarity shifts more accurately than fine-tuned BERT and generally performs strongly in open-domain sentiment analysis, though some domains remain limited.
  • Prompting: Few-shot prompting improves performance across tasks and domains, while self-consistency improves results; chain-of-thought prompting instead diminishes performance.

2 Background and Related Work

The paper situates ChatGPT within the rise of large language models and frames sentiment analysis as understanding opinions, sentiments, and emotions across domains and polarity shifts.

  • 2.1 Large Language Models: Large language models adapt to new tasks without training when given simple task instructions.Providing demonstration examples can make their predictions more accurate and controllable through in-context learning.
  • 2.1 Large Language Models: ChatGPT is a conversational system whose capabilities have prompted researchers to study its abilities and limitations across benchmarks.It was fine-tuned from GPT-3.5 using reinforcement learning from human feedback.
  • 2.2 Sentiment Analysis: Sentiment analysis identifies opinions, sentiments, and emotions in texts such as reviews, social media posts, and news articles.The field includes tasks ranging from polarity classification to fine-grained aspect analysis and comparative opinion extraction.
  • 2.2 Sentiment Analysis: The study focuses on whether ChatGPT can handle polarity shifts and open-domain sentiment analysis, two challenging problems for robust systems.Polarity shift concerns changing sentiment across context, time, or other texts, while open-domain analysis targets sentiment regardless of domain.

3 Evaluation

The evaluation compares ChatGPT with fine-tuned and state-of-the-art systems across seven tasks, benchmark datasets, and standard, polarity-shift, and open-domain settings. Results show strong zero-shot classification and robustness, but weaker extraction and domain-specific performance, with human judgments often exceeding exact-match metrics.

  • 3.1 Setup: The study evaluates seven sentiment analysis tasks across benchmark datasets using standard, polarity-shift, and open-domain settings.The tasks include SC, ABSC, E2E-ABSA, CSI, CEE, ECE, and ECPE.
  • 3.1 Setup: ChatGPT is compared with fine-tuned BERT baselines and task-specific SOTA models, while prompting techniques are used to elicit additional ability.Evaluation uses accuracy and macro F1 for classification, micro F1 for extraction, and task-specific F1 measures for emotion-cause analysis.
  • 3.2 Standard Evaluation: ChatGPT demonstrates competitive zero-shot sentiment classification, can rival fine-tuned BERT, but trails domain-specific fully supervised SOTA models.It achieves reasonable zero-shot performance on CSI but struggles on CEE.
  • 3.2 Standard Evaluation: ChatGPT shows higher ECE performance but lower ECPE performance, while often producing reasonable predictions that do not exactly match dataset annotations.The human evaluation suggests exact-match metrics can understate the desirability of its outputs on information extraction tasks.
  • 3.2 Standard Evaluation: Human evaluation increases ChatGPT’s average performance by 19% on E2E-ABSA and 42% on CEE compared with the original results.The evaluation indicates that many predictions judged poorly by exact-match metrics were plausible or aligned with human preferences.
  • 3.3 Polarity Shift Evaluation: In polarity-shift evaluation, ChatGPT is more robust than fine-tuned BERT, outperforming it on ABSC by 10% average accuracy and 8% average F1 score.Speculation is more challenging than negation, and fine-tuned BERT shows performance degradation across datasets relative to standard evaluation.
  • 3.4 Open Domain Evaluation: On open-domain ABSC, ChatGPT matches or exceeds multi-domain fine-tuned BERT on 7 out of 10 domains and surpasses it by 8% accuracy and 7% F1 score on average.However, performance remains poor in some domains, especially Twitter, finance, and METS-Cov.
  • 3.4 Open Domain Evaluation: The open-domain comparison indicates decent generalization, although its fairness is limited and some domain results remain poor.The study highlights social-media-related domains as areas where improving performance remains challenging.

4 Advanced Prompting Techniques

The paper tests advanced prompting to elicit ChatGPT’s sentiment-analysis capabilities, finding that few-shot prompting helps broadly while self-consistency helps and chain-of-thought hurts on the evaluated tasks.

  • 4 Advanced Prompting Techniques: The authors use ABSA tasks as the testbed for advanced prompting techniques.
  • 4.1 Few-shot Prompting: Few-shot prompting concatenates randomly selected training examples with the target input and evaluates 1, 3, 9, or 27 demonstrations on ABSC and ASPE.Three random seeds are used, and results are compared with fully supervised BERT and SOTA models.
  • 4.1 Few-shot Prompting: Figure 2 reports few-shot prompting results for ABSC and E2E-ABSA tasks.
  • 4.1 Few-shot Prompting: Few-shot prompting improves ABSC by 7% and 10% F1 on 14-Restaurant and 14-Laptop, respectively, with 27 demonstrations.
  • 4.1 Few-shot Prompting: Few-shot prompting significantly improves performance across tasks and datasets, surpassing fine-tuned BERT in some cases.Improvements on ASPE are present but relatively flat across the prompting curve.
  • 4.2 Chain-of-Thought and Self-Consistency: On E2E-ABSA, few-shot performance still lags far behind fine-tuned BERT, motivating experiments with more advanced prompting.
  • 4.2 Chain-of-Thought and Self-Consistency: Chain-of-thought added to standard few-shot prompting causes a noticeable performance drop rather than the expected gains.The authors speculate that the effect may depend on the evaluation tasks.
  • 4.2 Chain-of-Thought and Self-Consistency: Self-consistency clearly improves few-shot prompting regardless of whether chain-of-thought is included.

5 Conclusion

The conclusion presents ChatGPT as capable across several sentiment-analysis settings while emphasizing task- and domain-dependent boundaries. It also reports that prompting techniques can further induce its capabilities.

  • ChatGPT shows strong zero-shot abilities in sentiment classification, comparative opinion mining, and emotion cause analysis, sometimes matching fine-tuned BERT and domain-specific SOTA models.
  • Compared with fine-tuned BERT, ChatGPT handles polarity shifts more effectively and performs well in open-domain scenarios.
  • Experiments validate the effectiveness of popular prompting techniques for inducing ChatGPT’s sentiment-analysis capabilities.

Limitations

The paper’s evaluation is constrained by possible training-data leakage, limited prompt engineering, and coverage of only ChatGPT among powerful models.

  • The authors cannot determine whether test sets appeared in ChatGPT’s closed-source, large-scale unsupervised pre-training.They nevertheless observe deficiencies in domains such as medicine and social media.
  • The study does not conduct extensive prompt engineering, so better prompts may yield better performance.
  • The evaluation focuses mainly on ChatGPT because other powerful models are inaccessible through APIs or too costly to evaluate rigorously.
  • The evaluation is limited in scope because it does not include other equally powerful models.

Ethics Statement

The paper reports adherence to the ACL Ethics Policy and uses public benchmark data without human subjects or personally identifying information. Its application guidance distinguishes between annotation strictness, domain specificity, and resource availability.

  • The authors support the ACL Ethics Policy and use public benchmark datasets without human subjects or personally identifying information.
  • ChatGPT can be used with zero-shot or few-shot prompting when strict adherence to annotations or norms is not required.
  • Supervised specialized models remain preferable when strict annotation agreement, domain knowledge, or abundant labeled data is required.
  • ChatGPT is presented as viable for open-domain applications requiring generalization and promising for low-resource or zero-resource domains.

A.1 Prompts of ChatGPT

ChatGPT is prompted by task-specific instructions that it generates and selects through small-scale evaluation, while emotion cause tasks receive manually designed prompts because of their complexity.

  • ChatGPT generates three candidate task prompts, which are evaluated on a small example set before selecting the best one.For E2E-ABSA, the requested prompts target extracting aspect terms and sentiment polarity.
  • The final evaluation feeds each selected prompt and test example to ChatGPT and records its generated response.Responses are manually observed because they do not follow a fixed pattern.

A.2 Preparation of Polarity Shift Evaluation Datasets

The polarity-shift evaluation derives two datasets from SST-2 by detecting negation or speculation words and documents their seed words and dataset statistics.

  • SST-2-Neg and SST-2-Spec are derived from SST-2 by detecting whether sentences contain negation or speculation words.The seed words used for derivation are listed separately.
  • The evaluation materials include seed-word lists and statistics for the resulting datasets.

A.3 Examples on Human Evaluation

The study refines ChatGPT’s generated outputs before comparing them with references and baselines, acknowledging that this differs from leaving baseline outputs unrefined.

  • Manual output refinement addresses exact-match limitations when ChatGPT produces reasonable outputs that differ from references.
  • The authors acknowledge that refining ChatGPT outputs may appear unfair compared with unrefined baselines.
  • Different prompts have little effect on performance, and three repetitions show minimal deviation in results.The final evaluation is generally run once because API calls are costly.
  • Human-evaluation examples are provided from the E2E-ABSA task to illustrate ChatGPT’s responses.

A.4 Case Study for Standard Evaluation

The case studies examine ChatGPT’s task behavior across standard, polarity-shift, and open-domain settings. They show strong instruction following and some sentiment reasoning, alongside alignment, extraction, and domain-specific difficulties.

  • Standard evaluation: ChatGPT can produce reasonable aspect-sentiment analyses, but generated comparison relations may be reversed relative to annotations.Such semantic alignment differences can affect automatic evaluation.
  • Standard evaluation: ChatGPT follows emotion-cause instructions and can identify causes, but may omit the required clause ID.
  • Standard evaluation: ChatGPT can identify an emotion cause without the emotion clause being provided, although the selected cause may not be the most obvious.The prediction is nevertheless described as reasonable.
  • Polarity shift evaluation: ChatGPT accurately determines sentence-level sentiment under negation and speculation, but struggles with polarity for multiple parallel aspect terms.
  • Open-domain evaluation: In the books domain, unusual expressions and plot descriptions make sentiment prediction difficult, while reasonable but unannotated aspect-sentiment pairs affect E2E-ABSA evaluation.
  • Open-domain evaluation: In the medicine domain, ChatGPT can determine sentiment for a given aspect but struggles to extract aspect-sentiment pairs.
  • Prompting techniques: Chain-of-thought adds reasoning to few-shot demonstrations, while self-consistency aggregates sampled aspect-sentiment tuples using frequency thresholds.With N = 15, thresholds between 7 and 12 perform well in the reported experiments.
  • Additional evaluations and materials: The appendix includes comparisons with text-davinci-003, implicit sentiment analysis results, task-specific prompts, polarity-shift datasets, and human-evaluation examples.
Loading 2304.04339v2…