PaperlayerView exact revision on arXiv

Source-linked AI summary

A Wide Evaluation of ChatGPT on Affective Computing Tasks

Mostafa M. Amin, Rui Mao, Erik Cambria, Björn W. Schuller

arXiv:2308.13911v1cs.AIcs.CL

TL;DR

Evidence on ChatGPT's effectiveness across affective computing tasks remains limited, and earlier assessments covered a narrow task subset while rarely evaluating regression. This paper evaluates GPT-3.5 and GPT-4 across 13 problems, including regression tasks recast as pairwise ranking, and finds task-dependent performance: strong results in sentiment-related problems but difficulty with implicit signals.

  • Problem

    Earlier ChatGPT evaluations covered a limited subset of affective computing, and regression tasks were scarcely evaluated amid differing prompting strategies and difficulty eliciting objective scores.

    Source: §2 · p. 2

  • Method

    The paper evaluates GPT-3.5 and GPT-4 across 13 affective computing problems and models regression evaluation as pairwise ranking classification.

    Source: §1 · pp. 1–2, §3.3 · p. 3

  • Results

    RoBERTa leads overall, followed by GPT-4, while E2E and GPT-3.5 have no clear overall winner; ChatGPT performs strongly on sentiment-related tasks but poorly on implicit-signal tasks.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The reported strengths and weaknesses vary by task, with implicit or latent social-signal problems remaining particularly difficult for the ChatGPT models.

    Source: §5 · p. 6, §4 · p. 6

    ChatGPT responses are not fully reliable for properly formatting complex predictions, with reliability reported for straightforward predictions requiring a short answer.

    Source: §4 · p. 6

Abstract

from arXiv · show

With the rise of foundation models, a new artificial intelligence paradigm has emerged, by simply using general purpose foundation models with prompting to solve problems instead of training a separate machine learning model for each problem. Such models have been shown to have emergent properties of solving problems that they were not initially trained on. The studies for the effectiveness of such models are still quite limited. In this work, we widely study the capabilities of the ChatGPT models, namely GPT-4 and GPT-3.5, on 13 affective computing problems, namely aspect extraction, aspect polarity classification, opinion extraction, sentiment analysis, sentiment intensity ranking, emotions intensity ranking, suicide tendency detection, toxicity detection, well-being assessment, engagement measurement, personality assessment, sarcasm detection, and subjectivity detection. We introduce a framework to evaluate the ChatGPT models on regression-based problems, such as intensity ranking problems, by modelling them as pairwise ranking classification. We compare ChatGPT against more traditional NLP methods, such as end-to-end recurrent neural networks and transformers. The results demonstrate the emergent abilities of the ChatGPT models on a wide range of affective computing problems, where GPT-3.5 and especially GPT-4 have shown strong performance on many problems, particularly the ones related to sentiment, emotions, or toxicity. The ChatGPT models fell short for problems with implicit signals, such as engagement measurement and subjectivity detection.

1 INTRODUCTION

This work broadens ChatGPT evaluation in affective computing, testing 13 problems and introducing a pairwise-ranking approach for regression tasks. ChatGPT performs strongly on sentiment-related tasks but struggles with implicit signals.

1 INTRODUCTION

  • The foundation-model paradigm uses a general model prompted for multiple problems rather than training a specialized model for each one.The passage describes GPT-3.5 and GPT-4 as ChatGPT foundation models and notes that their capabilities are being studied across domains.Source: §1 · p. 1
  • The earlier evaluation [10] was manual, used a small evaluation set, had less-developed prompting, and omitted regression-based tasks.It covered suicide tendency detection, Big Five personality assessment, and sentiment analysis; its results were comparable to classical methods but worse than fine-tuned RoBERTa.Source: §1 · p. 1
  • The evaluation covers 34 setups across 13 affective computing problems, from aspect extraction and sentiment to toxicity, personality, and subjectivity detection.The authors describe the selected tasks as prominent in affective computing and collectively spanning varied dimensions of affective language understanding.Source: §1 · p. 1
  • ChatGPT excels on sentiment-related tasks, outperforming supervised baselines in opinion extraction, emotions, and sentiment intensity ranking, but struggles with implicit-signal tasks.The reported implicit-signal difficulties include engagement measurement, personality assessment, sarcasm detection, and subjectivity detection.Source: §1 · pp. 1–2
  • The paper contributes a wide evaluation of GPT-3.5 and GPT-4, a prompting framework, and a framework that recasts regression as pairwise ranking classification.Source: §1 · p. 2

2 RELATED WORK

Earlier ChatGPT assessments covered a confined subset of affective computing tasks, while prompting strategies and regression-task evaluations remained limited. The authors motivate broader task coverage and more unified evaluation.

2 RELATED WORK

  • Related work assesses foundation models in affective computing and NLP, including ChatGPT studies of sentiment, aspect extraction, personality, and emotion recognition.Other cited work explores a broader range of NLP tasks and ChatGPT fusion with traditional NLP methods.Source: §2 · p. 2
  • Prior ChatGPT assessments concentrated on sentiment analysis and emotion detection, motivating broader evaluation across affective computing tasks and application contexts.The passage also frames wider coverage as a way to examine possible performance biases.Source: §2 · p. 2
  • Different prompting strategies have produced discrepant findings, while regression tasks have been scarcely evaluated because eliciting objective scores across varied scales is difficult.The passage motivates a unified prompting framework and an effective regression-evaluation methodology.Source: §2 · p. 2

3.1 Utilising ChatGPT

The study queried GPT-3.5 and GPT-4 through the OpenAI API using task-specific system prompts and example text as the user message. It excluded examples that could not be parsed simply.

3.1 Utilising ChatGPT

  • The study queried ChatGPT through the OpenAI API using gpt-3.5-turbo-0301 for GPT-3.5 and gpt-4-0314 for GPT-4.Source: §3.1 · p. 2
  • For each problem and target-label combination, the system prompt specified the task, and each example's input text was sent in a user message.The assistant response served as the example's answer.Source: §3.1 · p. 2
  • Examples without simple parsing criteria were excluded; the authors report that more specific system prompts made most examples easy to parse.The passage contrasts this prompt specificity with the previous work.Source: §3.1 · p. 2

3.2 Prompting

The prompts assign ChatGPT an expert role, specify the prediction task, and constrain the response format. Word-level prompts request aspect or opinion spans, while sentence-level prompts cover classification and pairwise comparisons.

3.2 Prompting

  • The general prompt pattern assigns an expert role, defines the prediction task and labels, and specifies allowed and disallowed answer formats without reasoning explanations.Source: §3.2 · p. 2

3.2.1 Word-level labels prompts

  • The aspect prompt asks for aspect words and one of four sentiment targets, omitting background targets or returning BACKGROUND when none apply.The target labels are positive, negative, neutral, and conflict.Source: §3.2.1 · p. 2
  • The opinion-extraction prompt requests explicit sentiment-bearing words or expressions linked to aspects, preserving their wording and excluding implied sentiment.It requests BACKGROUND when no sentiment-bearing words are present.Source: §3.2.1 · pp. 2–3

3.2.2 Sentence-level labels prompts

  • Sentence-level prompts ask for positive-or-negative sentiment classification and pairwise choices of stronger positive sentiment or emotion intensity.The sentiment prompt disallows a neutral answer; intensity prompts compare texts A and B.Source: §3.2.2 · p. 3
  • The shared answer-format instruction restricts responses to one of two specified labels and prohibits explanations or hedging phrases.Source: §3.2.2 · p. 3
  • Other sentence-level prompts classify suicide tendency, six toxicity traits, well-being-related stress, or whether stress is discussed or expressed.For toxicity, each prompt targets one trait at a time; stress may be implicit or explicit.Source: §3.2.2 · p. 3
  • The engagement prompt compares tweets by estimated virality, while the personality prompt compares which text expresses greater intensity of a Big Five trait.Source: §3.2.2 · p. 3
  • The sarcasm prompt asks whether a text is sarcastic, and the subjectivity prompt distinguishes opinionated or biased text from objective or factual description.The subjectivity prompt concerns how a text is described, not what it describes.Source: §3.2.2 · p. 3

3.3 Pairwise comparison to solve regression tasks

The framework turns regression evaluation into pairwise ranking classification, comparing sampled pairs of examples rather than asking ChatGPT for objective scores. It also describes a prediction procedure, though the paper does not use it.

3.3 Pairwise comparison to solve regression tasks

  • Regression evaluation becomes binary classification: for each sampled pair, determine whether y_a > y_b.This avoids requesting objective scores from ChatGPT when dataset scales and annotation criteria vary.Source: §3.3 · p. 3
  • The method samples M = 4N pairs using a small-world graph to keep the comparisons relatively few yet representative.The graph is described as densely connected without an excessive number of connections.Source: §3.3 · p. 3
  • A successive-halving procedure could predict scores by comparing against median items until the remaining range is narrow, but it is not used here.The proposed procedure is similar to binary search.Source: §3.3 · p. 3

3.4 Compared Model

The comparison model is an RNN framework built around a bidirectional LSTM and fully connected layers. Its output activation, optimizer, loss, class weighting, and hyperparameter tuning vary by task or setup.

3.4 Compared Model

  • The RNN baseline uses a bidirectional LSTM followed by N fully connected layers and a final prediction layer.Each LSTM direction has L units, and the fully connected layers have U units with ReLU activations.Source: §3.4 · p. 3
  • The output activation is sigmoid for most setups, with softmax for aspect-based tasks, tanh for sentiment ranking, and ReLU for engagement measurement.Source: §3.4 · p. 3
  • Training uses Adam, crossentropy for classification, and MAE for regression; toxicity detection and aspect extraction apply class weighting for imbalance.SMAC tunes the weighting and model hyperparameters; each problem uses 25 combinations and training runs use early stopping.Source: §3.4 · pp. 3–4

3.5 Text Features

The study trains RNNs with two input representations: raw word IDs or features extracted from RoBERTa. The word-ID setup learns embeddings jointly with the model, while RoBERTa supplies subword-level vectors.

3.5 Text Features

  • The end-to-end (E2E) representation uses raw text as word IDs, limited to the 2,000 most common training-set words, with jointly trained 128-dimensional embeddings.Source: §3.5 · p. 4
  • The alternative represents each input using RoBERTa-base features, with a 768-dimensional vector for each subword.The extracted features form a sequence corresponding to the sentence’s subwords.Source: §3.5 · p. 4
  • For word-based aspect labels, training assigns each subword its word’s label, while evaluation uses the first subword’s prediction for that word.Source: §3.5 · p. 4

3.6 Datasets

The evaluation uses 13 datasets for 13 affective-computing tasks, with original splits when available and downsampling of test sets when needed. The datasets span reviews, social media, Reddit, and news headlines.

3.6 Datasets

  • The aspect-based tasks use SemEval 2014 and 2015 laptop and restaurant reviews, while sentiment analysis uses Twitter140 with negative or positive labels.For Twitter140, the small provided test set leads the authors to split the large training set into train, development, and test sets.Source: §3.6 · p. 4
  • Sentiment-intensity ranking uses SemEval-2017 microblog data with scores in [-1, 1], while emotion-intensity ranking uses EmoInt with four emotions and scores in [0, 1].Source: §3.6 · p. 4
  • The dataset-size table covers the selected data after downsampling and possible splitting, and defines ABSA as aspect extraction, opinion extraction, and aspect polarity classification.Source: Table 1 · p. 4
  • Suicide-tendency data come from Reddit depression and teenagers subreddits; toxicity data are human-annotated Wikipedia comments, with the negative class downsampled.The toxicity labels cover toxicity, severe toxicity, obscenity, threats, insults, and identity-based hate.Source: §3.6 · p. 4
  • Well-being datasets use Reddit and Twitter, engagement uses TEDTalks-related tweets, and personality uses transcripts from annotated videos.The engagement target is log10(number of retweets + 1); personality labels follow the Big Five model.Source: §3.6 · p. 4
  • Sarcasm data use TheOnion and HuffPost headlines, while subjectivity data use Rotten Tomatoes reviews labeled subjective or objective.Source: §3.6 · pp. 4–5

4 RESULTS

Across the evaluated problems, the RoBERTa-feature RNN performs best in most cases, while GPT-4 is often competitive and leads on several sentiment- and emotion-related tasks. ChatGPT performs well on some problems involving extremely negative emotions but struggles with implicit social signals, and complex predictions can still create parsing problems.

4 RESULTS

  • The RoBERTa-feature RNN performs best on the majority of problems for both metrics, while GPT-4 ranks second overall.GPT-3.5 is often significantly behind the RNN, though GPT-3.5 and the end-to-end baseline have no clear overall winner across all problems.Source: §4 · p. 5
  • RoBERTa leads aspect extraction, whereas GPT-4 leads opinion extraction, especially on UAR.GPT-4’s extra surrounding words on aspect expressions may contribute to its lower aspect-extraction performance.Source: §4 · p. 5
  • In sentiment-related tasks, RoBERTa and GPT-4 outperform GPT-3.5 and the end-to-end baseline; GPT-4 leads emotions intensity ranking.GPT-3.5 and the end-to-end baseline have relatively close performance on sentiment problems.Source: §4 · p. 5
  • On suicide tendency, well-being, and toxicity, ChatGPT performs strongly in many cases, significantly outperforming the end-to-end baseline.The authors suggest safety-focused training may contribute to its toxicity performance.Source: §4 · pp. 5–6
  • For well-being assessment, GPT-3.5 and GPT-4 lead on long Reddit bodies, while RoBERTa leads by a wide margin on shorter texts.ChatGPT also leads on toxicity detection but is less competent than RoBERTa at identifying specific toxicity reasons.Source: §4 · p. 6
  • On engagement, personality, sarcasm, and subjectivity, GPT-3.5 is significantly worse than RoBERTa in all cases and usually worse than the end-to-end baseline.GPT-4 improves only slightly, surpassing the end-to-end baseline on sarcasm and subjectivity; engagement results are close to random, and alternative formulations did not improve them.Source: §4 · p. 6
  • Personality assessment results preserve a consistent relative ordering across five labels under pairwise ranking, supporting the regression-evaluation technique; the study reports accuracy, UAR, and permutation significance tests.Time-series labels use micro-averaging.Source: §4 · pp. 5–6
  • Parsing problems are mostly resolved, but compound aspect predictions remain unreliable, and formatting is dependable mainly for straightforward, short answers.For aspect tasks, responses may improvise formats or modify input words.Source: §4 · p. 6

5 CONCLUSION

The paper evaluates GPT-3.5 and GPT-4 on 13 affective computing problems and compares them with an end-to-end RNN and an RNN using RoBERTa features. RoBERTa-feature RNNs lead overall, while ChatGPT performs strongly on many sentiment and negative-emotion tasks but falls short on implicit signals.

5 CONCLUSION

  • Across the 13 problems, the RoBERTa-feature RNN leads overall, followed by GPT-4; GPT-4 is mostly better than GPT-3.5, except on aspect extraction.GPT-4’s advantage over GPT-3.5 is statistically significant in most cases.Source: §5 · p. 6
  • ChatGPT performs strongly on sentiment-related problems and especially on well-being assessment over long texts and toxicity detection, but falls short on tasks with implicit signals.GPT-3.5 is significantly worse than the end-to-end RNN on most such tasks; GPT-4 is at best slightly better than it on two.Source: §5 · p. 6
  • The authors identify reinforced prompt design and synergistic combinations of the compared approaches as possible future efforts.Source: §5 · p. 6
Loading 2308.13911v1…