Source-linked AI summary
Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPT
Mostafa M. Amin, Erik Cambria, Björn W. Schuller
TL;DR
The paper examines whether ChatGPT's potential emergence extends across NLP tasks, including affective-computing classification. It evaluates ChatGPT against specialised baselines and finds decent generalist performance, but task-specialised models remain stronger overall.
Problem
The paper examines whether ChatGPT's potential emergence extends across NLP tasks, amid questions about whether it is a general NLP solver.
Source: RELATED WORK · p. 2Method
The study evaluates ChatGPT on seven classification tasks across personality, sentiment, and suicide tendency, comparing it with specialised models including RoBERTa, Word2Vec, and BoW.
Source: Baselines · p. 5, CONCLUSION · p. 7Results
ChatGPT performs best on sentiment, poorly on personality, and averagely on suicide assessment, while remaining less capable than the best specialised model for downstream tasks.
Source: Limitations · p. 7, RESULTS · p. 7Takeaways & Limitations
ChatGPT is a generalist that can perform decently across problems without specialised training, while specialised models remain the best option for optimal task performance.
Source: RESULTS · p. 7, Limitations · p. 8Evaluation is constrained by small datasets of 497, 362, and 509 examples and ChatGPT's manual-only access, which permits approximately 25–35 requests per hour.
Source: Limitations · p. 7
Abstract
from arXiv · showhide
ChatGPT has shown the potential of emerging general artificial intelligence capabilities, as it has demonstrated competent performance across many natural language processing tasks. In this work, we evaluate the capabilities of ChatGPT to perform text classification on three affective computing problems, namely, big-five personality prediction, sentiment analysis, and suicide tendency detection. We utilise three baselines, a robust language model (RoBERTa-base), a legacy word model with pretrained embeddings (Word2Vec), and a simple bag-of-words baseline (BoW). Results show that the RoBERTa trained for a specific downstream task generally has a superior performance. On the other hand, ChatGPT provides decent results, and is relatively comparable to the Word2Vec and BoW baselines. ChatGPT further shows robustness against noisy data, where Word2Vec models achieve worse results due to noise. Results indicate that ChatGPT is a good generalist model that is capable of achieving good results across various problems without any specialised training, however, it is not as good as a specialised model for a downstream task.
RELATED WORK
Related work frames the question of whether ChatGPT can act as a general NLP solver and surveys its capabilities across several tasks, including sentiment analysis and machine translation.
RELATED WORK
- [7] examines whether ChatGPT is a general NLP solver across reasoning, summarisation, named entity recognition, and sentiment analysis.Source: RELATED WORK · p. 2
- [8] explores GPT language models, including ChatGPT, for machine translation.Source: RELATED WORK · p. 2
- [5] examines systematic errors made by ChatGPT.Source: RELATED WORK · p. 2
METHOD
The evaluation framework compares ChatGPT on classification tasks with specialised baselines, while the sentiment dataset presents a noisy-text setting and excludes neutral test examples.
METHOD
- The framework evaluates ChatGPT across classification tasks and compares it with three baselines, including specialised models trained for downstream tasks.Source: METHOD · p. 2, Sentiment Dataset · p. 3
Sentiment Dataset
- Sentiment140 contains noisy Twitter text with positive or negative labels; its 497-tweet test portion was filtered to 359 after neutral examples were removed.Source: METHOD · p. 3
METHOD
- The Suicide and Depression dataset comes from Reddit communities including SuicideWatch, depression, and teenagers.Source: Sentiment Dataset · p. 3
Suicide and Depression Dataset
The dataset passages describe publicly available resources and their construction, including Reddit-derived suicide and depression labels and the First Impressions personality data.
Suicide and Depression Dataset
- The paper uses publicly available datasets for reproducibility and presents their statistics in Table 1.Source: Datasets · p. 3
- Suicide and Depression examples longer than 512 characters were excluded, and the remaining data were divided into Train, Dev, and Test portions.Source: Datasets · p. 3
- First Impressions represents personality with the Big Five traits and uses transcriptions from 15-second videos whose labels derive from multimodal pairwise comparisons.Source: Personality Dataset · p. 3
Personality Dataset
For personality prediction, the study uses First Impressions transcriptions and manually queries ChatGPT in batches, with prompt wording and output formatting specified to obtain usable predictions.
Personality Dataset
- First Impressions personality labels are based on multimodal video comparisons, but the models use video transcriptions and treat prediction as regression with Mean Absolute Error loss.Source: ChatGPT querying mechanism · p. 3
- ChatGPT is queried manually with reformatted test examples in chunks of 25, using a new conversation for each chunk and collecting one answer per example.Source: ChatGPT querying mechanism · pp. 3–4
- The pipeline figure identifies the approaches in order: ChatGPT, RoBERTa, Word2Vec, and BoW.Source: Figure 1 · p. 4
- The prompt asks ChatGPT to guess a yes-or-no answer and specifies the response format because direct questions and loose formatting can hinder answer collection.Source: ChatGPT querying mechanism · p. 4
Question Formulation
The prompts ask ChatGPT to guess labels for supplied text across personality, sentiment, and suicide-related tasks.
Question Formulation
- For Big Five, ChatGPT is asked to guess each of the five traits as low or high, using bullet points.The example text replaces {text} in the prompt, with quotation marks retained.Source: Question Formulation · p. 4
- For sentiment analysis, ChatGPT is asked to guess whether the text is positive, neutral, or negative.Source: Question Formulation · p. 4
- The passage introduces a separate question formulation for the suicide problem but does not provide its wording.Source: Question Formulation · p. 4
Parsing Responses
ChatGPT responses require parsing because answers can use arbitrary formats even when their content is the same.
Parsing Responses
- ChatGPT responses need parsing because the same answer content can appear in arbitrary formats.Source: Parsing Responses · p. 4
- The RoBERTa baseline converts text into subword tokens and 768-dimensional pooled features before an MLP predicts the label.The model has 110 million parameters and is the smaller of two RoBERTa instances described in the passage.Source: Parsing Responses · p. 5
RoBERTa Language Model
The section describes a task-tuned RoBERTa MLP and compares it with Word2Vec and bag-of-words baselines trained on the task data.
RoBERTa Language Model
- SMAC selects the MLP specification from 100 models per task, trained for up to 300 epochs with early stopping and selected by Dev performance.The search varies hidden-layer count, first-layer width, optimizer, and learning rate; hidden layers halve in width, with a minimum of 32 neurons.Source: RoBERTa Language Model · p. 5
Baselines
- The baselines are task-trained RoBERTa, pretrained Word2Vec embeddings with a simple classifier, and linear-classifier bag-of-words.Hyperparameters are tuned on the Train portion with Dev validation using SMAC, and selected settings are listed in Table 2.Source: Baselines · p. 5
Word2Vec Word Embeddings
Word2Vec represents each text by averaged pretrained word embeddings, while the discussion specifies SVM tuning and kernel choices.
Word2Vec Word Embeddings
- Word2Vec tokenizes text, averages pretrained embeddings into a 300-dimensional feature vector, then uses an SVM for prediction.Source: Word2Vec Word Embeddings · p. 6
- The evaluation uses UAR to expose class-performance imbalance and a randomized permutation test for statistical significance.The main experimental results are presented in Table 3.Source: Word2Vec Word Embeddings · p. 6
Discussion
- The SVM tunes C over 20 log-sampled values from 10^-6 to 10^4; sentiment uses a linear kernel because its larger dataset makes RBF computationally impractical.Source: Discussion · p. 6
Bag of Words
The BoW baseline uses only in-domain data, while the supplied findings show ChatGPT’s performance varies by task. Its sentiment results contrast with its weaker personality results.
Bag of Words
- On personality assessment, ChatGPT is inferior to all three baselines on every trait, significantly worse than RoBERTa on all traits and Word2Vec on three.The UAR results point to similar relative performance, with lower values for openness and agreeableness across all baselines.Source: Discussion · p. 6
- The BoW baseline uses only in-domain training data and no other data for upstream or downstream training.It relies on neither knowledge transfer nor large-scale training.Source: Bag of Words · p. 6
- On sentiment analysis, ChatGPT is slightly better than RoBERTa and BoW and significantly better than Word2Vec.The authors suggest missing subword encodings may disadvantage Word2Vec and BoW on noisy Twitter text, while subword representations can remain meaningful despite typos.Source: Bag of Words · p. 6
- The TF-IDF baseline represents sentences with normalized word-count vectors over the 10,000 most common training-set words and trains a linear-kernel SVM using SGD.Features are scaled to [−1, 1], and the SGD learning rate is tuned using SMAC.Source: Bag of Words · p. 6
RESULTS
Across the evaluated tasks, Word2Vec and BoW are more or less on par with ChatGPT, while RoBERTa performs significantly better. The results also describe task-specific conditions and report a response-formatting issue.
RESULTS
- The suicide assessment task is described as less difficult and larger-data than personality assessment, with texts less noisy than the sentiment dataset.The passage frames suicide assessment as classifying extreme negative sentiment and notes prior work [7] found ChatGPT better at predicting negative than positive sentiment.Source: RESULTS · p. 6
- The evaluation covers personality traits, sentiment analysis, and suicide tendency assessment.The results passage identifies classification accuracy as a measure.Source: RESULTS · p. 6
- Word2Vec and BoW are more or less on par with ChatGPT, whereas RoBERTa is significantly better than all three on the evaluated tasks.The authors also report that ChatGPT generally does not differ significantly from BoW in almost all cases, while task-fine-tuned RoBERTa performs better.Source: RESULTS · p. 7
- ChatGPT returned arbitrary formatting despite explicit instructions specifying the desired response format.The authors note that response parsing was part of their experiments.Source: RESULTS · p. 7
CONCLUSION
Across seven classification tasks in three affective-computing problems, the framework evaluates ChatGPT against specialized models. The conclusion favors task-trained RoBERTa for performance, while noting that evaluation relied on small manually collected samples.
CONCLUSION
- The evaluation framework compares ChatGPT with specialized models across seven classification tasks covering personality assessment, sentiment analysis, and suicide tendency assessment.The baselines were RoBERTa, Word2Vec, and bag-of-words.Source: CONCLUSION · p. 7
Limitations
- Evaluation used only 497, 362, and 509 examples across the three tasks because ChatGPT was available for manual rather than automated large-scale testing.The authors identify the small evaluation sample as the most crucial limitation.Source: Limitations · p. 7
- RoBERTa fine-tuned for a specific task significantly outperformed ChatGPT, while ChatGPT showed no statistically significant difference from BoW in almost all cases.The authors conclude specialized models remain the best option for optimal performance.Source: Limitations · p. 8
- Future experiments will add explainability and computational-efficiency metrics and expand comparisons to more models, tasks, and datasets.Proposed extensions include prompt-based classification, neurosymbolic AI, sarcasm detection, metaphor understanding, and more complex sentiment datasets.Source: Limitations · p. 8