Source-linked AI summary
Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks
Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, Gareth Tyson
TL;DR
The paper asks whether ChatGPT can reproduce human-generated labels for social computing research, a question motivated by the cost and centrality of human annotation. It relabels five datasets and finds moderate overall agreement with human labels, alongside substantial variation across tasks and labels. The authors therefore identify potential for annotation while emphasizing important limitations and avenues for refinement.
Problem
The paper examines whether ChatGPT can reproduce human-generated annotations in social computing tasks, where human annotation remains central to producing ground-truth data.
Method
The study uses ChatGPT to relabel five datasets covering stance detection, sentiment analysis, hate speech, and bot detection, using prompts that specify candidate labels and request explanations.
Results
ChatGPT achieves an average accuracy of 0.609 across the five annotation tasks, but performance varies substantially across labels.
Takeaways & Limitations
The findings indicate that ChatGPT has potential to generate annotations similar to human labels and provide a basis for further research on ChatGPT-assisted annotation.
Takeaways & Limitations
The study uses only a single prompt for annotation, while performance may also be weaker on newer domains that emerged after ChatGPT’s initial training.
Abstract
from arXiv · showhide
The release of ChatGPT has uncovered a range of possibilities whereby large language models (LLMs) can substitute human intelligence. In this paper, we seek to understand whether ChatGPT has the potential to reproduce human-generated label annotations in social computing tasks. Such an achievement could significantly reduce the cost and complexity of social computing research. As such, we use ChatGPT to relabel five seminal datasets covering stance detection (2x), sentiment analysis, hate speech, and bot detection. Our results highlight that ChatGPT does have the potential to handle these data annotation tasks, although a number of challenges remain. ChatGPT obtains an average accuracy 0.609. Performance is highest for the sentiment analysis dataset, with ChatGPT correctly annotating 64.9% of tweets. Yet, we show that performance varies substantially across individual labels. We believe this work can open up new lines of analysis and act as a basis for future research into the exploitation of ChatGPT for human annotation tasks.
1 INTRODUCTION
The paper examines whether ChatGPT can reproduce human-generated annotations across five social computing datasets. It finds potential for annotation, but performance differs substantially across labels and tasks.
- Human annotation remains central to social computing because it produces ground-truth data for training models on tasks such as stance, hate speech, sentiment, and bot detection.
- The study evaluates whether ChatGPT can reproduce human-generated labels across five text-based social computing annotation tasks.
- 64.9% of tweets were correctly annotated on the sentiment dataset, the strongest reported task performance.
- 57.1% of tweets were correctly annotated on the hate speech task, the weakest reported task performance.
- Performance varies substantially across labels, including precision of 0.353 for anti-Asian hate speech versus 0.791 for counterspeech.
- The authors frame the work as a basis for future research using ChatGPT in human annotation tasks.
2 RELATED WORK
Related work has applied ChatGPT to multiple NLP annotation tasks, but most studies focus on one task. This paper instead compares performance across several annotation settings while highlighting unresolved variation and leakage concerns.
- Prior studies investigate ChatGPT for annotating misinformation, hate speech, stance, sentiment, genre, topic, and fake news.
- Earlier stance-detection work reports that ChatGPT can outperform most state-of-the-art models in zero-shot settings.
- Reported performance can vary across ChatGPT model versions, potentially because past prompts may enter training data for later generations.
- The paper also identifies a tendency to overestimate neutral stances in broader stance annotation contexts.
- Most existing literature focuses on a single annotation task, whereas this study performs a comparative analysis across different tasks.
3 METHODOLOGY
The study compares ChatGPT annotations with human labels across five English Twitter datasets spanning stance, hate speech, sentiment, and bot detection. It uses task-specific prompts containing topics, candidate labels, tweet text, and requested explanations, then extracts labels from responses.
- Dataset selection: The comparative study uses five datasets covering stance detection, hate speech detection, sentiment detection, bot detection, and Russo-Ukrainian stance detection.
- Dataset selection: The datasets are English-language and human-annotated so ChatGPT labels can be compared with existing human annotations.
- Datasets: The COVID-19-Stance dataset contains manually crowdsourced labels of “In-favor,” “Against,” and “Neither” across four topics.
- Datasets: The COVID-HATE dataset labels tweets as “Hate,” “Counterspeech,” or “Neutral” using pretrained human annotators and agreement validation.
- Datasets: The sentiment dataset uses positive, neutral, and negative labels, while ChatGPT receives no topic because the human annotators were not given specific topics.
- Datasets: The Russo-Ukrainian dataset tests annotation in a recent domain that arose after ChatGPT’s launch.
- ChatGPT annotation: ChatGPT is configured with gpt-3.5-turbo and prompted using a generalized classification template adapted to each dataset’s annotation strategy.
4 RESULTS AND ANALYSIS
ChatGPT annotations were compared with human gold-standard labels across five social-computing tasks. Overall performance showed potential for automated annotation, but accuracy remained limited and varied substantially across labels and tasks.
- 97.5% of 35,618 tweets received extractable ChatGPT labels, while 2.5% were judged unclear or insufficiently informative.
- 0.609 average accuracy was achieved across the five annotation tasks, while every dataset had a weighted F1-score below 0.65.The evaluation treated original human annotations as the gold standard and used weighted F1-score for performance assessment.
- ChatGPT’s performance varied across distinct labels in every annotation task, with especially large differences in Russo-Ukrainian Sentiment and Bot Detection.
- Task Analysis and Implications: Sentiment Analysis ranked first with a weighted F1-score of 0.646 and 64.9% correctly annotated tweets, although 36% of negative and 44% of positive tweets were labeled Neutral.
- Task Analysis and Implications: Stance Detection ranked second with a weighted F1-score of 0.604 and 61.2% correctly annotated tweets, while 35% of non-stance tweets were labeled In-favor and 22% Against.
- Task Analysis and Implications: Hate Speech ranked third with a weighted F1-score of 0.581; Hate had recall 0.969 but precision 0.353, and 51% of Neutral tweets were labeled Hate.
- Task Analysis and Implications: Bot Detection ranked fourth with a weighted F1-score of 0.572, including F1-scores of 0.748 for Human and 0.364 for Bot; 77% of bot tweets were mislabeled Human.
- Task Analysis and Implications: Russo-Ukrainian Sentiment ranked fifth with a weighted F1-score of 0.551, including F1-scores of 0.733 for Pro-Russia and 0.429 for Pro-Ukraine.The authors report that performance may be weaker for newer domains and that contextual change can affect stance annotation.
5 CONCLUSION AND DISCUSSION
The study evaluates ChatGPT as an annotator across five datasets and finds task- and label-dependent performance, alongside limitations in prompting and contextual inference. These findings support further research on automated and human-in-the-loop annotation.
- The study uses a standard prompt design and identifies prompt specialization, contextual data, staged queries, and human-in-loop approaches as future improvements.The authors report that only a single prompt was used and propose task-specific prompts and secondary data such as context.
- The authors conclude that ChatGPT has potential for data annotation while also highlighting limitations that require further investigation.They frame the work as a basis for future research into automated text annotation.
- ChatGPT’s performance varies across annotation tasks and labels, including substantial differences between COVID stance labels.F1-score = 0.689 for Wearing a Face Mask versus F1-score = 0.465 for Keeping Schools Closed.
- ChatGPT appears stronger when annotation relies on knowledge about events than when tasks depend on less objective information such as bot-related features.The authors suggest that limited incorporation of social-network information and profile metadata may help explain weaker bot-detection performance.
B EXAMPLES OF TWEET, PROMPT AND CHATGPT’S RESPONSE.
The appendix presents examples related to prompts and ChatGPT responses for classification annotation tasks. The supplied passages identify the examples table but do not describe its individual entries.
- Table 6 contains examples of tweets, prompts, and ChatGPT responses for the five annotation tasks.