Source-linked AI summary
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli
TL;DR
Manual annotation is essential for building training data and evaluation standards, but existing human and crowd-worker approaches involve quality, cost, or scalability trade-offs. The paper systematically compares zero-shot ChatGPT with MTurk workers and trained annotators across text-annotation tasks and datasets, finding stronger performance for ChatGPT on most accuracy comparisons and all intercoder-agreement comparisons. It concludes that LLMs have substantial potential to improve the efficiency of text annotation.
Problem
NLP research needs high-quality labeled data for classifier training and evaluating unsupervised models, while manual annotation can be costly and crowd-worker quality may be insufficient for complex tasks.
Method
The paper compares zero-shot ChatGPT, MTurk workers, and trained annotators on the same annotation tasks across four datasets using accuracy and intercoder agreement.
Results
ChatGPT outperforms MTurk on accuracy for most tasks and exceeds both MTurk and trained annotators in intercoder agreement across all tasks.
Takeaways & Limitations
The findings demonstrate the potential of LLMs to transform text-annotation procedures and increase annotation efficiency.
Takeaways & Limitations
Some annotation tasks were not applicable to every dataset, and human annotation attempts for several news-article tasks were unsuccessful.
Abstract
from arXiv · showhide
Many NLP applications require manual data annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd-workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT's intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk. These results show the potential of large language models to drastically increase the efficiency of text classification.
1 Introduction
The paper evaluates zero-shot ChatGPT as an alternative to human text annotation, comparing its accuracy and consistency with MTurk workers and trained annotators across varied tasks and datasets. It reports stronger performance for ChatGPT on most accuracy comparisons and all intercoder-agreement comparisons.
- Motivation: High-quality labeled data are needed to train classifiers and evaluate unsupervised models, but producing them manually can require substantial annotation effort.Researchers commonly annotate texts for relevance, topics, sentiment, stance, or conceptual categories.
- Motivation: Trained annotators tend to provide higher-quality data at significant cost, whereas crowd-workers are cheaper and more flexible but may provide insufficient quality for complex tasks.The paper also notes concerns about declining MTurk data quality and the reduced practicality of alternative platforms.
- Contribution: Zero-shot ChatGPT classifications outperform MTurk annotations at a fraction of the cost, motivating a systematic evaluation of LLM-based annotation.The study presents itself as, to the authors’ knowledge, the first systematic evaluation of this kind.
- Contribution: Across four datasets, ChatGPT’s accuracy exceeds MTurk’s for most tasks, while intercoder agreement exceeds both MTurk and trained annotators in all tasks.Accuracy is agreement with trained annotators; the study compares both metrics across the datasets.
- Results: Intercoder agreement averaged 56% for MTurk, 79% for trained annotators, 91% for ChatGPT at temperature = 1, and 97% for ChatGPT at temperature = 0.2.The results suggest that lower temperature may increase consistency without decreasing accuracy.
3 Discussion
The discussion frames ChatGPT as a potentially transformative tool for research annotation, based on comparisons across multiple datasets, tasks, annotator groups, and model settings. It highlights practical directions for extending this evaluation and integrating LLMs with human labeling.
- Discussion: The paper identifies multilingual performance, few-shot learning, semi-automated labeling, chain-of-thought prompting, and comparisons across LLMs as promising directions.These directions extend evaluation beyond the current annotation setup.
- Discussion: The study uses four datasets comprising tweets and news articles collected across different samples and time periods.The datasets include 2,382 content-moderation tweets, 1,856 US Congress tweets, 1,606 news articles, and 500 tweets from January 2023.
- Discussion: The annotation tasks cover relevance, six-class topic detection, stance toward repealing Section 230, and general problem-versus-solution frame detection.These tasks operationalize several common text-classification problems in the study’s domains.
- Discussion: Trained annotators, MTurk workers, and ChatGPT performed the same tasks using shared instructions, enabling direct comparison across annotation approaches.Crowd-workers were subject to qualification, approval-rate, location, and workload restrictions.
- Discussion: ChatGPT annotations used the gpt-3.5-turbo API, separate tweet-by-tweet prompts, and temperature settings of 1 and 0.2 to examine output determinism.Two responses were collected at each temperature to calculate intercoder agreement, with new chat sessions for every tweet.
- Discussion: Accuracy was measured against agreed trained-annotator labels, while intercoder agreement measured the percentage of instances receiving the same label from two annotators.These metrics capture agreement with the benchmark and consistency within each annotator group, respectively.
S1 Annotation Codebooks
The codebooks establish the study’s annotation context and note that task coverage differs across datasets. Content moderation concerns screening user-posted content against platform rules, with difficult boundary cases involving politics and vaccines.
- Dataset coverage: Annotation tasks were not conducted for every dataset, because some tasks were unsuitable for particular sources.Stance, problem/solution, and topic modeling were unsuitable for US Congress tweets, while human annotation for stance, topic, and policy frames was unsuccessful for news articles.
- Content moderation: Content moderation screens and monitors user-posted content to decide whether it complies with platform rules.Content may be published, removed, flagged, labeled, or disputed based on whether it is illegal, hateful, inappropriate, or otherwise noncompliant.
- Content moderation: Moderation decisions become contested when content concerns issues such as vaccine safety or politics.Disagreement can involve what to block, how to moderate it, how effective moderation is, and whether governments, platforms, human moderators, or artificial intelligence should act.
S1.2 Background on political tweets (to be used for tweets by the US Congress members)
For the political-content task, tweets are classified according to whether they concern politics or government policy. The codebook directs annotators to use textual and contextual indicators of political messaging.
- Political content: “Political content” covers tweets about politics or government policies at local, national, or international levels.The category includes discussion of political figures, events, issues, political language, and hashtags.
- Coding guidance: Annotators should consider political keywords, figures, events, linked sources, and the tweet’s overall tone or sentiment.These factors help determine whether a tweet conveys a political message or viewpoint.
S1.3 Task 1: Relevance (Content Moderation)
The content-moderation relevance task distinguishes tweets that discuss moderation from tweets that are merely examples of moderated content. Annotators assign binary relevant or irrelevant labels using the codebook definition.
- Procedure: Annotators first read each tweet carefully and then assign either relevant (1) or irrelevant (0).
- Relevant: Tweets are coded relevant when they directly discuss content-moderation rules, practices, government regulation, or flagging.The task includes moderation by social-media platforms and government regulation of online content moderation.
- Irrelevant: Tweets are coded irrelevant when they do not discuss content moderation or are themselves examples of moderated content.Examples include tweets labeled disputed, claims that something is false, and sensitive content subject to moderation.
S1.4 Task 2: Relevance (Political Content)
The political-content relevance task uses a binary classification: tweets containing political content are relevant, while tweets without it are irrelevant.
- Procedure: Annotators classify each tweet as either relevant (1) or irrelevant (0).
- Coding rule: Tweets are relevant when they include political content and irrelevant when they do not.Political content is determined according to the codebook’s definition.
S1.5 Task 3: Problem/Solution Frames
This task classifies tweets according to whether they frame content moderation as a problem, a solution, or neither.
- Tweets are classified as describing content moderation as a problem, solution, or neutral.The neutral category covers tweets that report moderation activity without linking it to advantages or disadvantages.
- Problem frames emphasize negative effects such as restrictions on free speech or bias in moderation decisions.
- Solution frames emphasize positive effects such as protecting users from harmful content.Examples include hate speech, misinformation, illegal adult content, and spam.
S1.6 Task 4: Policy Frames (Content Moderation)
This task assigns content-moderation tweets to policy topics, using defined categories covering economic, moral, legal, security, health, social, and political dimensions.
- Tweets are classified into one of the defined policy topics after careful reading.
- Economy covers the costs, benefits, and monetary or financial implications of the issue.Capacity and resources covers the availability or lack of physical, human, geographical, spatial, and financial resources for policy implementation.
- Morality covers perspectives or actions compelled by doctrine, duty, honor, righteousness, ethics, or social responsibility.
- Fairness and equality covers how laws, punishment, rewards, and resources are distributed, including competing group or individual interests.
- Legal and security categories cover constitutional authority, policy evaluation, law enforcement and crime, and threats or protection of people and groups.The remaining categories include health and safety, quality of life, and cultural identity.
S1.7 Task 5: Policy Frames (Political Content)
This task classifies political-content tweets into defined policy-frame topics spanning economic, moral, legal, security, social, public-opinion, political, regulatory, and other considerations.
- Tweets are classified into one of the defined topics after careful reading.
- Economy covers monetary implications, while capacity and resources covers resources and system capacity for implementing policy goals.
- Morality concerns ethics or social responsibility, while fairness and equality concerns distribution and competing rights or interests.
- Legal and security frames cover constitutional authority, policy effectiveness, law enforcement and crime, and protection from threats.
- Additional topics cover health and safety, quality of life, cultural identity, public opinion, political considerations, external regulation and reputation, and other issues.
S1.8 Task 6: Stance Detection
This task identifies whether tweets express a positive, negative, or neutral stance toward Section 230, a law concerning online-platform liability for user content.
- Section 230 protects websites and online platforms from legal responsibility for content posted by their users.Platforms may still moderate content and remove material violating their own policies.
- Tweets are classified as having a positive, negative, or neutral stance toward Section 230.
S1.9 Task 7: Topic Detection
This task classifies tweets about content moderation by topic, using six predefined categories and an OTHER fallback. Annotators are instructed to read each tweet carefully before assigning its topic.
- SECTION 230 covers the U.S. law protecting online platforms from legal responsibility for user-posted content.
- TRUMP BAN covers platform decisions to suspend Donald Trump’s account.
- TWITTER SUPPORT covers requests directed to Twitter’s support account or help center, while PLATFORM POLICIES covers platform rules and practices.
- COMPLAINTS covers objections or suggestions concerning deplatforming, moderation, or account suspensions, while unrelated texts belong to OTHER.
- Annotators should carefully read each tweet and pay close attention to details before classifying it.
- Annotators classify each tweet according to its topic, defined by the text’s function, author’s purpose, and form.
- The categories include SECTION 230, TRUMP BAN, TWITTER SUPPORT, PLATFORM POLICIES, COMPLAINTS, and OTHER.