Source-linked AI summary
AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen
TL;DR
Manual NLP annotation is expensive and time-consuming, motivating whether GPT-3.5 can serve as a crowdsourced annotator. AnnoLLM uses explain-then-annotate prompts and also constructs a conversation-based retrieval dataset; it matches or surpasses crowdsourced annotators across three tasks.
Problem
Manual annotation is labor-intensive and time-consuming, while the paper asks whether GPT-3.5 can replace crowdsourced annotators.
Method
AnnoLLM generates explanations for labeled examples, incorporates them into few-shot chain-of-thought prompts, and uses the prompts to annotate unlabeled data.
Results
AnnoLLM surpasses or matches crowdsourced annotators across QK, BoolQ, and WiC, while its QK result surpasses crowdsourced annotators.
Takeaways & Limitations
AnnoLLM demonstrates the feasibility of using LLMs to substitute crowdsourced annotators and supports construction of a conversation-based information retrieval dataset.
Abstract
from arXiv · showhide
Many natural language processing (NLP) tasks rely on labeled data to train machine learning models with high performance. However, data annotation is time-consuming and expensive, especially when the task involves a large amount of data or requires specialized domains. Recently, GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. In this paper, we first claim that large language models (LLMs), such as GPT-3.5, can serve as an excellent crowdsourced annotator when provided with sufficient guidance and demonstrated examples. Accordingly, we propose AnnoLLM, an annotation system powered by LLMs, which adopts a two-step approach, explain-then-annotate. Concretely, we first prompt LLMs to provide explanations for why the specific ground truth answer/label was assigned for a given example. Then, we construct the few-shot chain-of-thought prompt with the self-generated explanation and employ it to annotate the unlabeled data with LLMs. Our experiment results on three tasks, including user input and keyword relevance assessment, BoolQ, and WiC, demonstrate that AnnoLLM surpasses or performs on par with crowdsourced annotators. Furthermore, we build the first conversation-based information retrieval dataset employing AnnoLLM. This dataset is designed to facilitate the development of retrieval models capable of retrieving pertinent documents for conversational text. Human evaluation has validated the dataset's high quality.
1 Introduction
AnnoLLM addresses the cost and labor of manual NLP annotation by using GPT-3.5 with task guidance, category definitions, examples, and generated explanations. Its explain-then-annotate design is reported to match or surpass crowdsourced annotators across three datasets.
- Manual labeling is labor-intensive and time-consuming, limiting the availability of large datasets for NLP tasks.
- The paper asks whether GPT-3.5 can replace crowdsourced annotators, given its zero-shot and few-shot capabilities.
- AnnoLLM provides annotation guidance and generates explanations for demonstrated examples before annotating unlabeled data.
- The system is presented as an LLM-powered annotation system based on explain-then-annotate.
- Across three datasets, AnnoLLM either surpasses or matches crowdsourced annotators.
2 Approach
The approach adapts human annotation guidance for GPT-3.5, then uses generated explanations to form few-shot chain-of-thought prompts. AnnoLLM also constructs a conversation-based information retrieval dataset by generating, expanding, and filtering conversational examples.
- 2 Approach: Human-style annotation guidance consists of task descriptions, category definitions, and demonstrated examples.
- 2 Approach: GPT-3.5 receives task descriptions and category definitions for zero-shot annotation, and demonstrated examples for few-shot annotation.
- 2 Approach: AnnoLLM asks GPT-3.5 to explain why each demonstrated example has its ground-truth label, then incorporates those explanations into few-shot CoT prompts.
- 2.3 GPT-3.5 as a Few-shot Data Creator: Conversation-based retrieval datasets are needed because traditional retrieval models perform poorly on conversational retrieval tasks.
- 2.3 GPT-3.5 as a Few-shot Data Creator: The dataset-generation process expands source paragraphs, generates conversations from the expanded text and query, and filters pairs for relevance.
3 Experiment on Data Annotation
AnnoLLM is evaluated on QK, WiC, and BoolQ using accuracy, with comparisons against prompting baselines, crowdsourced annotators, and supervised models. Results show strong performance overall, while ablations and prompt analyses examine explanation quality, label guidance, and template stability.
- Experimental Setup: AnnoLLM is evaluated on QK, BoolQ, and WiC, all binary-classification tasks measured with accuracy.QK assesses query-keyword relevance, BoolQ is yes/no question answering, and WiC disambiguates word senses in context.
- Experimental Results: On QK, AnnoLLM with 4-shot CoT outperforms zero-shot and few-shot counterparts by around 6 and 8 points, respectively, and surpasses crowdsourced annotators.The comparison uses development and test-set results reported in Table 2.
- Experimental Results: On WiC, AnnoLLM with 8-shot CoT significantly outperforms its few-shot counterpart, but remains below crowdsourced annotators.The paper attributes the remaining gap to the task’s inherent complexity, noting a similar gap for supervised models.
- Experimental Results: On BoolQ, AnnoLLM with 8-shot CoT surpasses human annotators and is comparable to supervised models, without significant improvement over few-shot prompting.Its advantage appears more clearly in prompt robustness than in the main performance comparison.
- Overall Findings: Across three tasks, AnnoLLM surpasses or matches human performance, supporting its potential to replace crowdsourced annotators.The paper also presents this as evidence that CoT can work beyond typical reasoning tasks.
- Ablation Study: Removing ground-truth labels when generating explanations decreases QK test performance by approximately 3 points, indicating that label guidance matters for explanation generation.The authors suggest that unguided generation may produce explanations for incorrect answers.
- More Analysis and Discussion: Few-shot CoT prompts remain more stable across BoolQ templates, whereas few-shot prompting is highly template-sensitive and can fall from around 89 to below 80 points.Across different generated explanations, CoT prompts also yield similar performance on QK, WiC, and BoolQ.
4 Experiment on Data Creation
AnnoLLM is used to create a conversation-based information retrieval dataset, whose retrieval and human-evaluation results assess its usefulness and quality.
- Dataset and evaluation: 71,557 training and 3,000 test instances comprise the ConIR dataset built from MS-MARCO passage ranking data.The dataset construction uses ChatGPT to enrich paragraphs and generate and filter irrelevant conversations; retrieval is evaluated with MRR@10 and Recall@k.
- Retrieval results: DPR and PROD trained on MS-MARCO perform poorly on ConIR in zero-shot evaluation, showing that traditional-dataset training does not transfer directly.The result motivates evaluating models trained specifically on the conversation-based dataset.
- Retrieval results: DPR fine-tuned on ConIR performs much better than its zero-shot counterpart, highlighting the necessity of the ConIR training set.
- Human evaluation: Human evaluation rates ConIR conversations for fluency, relevance, and factual consistency against paired passages.The evaluation samples 100 conversations and uses three annotators with 5-point fluency and 3-point relevance and consistency scales.
- Human evaluation: Fleiss’ kappa is 0.55, indicating moderate inter-annotator agreement for the human evaluation.
5 Related Work
The paper situates AnnoLLM among LLM-based learning and pseudo-annotation methods, proposing GPT-3.5 as a guided data annotator.
- Large-scale Language Models: GPT models support few-shot in-context learning, while GPT-3.5 models include textdavinci-003 and ChatGPT trained with instructions and human feedback.
- Large-scale Language Models: The paper proposes converting GPT-3.5 into a task-specific data annotator through detailed annotation instructions similar to those given to human annotators.
- Pseudo Annotated Data: Pseudo-annotated data is commonly used when labeled data is limited, but prior GPT-3 labels remained below manually labeled data quality.
- Pseudo Annotated Data: Related pseudo-labeling methods include back-translation and rule-based approaches applied across tasks such as machine translation and style transfer.
6 Conclusion
AnnoLLM uses explain-then-annotate prompting to generate labels and constructs a conversation-based retrieval dataset for conversational retrieval research.
- Conclusion: AnnoLLM applies a two-step explain-then-annotate process in which LLM-generated few-shot chain-of-thought prompts annotate unlabeled data.The paper reports feasibility of substituting crowdsourced annotators across three datasets.
- Conclusion: The paper introduces the ConIR dataset, created with AnnoLLM to facilitate research on conversation-based information retrieval.
A Prompts for Generating Explanations
The appendix documents explanation-generation, zero-shot, few-shot, and few-shot chain-of-thought prompts for QK, WiC, and BoolQ.
- A Prompts for Generating Explanations: Tables 8–11 illustrate LLM inputs and outputs for generating explanations on QK, WiC, and BoolQ, with Table 11 marking incorrect explanations.
- B Zero-shot Prompts: Tables 12–14 provide zero-shot prompts for QK, WiC, and BoolQ tasks.
- C Few-shot Prompts: The QK prompt defines relevance categories and demonstrates labels such as Bad and Not bad for query-keyword pairs.Examples include “sharepoint migration tool file share” as Bad and “rv sale used class c” as Not bad.
- C Few-shot Prompts: Additional QK demonstrations label related and unrelated keywords, including browser alternatives, clothing mismatches, and photo-buying queries.
- C Few-shot Prompts: Tables 15–17 contain few-shot exemplar prompts for QK, WiC, and BoolQ, including additional passage-question-answer demonstrations.
- C Few-shot Prompts: WiC prompts determine whether a target word has the same meaning across two sentences while ignoring tense and grammatical form.Examples cover words including place, hold, summer, approach, run, head, meet, and development.
- C Few-shot Prompts: The remaining WiC examples distinguish senses for head, meet, and development, while the prompt template supplies target word and sentence placeholders.
- C Few-shot Prompts: BoolQ prompts pair passages with yes/no questions and demonstrations whose answers follow evidence supporting or denying the question.Examples include Windows Movie Maker, Epsom station, and Da Vinci’s Demons.
F Prompts for Constructing the Conversation-based Information Retrieval Dataset
The dataset construction uses zero-shot prompts to enrich passages and generate conversations, followed by few-shot chain-of-thought filtering for irrelevant conversations.
- The enrichment prompt begins by calculating a salary-range difference from maximum and minimum salary figures.The example uses $45,000 and $75,000, producing a $30,000 difference.
- The enrichment and conversation-generation prompts distinguish input text from generated output using blue and red colors, respectively.
- A zero-shot prompt generates a conversation in which one participant asks questions based on a passage and ChatGPT answers according to that passage.
- A few-shot chain-of-thought prompt is used to filter out irrelevant conversations.
G Details on Human Evaluation
Human evaluation presents annotators with a multi-turn conversation paired with a passage and asks them to assess fluency, relevance, and consistency.
- Annotators read both the multi-turn conversation and its paired passage to understand their topics and significant details.
- They then evaluate the conversation's fluency, relevance, and consistency with the provided passage.
G.1 Fluency
Fluency is evaluated through a question asking how fluent the conversation is, using a five-point Likert scale with defined comprehension and grammatical-error criteria.
- Annotators first answer how fluent they think the generated conversation is.
- Fluency is scored on a 5-point Likert scale from 1 to 5.
- The scale ranges from unintelligible and wholly nonfluent conversations at 1 to conversations understood with several grammatical errors at 4.
G.2 Relevance
Relevance is evaluated by asking how relevant the conversation is to its paired passage and scoring the relationship on a three-point Likert scale.
- Graders assess relevance by considering whether the generated conversation relates to the given passage.
- Relevance is scored on a 3-point Likert scale from 1 to 3.
- Scores distinguish completely unrelated topics, few related topics, and mostly related topics.
G.3 Consistency
Consistency is assessed by comparing facts in a conversation with those in a given passage. Graders use a three-point Likert scale, with human evaluation results reported in Table 30.
- Table 30 reports the frequency distribution for each annotation score and the average annotation score.
- Graders assess how consistent a conversation is with a given passage.
- The annotation uses a 3-point Likert scale based on how many conversation facts are supported by the passage.A score of 1 means no facts appear in the passage, 2 means few are supported, and 3 means most are consistent.