Source-linked AI summary
CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy F. Chen, Zhengyuan Liu, Diyi Yang
TL;DR
LLMs can annotate text at lower cost, but their performance varies and prior work has given limited attention to allocating work collaboratively with humans. CoAnnotating uses uncertainty to guide instance-level work allocation and Pareto analysis to compare quality and cost. The reported results show more efficient and accurate allocation than random allocation, alongside tradeoffs that vary with task difficulty.
Problem
LLMs offer lower-cost annotation, but prior work has largely treated them as replacements for humans rather than studying how to allocate work between them while balancing quality and cost.
Source: §1 · pp. 1–2Method
CoAnnotating estimates LLM annotation capability with uncertainty, routes instances between humans and LLMs, and uses Pareto analysis to compare annotation quality and cost.
Source: §3 · p. 3, §3.4 · p. 5Results
Uncertainty-based allocation is more efficient and accurate than random allocation, while confidence scores are generally well-calibrated but not always reliable.
Source: §1 · p. 2Takeaways & Limitations
Some annotation work can be outsourced while maintaining human-level performance on straightforward tasks, whereas nuanced tasks involve a quality–cost tradeoff.
Source: §1 · p. 2The study targets human-level annotation performance, treats human labels as gold, and does not examine where LLMs annotate better than humans.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https://github.com/SALT-NLP/CoAnnotating.
1 Introduction
LLMs offer a lower-cost option for text annotation, but their performance varies by task and can fall short on difficult tasks. CoAnnotating proposes allocating work between humans and LLMs using instance-level uncertainty, with results showing more efficient and accurate allocation than a random baseline.
1 Introduction
- Labeled data remains valuable because fine-tuned models still outperform LLMs on most cited tasks, supporting investigation of LLM-assisted dataset construction.The introduction identifies constructing high-quality datasets and improving fine-tuned models as goals.Source: §1 · p. 1
- LLMs can lower annotation costs and sometimes outperform human annotators, but their zero-shot performance falls short on more difficult and pragmatic tasks.Earlier work largely compared humans and LLMs as competitors rather than examining their collaboration.Source: §1 · pp. 1–2
- CoAnnotating estimates LLM annotation expertise for individual instances, enabling allocation decisions more contextualized than dataset-level assessments.It aims to balance the complementary annotation quality and cost profiles of humans and LLMs.Source: §1 · p. 2
- Uncertainty-guided allocation is more efficient and accurate than random allocation, while LLM confidence is generally well-calibrated but not always reliable.Some work can be outsourced while retaining human-level performance on straightforward tasks; nuanced tasks involve a quality–cost tradeoff.Source: §1 · p. 2
2 Related Work
Prior work addresses annotation through weak supervision, comparisons of LLMs with human annotators, and human-machine collaboration. CoAnnotating builds on these lines by using uncertainty to allocate annotation work while considering both dataset quality and cost.
2.1 Weak Supervision
- Weak supervision draws noisy labels from heuristics, feature annotation, knowledge bases, pretrained models, and third-party tools, and can be combined with active learning.This paper explores LLMs as a more efficient labeling source that may free resources for other research work.Source: §2.1 · p. 2
2.2 LLMs for Annotation
- Prior LLM annotation studies report strong results on some tasks, but also performance gaps on challenging tasks; they typically compare LLMs with humans or fine-tuned models as competitors.CoAnnotating instead treats humans and LLMs as potential collaborators annotating the same dataset.Source: §2.2 · pp. 2–3
2.3 Human-Machine Collaboration for Dataset Creation
- Human-machine dataset creation methods combine generation with human revision, average or vote over machine and human outputs, or explore random and active assignments.CoAnnotating considers quality and cost together, using uncertainty to inform human-AI work allocation.Source: §2.3 · p. 3
3 CoAnnotating Framework
CoAnnotating routes individual unlabeled instances to humans or LLMs using estimated LLM uncertainty. It combines prompt variation, uncertainty-based allocation, and Pareto analysis to examine annotation quality–cost tradeoffs.
3 CoAnnotating Framework
- The framework decides whether each unlabeled training instance should be annotated by a human or an LLM, using LLM uncertainty to pursue higher quality and lower cost.Its workflow includes expertise estimation, work allocation, and cost–performance Pareto analysis, with pilot-dataset insights informing use on the original unlabeled dataset.Source: §3 · p. 3, Figure 2 · p. 4
3.1 Prompt Construction
- Seven prompt types vary sentence order, paraphrased instructions, question format, and confirmation bias to elicit responses for each instance.The prompt examples use paraphrase detection as the concrete task.Source: §3.1 · p. 3, Table 1 · p. 3
3.2 Uncertainty Computation
- The framework estimates instance uncertainty through self-evaluation or entropy using multiple LLM annotations, with different prompts used to capture prompt variation.Self-evaluation requests a confidence score; entropy uses prediction frequencies, with larger entropy indicating more uncertainty.Source: §3.2 · p. 4
3.3 Work Allocation Strategies
- Allocation strategies assign the lowest-uncertainty instances to the LLM and send the remaining instances to humans; random allocation serves as a comparison baseline.Self-evaluation ranks confidence scores, while entropy ranks response consistency.Source: §3.3 · pp. 4–5
3.4 Strategy Selection
- Strategy selection treats annotation quality and cost as competing objectives and uses Pareto-efficient quality–cost combinations to guide the LLM allocation ratio.Quality is measured by the classification performance of a fine-tuned model; cost sums human and LLM annotation costs.Source: §3.4 · p. 5
4 Experiments
The experiments cover six classification datasets spanning topic classification, semantic similarity, and nuanced comprehension. They use ChatGPT annotations and compare annotation strategies through classifier performance and monetary cost.
4.1 Datasets
- The six classification datasets cover topic classification, semantic similarity, and nuanced comprehension, using TREC, AG News, MRPC, TempoWiC, Tweet Stance Detection, and Conversation Gone Awry.Source: §4.1 · p. 5
- When an LLM's utility matches or exceeds a human annotator's, the setup favors the LLM; CoAnnotating supports decisions on harder quality-cost tradeoffs.Source: §4.1 · p. 5
4.2 LLM Annotation
- ChatGPT (gpt-3.5-turbo) annotates through task-specific prompts; ambiguous responses become a new class, and final LLM labels use majority vote across generations.Source: §4.2 · p. 5
4.3 Evaluation
- They compare strategies by fine-tuning the same RoBERTa base classifier and measuring macro F1, chosen because LLM annotations are unbalanced on some datasets.Source: §4.3 · p. 6
- Cost evaluation covers monetary cost, estimating human annotation from dataset reports or a stated fallback and ChatGPT cost from prompt tokens and API pricing.When reported human-cost information is unavailable, the study assumes five independent annotators per instance at $15/hour; ChatGPT pricing is $0.002/1k tokens at experimentation time.Source: §4.3 · p. 6
5 Results
Across allocation comparisons, uncertainty-guided selection improves on random allocation at a fixed share, while entropy is a more dependable capability signal than self-reported confidence. Pareto analysis shows that some tasks permit outsourcing at human-level performance, whereas nuanced tasks incur quality trade-offs.
5.1 Strategy Comparison
- Entropy thresholds track decreasing alignment across datasets, but confidence is unreliable: 94.3% of AG News data scored 0.8–1, while MRPC alignment worsened as confidence rose.ChatGPT’s confidence distribution also skews toward high values.Source: §5.1 · p. 6
- At 40% ChatGPT allocation on TempoWIC, uncertainty-guided annotation produced test F1 56.9 versus 53.2 for random allocation.The comparison reflects downstream fine-tuned model performance.Source: §5.1 · p. 7, Table 2 · p. 7
5.2 Pareto Efficient Allocation
- Pareto curves identify quality-cost-efficient allocations; AG News could outsource 33% at human-level performance, whereas any outsourcing on nuanced stance or conversation tasks compromises quality.The frontier marks choices where neither higher test accuracy at the same budget nor lower cost at the same quality is possible.Source: §5.2 · pp. 7–8, Figure 5 · p. 8
5.3 Qualitative Analysis
- High-uncertainty cases include sarcasm, incomplete sentences, numerical reasoning, and misleading keywords in opinion mining, paraphrase detection, and topic classification.The abortion-stance example “as a former fetus I oppose” illustrates confusion from an incomplete sentence.Source: §5.3 · pp. 8–9
5.4 Ablation Study
- Entropy computed across different prompts better quantifies expertise than same-prompt entropy, which is zero for most data; different-prompt allocation also consistently yields higher test performance.The same-prompt approach still shows some decreasing alignment as its threshold increases.Source: §5.4 · p. 9
6 Conclusion
CoAnnotating treats humans and LLMs as collaborators, using uncertainty to estimate LLM annotation capability and guide work allocation. The framework helps practitioners compare cost-performance tradeoffs.
6 Conclusion
- CoAnnotating uses uncertainty to estimate LLM annotation capability and guide work allocation across datasets.It applies the Pareto efficiency concept to compare strategies and understand cost-performance tradeoffs.Source: §6 · p. 9
- Empirical results demonstrate that the framework can achieve greater cost efficiency.Source: §6 · p. 9
- The framework offers insights into self-reported confidence reliability, prompt sensitivity, and how much human annotation work LLMs may free for more meaningful areas.Source: §6 · p. 9
7 Limitations
The study is an initial exploration targeting human-level annotation performance, with experiments limited to English classification tasks. The authors identify data leakage and the use of human annotations as gold labels as important scope boundaries.
7 Limitations
- Possible dataset leakage may result in lower entropy values for LLM responses.Source: §7 · p. 9
- The study treats human annotations as gold labels and targets human-level, not superhuman-level, annotation performance.Future work could investigate instances where LLMs annotate better than humans.Source: §7 · p. 9
- Experiments cover only classification tasks and English datasets, although the authors describe the idea as generalizable to generation and other languages.Source: §7 · p. 9
- The framework groups human and LLM annotators rather than modeling variation within each group, such as experts, crowd workers, or different LLMs.Source: §7 · p. 9
Ethical Statement
The authors acknowledge that using LLMs as labelers may perpetuate biases present in their Internet training data. They state that these risks include demographic under-representation, cultural stereotypes, and linguistic biases.
Ethical Statement
- Internet-trained LLM labelers may perpetuate under-representation of demographic groups, cultural stereotypes, and linguistic biases.Source: Ethical Statement · p. 9
- The authors believe the benefits of the collaborative co-annotation framework outweigh its potential risks related to these biases.Source: Ethical Statement · p. 9