Source-linked AI summary
Automated Annotation with Generative AI Requires Validation
Nicholas Pangakis, Samuel Wolken, Neil Fasching
TL;DR
Manual annotation can be time-consuming and expensive, while LLM annotation performance varies across tasks. The paper proposes a workflow for validating LLM labels against expert-labeled data and reports both strong labeling potential and substantial variation across tasks, including within datasets.
Problem
Manual annotation can be time-consuming and expensive, and crowd workers may experience limited attention and fatigue.
Source: §1 · p. 2Method
The paper proposes a workflow for augmenting annotation projects and defines high-quality reference labels as a carefully labeled subset from subject-matter experts.
Source: §1 · p. 3Results
LLMs can provide high-quality labels for a fraction of the cost and time of alternatives, but performance varies even across tasks within a dataset.
Source: §5 · p. 15Takeaways & Limitations
Task-level variation means the paper’s reported potential does not imply uniform performance across annotation tasks.
Source: §5 · p. 15Reclassifying the same cases after codebook edits overfits that subset and should not be used as the performance benchmark.
Source: §2 · p. 6
Abstract
from arXiv · showhide
Generative large language models (LLMs) can be a powerful tool for augmenting text annotation procedures, but their performance varies across annotation tasks due to prompt quality, text data idiosyncrasies, and conceptual difficulty. Because these challenges will persist even as LLM technology improves, we argue that any automated annotation process using an LLM must validate the LLM's performance against labels generated by humans. To this end, we outline a workflow to harness the annotation potential of LLMs in a principled, efficient way. Using GPT-4, we validate this approach by replicating 27 annotation tasks across 11 datasets from recent social science articles in high-impact journals. We find that LLM performance for text annotation is promising but highly contingent on both the dataset and the type of annotation task, which reinforces the necessity to validate on a task-by-task basis. We make available easy-to-use software designed to implement our workflow and streamline the deployment of LLMs for automated annotation.
1 Introduction
LLMs may make text annotation faster and cheaper, but performance varies across tasks, and prior findings offer limited guidance for new datasets. The paper argues for task-by-task validation against high-quality human labels and tests a workflow for doing so.
1 Introduction
- Ineffective prompts, noisy text, and difficult tasks can undermine LLM annotation, so researchers should validate against high-quality human labels task by task.The paper defines high-quality labels as a carefully labeled subset from subject-matter experts.Source: §1 · p. 3
- The authors replicate 27 annotation processes across 11 non-public datasets from recent high-impact social-science articles, classifying over 200,000 samples.Source: §1 · p. 4
- 0.850 median accuracy and 0.707 median F1 were reported across tasks, but performance varied by dataset and task; nine tasks had precision or recall below 0.5.These results reinforce the need to validate LLM performance for each task.Source: §1 · p. 4
- The paper identifies four validation-guided uses: checking human labels, prioritizing cases for review, creating data for supervised classifiers, and labeling an entire corpus.Source: §1 · p. 4
- Repeated classifications produce a consistency score, and higher scores correlate with correctness, making the score useful for identifying edge cases.The score approximates consistency by comparing repeated classifications with the modal answer.Source: §1 · pp. 4–5
2 Workflow and validation
The workflow pairs task-specific instructions and human annotation with LLM predictions, then evaluates and refines the model before deciding whether to automate. The authors also describe its evaluation scope, sample-size guidance, and annotation costs.
2 Workflow and validation
- The five-step workflow begins with task-specific codebook instructions, then has at least two subject-matter experts and the LLM annotate the same samples.Source: §2 · pp. 5–6
- Use the same codebook as the LLM prompt, compare predictions with human labels using accuracy, recall, precision, and F1, and refine instructions if initial performance is low.Source: §2 · p. 6
- After refinement, test against the remaining human-labeled samples; held-out performance determines whether the LLM can be used for automated annotation.Source: §2 · p. 6
- The recommended sample size is 250–1,250, though tasks targeting cases below 1% prevalence may require substantially more hand-labeled samples.Sample size depends on task type and class imbalance.Source: §2 · p. 6
- The evaluation uses original human labels as ground truth and secured, non-public datasets to reduce potential contamination; multiclass dimensions are treated as separate binary tasks.Source: §2 · p. 7
- Across the 27 tasks, GPT-4 labeled slightly over 200,000 samples for approximately $420; a 1,000-sample dataset took about 2–3 hours for seven iterations.The authors describe the low cost and relatively rapid speed as potential value for social-science text analysis.Source: §2 · pp. 7–8
- The authors observe no relationship between LLM performance and whether the original codebook was available.Source: §2 · p. 7
3 Results
Across 27 replicated tasks, classification performance was promising but variable; consistency tracked performance, while codebook updates generally brought modest gains.
3 Results
- Across 27 tasks, LLM classification achieved a median F1 score; in an extreme example, F1 ranged from 0.259 to 0.811 across two separate tasks within Card et al. (2022).Table 1 reports LLM classification performance across 27 tasks from 11 datasets.Source: §3 · pp. 8, 10, Table 1 · p. 8
- 20 of 27 tasks had stronger recall than precision; eight exceeded 0.7 on both, while nine had precision or recall below 0.5 and three had both below 0.5.These task-level precision and recall comparisons are shown in Figure 2.Source: §3 · p. 8, Figure 2 · p. 9
3.1 Consistency scores
- Consistency scores correlated with accuracy, TPR, and TNR; fully consistent classifications had TPR 16.4 and TNR 21.4 percentage points higher.85.1% of labeled samples had a consistency score of 1.0, making the scores useful for identifying edge cases or difficult annotations.Source: §3.1 · p. 10, Figure 3 · p. 11
3.2 Codebook updates
- Codebook updates usually brought modest accuracy and F1 improvements, driven by more frequent precision gains, while recall decreased more often than it improved.The magnitude of improvement was generally small, and the authors found prompt construction was not a significant performance lever.Source: §3.2 · p. 12, Figure 4 · p. 13
4 Use cases
The authors describe four potential use cases for automated annotation, with integration choices dependent on validation performance and project constraints.
4 Use cases
- Researchers’ LLM integration choices may depend on performance against human-labeled data, budget, dataset size, and human-annotator availability.The paper presents four potential use cases for automated annotation with LLMs.Source: §4 · p. 13, Table 2 · p. 14
- When performance is poor, researchers can refine the codebook with new labeled data, stop using LLMs for unsatisfactory dimensions, or disaggregate complex annotations.For hate speech, the paper gives threats, slurs, and stereotyping as possible component indicators to add to the codebook.Source: §4 · pp. 13, 15
5 Conclusion
The paper reports substantial variation in LLM annotation performance across social science tasks and recommends a flexible, human-centered workflow. Its 27-task replication finds performance varies even within datasets, supporting task-by-task validation.
5 Conclusion
- The authors recommend a flexible approach to LLM-augmented annotation that foregrounds human annotation and supports several workflow use cases.They present an original workflow and validate it by replicating 27 annotation tasks from 11 articles.Source: §5 · p. 15
- Across 27 annotation tasks from 11 high-impact social science articles, the authors report substantial performance heterogeneity, including across tasks within a dataset.They conclude that LLM performance depends on attributes of the text and the conceptual categories being measured.Source: §5 · p. 15
- LLMs can offer high-quality labels at a fraction of the cost and time of crowdsourced workers and undergraduate research assistants, but require task-by-task validation.The authors report significant heterogeneity even across tasks within a single dataset.Source: §5 · p. 15
Appendix
The appendix includes tables listing the sources of replicated annotation tasks and describing those tasks.
Appendix
- Table A1 lists the sources of annotation tasks replicated in the analysis.Source: Table A1 · p. 18
- The appendix uses Saha et al. (2023) to emphasize the Gab hate speech corpus’s application in social science, rather than citing only the original labeled-data source, Kennedy et al. (2022).Source: Appendix · p. 19
- Table A2 provides descriptions of the annotation tasks replicated in the analysis.Source: Table A2 · p. 20