PaperlayerView exact revision on arXiv

Source-linked AI summary

Want To Reduce Labeling Cost? GPT-3 Can Help

Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, Michael Zeng

arXiv:2108.13487v1cs.CLcs.AI

TL;DR

Human annotation is costly and time-consuming, motivating a more efficient way to create labels for NLP models. The paper uses GPT-3 to label unannotated data for training smaller models, and combines GPT-3 and human labeling under fixed budgets. GPT-3 labeling costs 50% to 96% less for equivalent performance, while mixed-label strategies can perform better than single-source labeling.

  • Problem

    Human-labeled data is costly and time-consuming, while deploying GPT-3 directly requires substantial resources and incurs token-based API charges.

  • Method

    GPT-3 labels unannotated data for training smaller models, with additional strategies mixing GPT-3 and human labels under fixed budgets.

  • Results

    Across NLU and NLG tasks, GPT-3 labels cost 50% to 96% less than human labels for the same performance, and trained in-house models can outperform GPT-3.

  • Takeaways & Limitations

    Combining GPT-3 and human labels can outperform fully human or fully GPT-3 labeling under limited budgets.

  • Takeaways & Limitations

    GPT-3 is not yet reliable enough for high-stakes labeling cases and is more suitable for low-stakes labeling.

Abstract

from arXiv · show

Data annotation is a time-consuming and labor-intensive process for many NLP tasks. Although there exist various methods to produce pseudo data labels, they are often task-specific and require a decent amount of labeled data to start with. Recently, the immense language model GPT-3 with 175 billion parameters has achieved tremendous improvement across many few-shot learning tasks. In this paper, we explore ways to leverage GPT-3 as a low-cost data labeler to train other models. We find that, to make the downstream model achieve the same performance on a variety of NLU and NLG tasks, it costs 50% to 96% less to use labels from GPT-3 than using labels from humans. Furthermore, we propose a novel framework of combining pseudo labels from GPT-3 with human labels, which leads to even better performance with limited labeling budget. These results present a cost-effective data labeling methodology that is generalizable to many practical applications.

1 Introduction

The paper investigates GPT-3 as a low-cost labeler for training smaller models, addressing the expense and time required for human annotation. Across NLP tasks, GPT-3 labels reduce cost, while combining GPT-3 and human labels can further improve performance under fixed budgets.

  • 1 Introduction: GPT-3 labels unannotated data to train smaller in-house models that are deployed for inference.The approach uses GPT-3-generated labels as training supervision rather than directly deploying GPT-3 for downstream tasks.
  • 1 Introduction: In-house models trained on GPT-3 labels can outperform GPT-3 itself under few-shot learning.The paper reports this pattern for models such as PEGASUS and RoBERTa across various NLU and NLG tasks.
  • 1 Introduction: Mixing GPT-3 and human labels under a fixed budget improves performance over using data from either single labeler.The study tests multiple budget ratios for assigning unlabeled data to GPT-3 and human annotators.
  • 1 Introduction: 50% to 96% lower labeling cost achieves the same performance as human labeling across a variety of NLP tasks.The analysis covers NLU and NLG tasks, including entailment, sentiment, topic and answer-type classification, summarization, and question generation.
  • 1 Introduction: An active labeling strategy has humans re-annotate GPT-3-labeled examples with the lowest confidence scores.This targets human effort toward examples where GPT-3 is least confident.

2 Method

The paper uses GPT-3 to label unannotated data for smaller downstream models, analyzes labeling costs, and combines GPT-3 and human supervision under fixed budgets.

  • GPT-3 labeling: GPT-3 labels unannotated data that trains smaller in-house models for downstream inference.The paper uses PEGASUS for NLG tasks and RoBERTalarge for NLU tasks.
  • GPT-3 labeling: Classification labels use GPT-3’s first output token, whereas generation labels use the entire generated output.The GPT-3 input combines several human-labeled sequences with a target input sequence.
  • GPT-3 labeling: GPT-3 prompts contain n human-labeled examples, with more shots typically improving quality but increasing labeling cost.The paper explores 2, 4, and 8 shots for NLU and 1, 2, and 3 shots for NLG; in SST-2, 8-shot labeling costs about 4.5 times more than 1-shot labeling.
  • Downstream models: Raw GPT-3 has inference costs that grow linearly with the number of instances and relatively high deployment latency.The paper instead trains smaller in-house models using GPT-3-generated labels.
  • Downstream models: GPT-3-labeled training can yield in-house classifiers whose error rate is lower than GPT-3’s under the stated consistency and expansion assumptions.The paper frames this improvement as self-training and provides theoretical justification.
  • GPT3-Human labeling: Random GPT3-Human labeling splits a fixed budget across disjoint GPT-3- and human-labeled sets, with an optional weight for noisier GPT-3 supervision.Experiments assign 0%, 25%, 50%, 75%, or 100% of the budget to each labeler.
  • GPT3-Human labeling: Active labeling sends GPT-3 labels with the highest uncertainty to humans for re-annotation, using first-token logits as confidence scores for NLU.The paper reports high correlation between these confidence scores and GPT-3 label accuracy.

3 Experiments

The experiments evaluate GPT-3, human, and mixed labeling strategies across NLU and NLG tasks under controlled labeling budgets. GPT-3 labeling substantially reduces cost in low-budget settings, while mixed and active strategies improve downstream performance.

  • Datasets: The evaluation covers 3 NLG and 6 NLU tasks, sampling up to 5.1K training cases and using original test sets when available.Human labeling is simulated with dataset labels.
  • Labeling strategy: The study compares fully human, fully GPT-3, and GPT-3–human mix-up strategies across shot counts, budget ratios, and random or active labeling.Budgets correspond to human-labeling costs for 10 through 5,120 samples.
  • Main result: Across all tasks in low-budget settings, fully GPT-3 labeling outperforms fully human labeling, while GPT-3–human mixing improves performance further.Figure 3 reports maxima over labeling hyperparameters with means and standard deviations from three trials.
  • Main result: 50%–96% labeling-cost savings achieve performance comparable to human labels under low budgets.On SST-2, $1.1 of GPT-3 labels matches $27.5 of human labels, while Gigaword saves 93.8% at the reported budgets.
  • Main result: When budgets are ample or unlimited, human labeling dominates because of higher label quality; with limited budgets, GPT-3 labeling is more cost-effective.For most NLU datasets, fewer-shot labeling can also label more data under the same budget, whereas label quality becomes pivotal as budgets increase.
  • Main result: In-house models trained with enough GPT-3 labels outperform raw GPT-3, extending the approach beyond cost-efficient annotation.The paper characterizes this as a semi-supervised method for boosting few-shot learners.
  • Active labeling: Active labeling uses GPT-3 logits as confidence scores and has humans relabel low-confidence instances, improving downstream performance over random labeling.In TREC, active labeling raises accuracy from 77% to 80% under the same $2.2 budget.

4 Related Work

The paper relates its cost-aware GPT-3 labeling approach to GPT-3 prompting, dual supervision, and self-training. Its distinction is explicit treatment of GPT-3 labeling cost when training smaller models.

  • GPT-3 Overview: Prior GPT-3 work studies bias removal and prompt-example ordering, while concurrent work distills GPT-3 knowledge with synthetic data.These works address GPT-3 use or improvement but differ in focus from this paper’s cost analysis.
  • Dual supervision: Dual supervision combines cheap and expensive labels; this paper applies that framework to different-cost labelers performing the same task.Earlier dual-supervision work often used different labeling tasks for humans.
  • Semi-supervised learning and Self Training: Self-training usually uses similar-sized teacher and student models with negligible teacher-labeling cost, unlike GPT-3 labeling.The paper presents explicit consideration of GPT-3 cost as a distinguishing challenge.
  • Semi-supervised learning and Self Training: The paper claims to be the first work explicitly considering GPT-3’s cost and its effect on reducing labeling cost.This claim is made in the context of prior self-training literature.

5 Conclusion

The paper concludes that GPT-3 can label unannotated data cost-effectively, that trained models can surpass raw GPT-3, and that mixing with humans improves results. It also identifies high-stakes labeling as a boundary.

  • Conclusion: GPT-3 labeling strategies significantly reduce cost while achieving the same performance as human-labeled data.The conclusion frames this as cost-efficient labeling with few-shot learners.
  • Conclusion: Models trained with GPT-3 labels can achieve better performance than raw GPT-3.This supports using GPT-3 labels to train smaller deployed models rather than deploying GPT-3 directly.
  • Conclusion: GPT3-Human outperforms both fully human and fully GPT-3 labeling, while active labeling beats random selection on multiple NLP tasks.Active labeling leverages GPT-3 confidence to direct human relabeling.
  • Future work: The authors plan to extend the methods to data augmentation that produces both instances and labels.This is stated as future work rather than an evaluated capability.
  • Scope boundary: GPT-3 is not yet reliable enough for high-stakes labeling, such as toxic-language identification, and is more suitable for low-stakes labeling.The stated scope boundary concerns labeling reliability in high-stakes cases.

A Proof of Theorem 2

The appendix defines the (a, c)-expansion property for class-conditional distributions and states that Theorem 2 follows directly from prior theoretical results under a consistency assumption.

  • Definition: The paper defines (a, c)-expansion for a class-conditional distribution using sets with class probability at most a and a distribution-dependent neighborhood.The full neighborhood definition is deferred to Wei et al. (2021).
  • Definition: If every class satisfies (a, c)-expansion, the overall sample distribution is said to satisfy (a, c)-expansion.This lifts the property from class-conditional distributions to the full distribution.
  • Proof: Theorem 2 is presented as a direct consequence of Theorem 4.3 in Wei et al. (2021).The proof uses the consistency assumption to obtain RB(G) = µ = 0 for the classifiers considered.

B GPT-Human Labeling

Figure 6 presents GPT3–human labeling performance when the labeling budget is equally split and active labeling is used.

  • The figure compares GPT3 and human labeling performance.
  • The labeling budget is equally split between GPT3 and human labeling.
  • The experiment adopts active labeling.
Loading 2108.13487v1…