Want To Reduce Labeling Cost? GPT-3 Can Help

1 Introduction

• GPT-3 labels unannotated data to train smaller in-house models that are deployed for inference.The approach uses GPT-3-generated labels as training supervision rather than directly deploying GPT-3 for downstream tasks.Source: §1 · p. 1
• In-house models trained on GPT-3 labels can outperform GPT-3 itself under few-shot learning.The paper reports this pattern for models such as PEGASUS and RoBERTa across various NLU and NLG tasks.Source: §1 · pp. 1–2
• Mixing GPT-3 and human labels under a fixed budget improves performance over using data from either single labeler.The study tests multiple budget ratios for assigning unlabeled data to GPT-3 and human annotators.Source: §1 · pp. 1–2
• 50% to 96% lower labeling cost achieves the same performance as human labeling across a variety of NLP tasks.The analysis covers NLU and NLG tasks, including entailment, sentiment, topic and answer-type classification, summarization, and question generation.Source: §1 · pp. 1–2
• An active labeling strategy has humans re-annotate GPT-3-labeled examples with the lowest confidence scores.This targets human effort toward examples where GPT-3 is least confident.Source: §1 · p. 2

GPT-3 labeling

• GPT-3 labels unannotated data that trains smaller in-house models for downstream inference.The paper uses PEGASUS for NLG tasks and RoBERTalarge for NLU tasks.Source: §2 · p. 2, §2.2 · p. 4
• Classification labels use GPT-3’s first output token, whereas generation labels use the entire generated output.The GPT-3 input combines several human-labeled sequences with a target input sequence.Source: §2.2 · p. 3
• GPT-3 prompts contain n human-labeled examples, with more shots typically improving quality but increasing labeling cost.The paper explores 2, 4, and 8 shots for NLU and 1, 2, and 3 shots for NLG; in SST-2, 8-shot labeling costs about 4.5 times more than 1-shot labeling.Source: §2.2 · pp. 3–4

Downstream models

• Raw GPT-3 has inference costs that grow linearly with the number of instances and relatively high deployment latency.The paper instead trains smaller in-house models using GPT-3-generated labels.Source: §2 · p. 2, §2.3 · p. 4
• GPT-3-labeled training can yield in-house classifiers whose error rate is lower than GPT-3’s under the stated consistency and expansion assumptions.The paper frames this improvement as self-training and provides theoretical justification.Source: §2.3 · p. 4

GPT3-Human labeling

• Random GPT3-Human labeling splits a fixed budget across disjoint GPT-3- and human-labeled sets, with an optional weight for noisier GPT-3 supervision.Experiments assign 0%, 25%, 50%, 75%, or 100% of the budget to each labeler.Source: §2.4 · p. 4
• Active labeling sends GPT-3 labels with the highest uncertainty to humans for re-annotation, using first-token logits as confidence scores for NLU.The paper reports high correlation between these confidence scores and GPT-3 label accuracy.Source: §2.4 · p. 4

Datasets

• The evaluation covers 3 NLG and 6 NLU tasks, sampling up to 5.1K training cases and using original test sets when available.Human labeling is simulated with dataset labels.Source: §3.1 · p. 5

Labeling strategy

• The study compares fully human, fully GPT-3, and GPT-3–human mix-up strategies across shot counts, budget ratios, and random or active labeling.Budgets correspond to human-labeling costs for 10 through 5,120 samples.Source: §3.2 · p. 6

Main result

• Across all tasks in low-budget settings, fully GPT-3 labeling outperforms fully human labeling, while GPT-3–human mixing improves performance further.Figure 3 reports maxima over labeling hyperparameters with means and standard deviations from three trials.Source: §3.3.1 · p. 6
• 50%–96% labeling-cost savings achieve performance comparable to human labels under low budgets.On SST-2, $1.1 of GPT-3 labels matches $27.5 of human labels, while Gigaword saves 93.8% at the reported budgets.Source: §3.3.1 · p. 7
• When budgets are ample or unlimited, human labeling dominates because of higher label quality; with limited budgets, GPT-3 labeling is more cost-effective.For most NLU datasets, fewer-shot labeling can also label more data under the same budget, whereas label quality becomes pivotal as budgets increase.Source: §3.3.1 · p. 7, §3.3.2 · p. 7
• In-house models trained with enough GPT-3 labels outperform raw GPT-3, extending the approach beyond cost-efficient annotation.The paper characterizes this as a semi-supervised method for boosting few-shot learners.Source: §3.3.2 · p. 7

Active labeling

• Active labeling uses GPT-3 logits as confidence scores and has humans relabel low-confidence instances, improving downstream performance over random labeling.In TREC, active labeling raises accuracy from 77% to 80% under the same $2.2 budget.Source: §3.3.3 · p. 7, Figure 5 · p. 7

GPT-3 Overview

• Prior GPT-3 work studies bias removal and prompt-example ordering, while concurrent work distills GPT-3 knowledge with synthetic data.These works address GPT-3 use or improvement but differ in focus from this paper’s cost analysis.Source: §4 · p. 8

Dual supervision

• Dual supervision combines cheap and expensive labels; this paper applies that framework to different-cost labelers performing the same task.Earlier dual-supervision work often used different labeling tasks for humans.Source: §4 · p. 8

Semi-supervised learning and Self Training

• Self-training usually uses similar-sized teacher and student models with negligible teacher-labeling cost, unlike GPT-3 labeling.The paper presents explicit consideration of GPT-3 cost as a distinguishing challenge.Source: §4 · p. 8
• The paper claims to be the first work explicitly considering GPT-3’s cost and its effect on reducing labeling cost.This claim is made in the context of prior self-training literature.Source: §4 · p. 8

Conclusion

• GPT-3 labeling strategies significantly reduce cost while achieving the same performance as human-labeled data.The conclusion frames this as cost-efficient labeling with few-shot learners.Source: §5 · p. 8
• Models trained with GPT-3 labels can achieve better performance than raw GPT-3.This supports using GPT-3 labels to train smaller deployed models rather than deploying GPT-3 directly.Source: §5 · p. 8
• GPT3-Human outperforms both fully human and fully GPT-3 labeling, while active labeling beats random selection on multiple NLP tasks.Active labeling leverages GPT-3 confidence to direct human relabeling.Source: §5 · p. 8

Future work

• The authors plan to extend the methods to data augmentation that produces both instances and labels.This is stated as future work rather than an evaluated capability.Source: §5 · p. 8

Scope boundary

• GPT-3 is not yet reliable enough for high-stakes labeling, such as toxic-language identification, and is more suitable for low-stakes labeling.The stated scope boundary concerns labeling reliability in high-stakes cases.Source: §5 · p. 8

Definition

• The paper defines (a, c)-expansion for a class-conditional distribution using sets with class probability at most a and a distribution-dependent neighborhood.The full neighborhood definition is deferred to Wei et al. (2021).Source: A Proof of Theorem 2 · p. 11
• If every class satisfies (a, c)-expansion, the overall sample distribution is said to satisfy (a, c)-expansion.This lifts the property from class-conditional distributions to the full distribution.Source: A Proof of Theorem 2 · p. 11

Proof

• Theorem 2 is presented as a direct consequence of Theorem 4.3 in Wei et al. (2021).The proof uses the consistency assumption to obtain RB(G) = µ = 0 for the classifiers considered.Source: A Proof of Theorem 2 · p. 11

• The figure compares GPT3 and human labeling performance.Source: Figure 6 · p. 11
• The labeling budget is equally split between GPT3 and human labeling.Source: Figure 6 · p. 11
• The experiment adopts active labeling.Source: Figure 6 · p. 11