Towards Zero-Label Language Learning

1 Introduction

• Pretraining on large unsupervised text corpora lets models learn contextualized representations that can be fine-tuned with relatively small task-specific datasets.The passage identifies this as a successful transfer-learning approach for reducing labeled-data requirements.Source: §1 · p. 1
• Few-shot inference with giant language models remains well below fine-tuned state-of-the-art methods on many tasks, including a 17.5-point SuperGLUE gap.The passage describes this gap as motivation to explore uses of giant language models beyond direct few-shot inference.Source: §1 · p. 1
• UDG prompts a pretrained language model with a task description, a few unlabeled examples, and a desired label to generate synthetic training examples.Task-specific models are then trained on the generated data, without human labels throughout the process.Source: §1 · pp. 1–2
• UDG achieves competitive results against strong supervised baselines, new state-of-the-art few-shot results on text classification and SuperGLUE, and the first reported SuperGLUE scores above human performance as of Dec 20, 2020.When combined with labeled data, UDG achieves the reported super-human SuperGLUE scores.Source: §1 · pp. 1–2

• Data augmentation has been popular for NLP quality improvement, especially in low-resource regimes; earlier approaches used token-level modifications to diversify samples.Source: §2 · p. 2
• Generative data augmentation often requires labeled examples to fine-tune generation models and heavy postprocessing to clean the data.Source: §2 · p. 2
• Unlike these approaches, the method generates data fully unsupervised and without fine-tuning the language model, presenting a zero-label learning paradigm.Source: §2 · p. 2
• The approach is related to knowledge retrieval: pretrained models’ retrieval ability largely guarantees high-quality examples and reduces irrelevant hallucinations.The passage connects this ability to models’ capacity to memorize training facts and serve as open knowledge bases (Petroni et al., 2019; Wang et al., 2020; Carlini et al., 2021).Source: §2 · p. 2

3.1 Background: Few-shot Inference

• Few-shot inference prompts a language model with a task description and a few input-label examples to infer the label for a query without gradient updates.The method uses much less task-specific data than fine-tuning; decoding commonly uses greedy search or beam search because exact argmax is intractable.Source: §3.1 · pp. 2–3
• GPT3 scores 71.8 versus T5’s 89.3 on SuperGLUE, illustrating that few-shot inference can underperform state-of-the-art fine-tuned models.The proposed explanation is that prompting for labels may force generation in a text distribution inconsistent with the model’s training data.Source: §3.2 · p. 3

3.2 Unsupervised Data Generation

• UDG prompts a language model with unlabeled examples and a natural-language description of a desired label to generate an input example, rather than predict the label.This decouples generation from prediction and uses prompts intended to resemble naturally occurring training-corpus text.Source: §3.2 · p. 3
• UDG creates a controllably sized synthetic labeled dataset for standalone or auxiliary task-specific training, without fine-tuning the generative model.The synthetic examples are generated using unsupervised data only.Source: §3.2 · p. 3

4.1 Unsupervised Text Classification

• For classification, the experiments generate one example per prompt using 32 unlabeled examples, with generation repeated 10K times per sentiment class and 1,000 times per topic class.The generated dataset is used as labeled training data with UDA and BERTLarge; NLA is added to filter noisy examples.Source: §4.1 · p. 4
• With NLA, UDG beats UDA and few-shot inference on all six classification tasks, and label-free UDG surpasses supervised BERTLarge on IMDb and Yelp-2.NLA filters examples when task-specific models disagree with synthetic labels at high confidence, improving UDG especially on noise-sensitive DBpedia.Source: §4.1 · p. 4
• More unlabeled examples in prompts improve synthetic-data quality and downstream performance, while generation without any example produces low-quality data.The prompt-example ablation is reported in Table 3.Source: Table 3 · p. 4, §4.1 · p. 5
• Performance continues to improve as more examples are generated per label class, though the gains diminish.The authors state that sampling more data requires no additional cost or supervision.Source: Figure 2 · p. 4, §4.1 · p. 5

4.2 Unsupervised Language Understanding

• On SuperGLUE, unsupervised few-shot generation outperforms all few-shot learning systems without labels, with gains concentrated on CB, RTE, and word sense disambiguation.The experiments use eight SuperGLUE tasks and fine-tune T5 on generated data; the T5 models are pretrained on unsupervised C4 data.Source: §4.2 · pp. 5–6

4.3 UDG as Data Augmentation

• Combined labeled and generated data improves SuperGLUE results and surpasses prior methods and human baselines.The combined model uses the public T5-XXL checkpoint with both labeled and generated data.Source: Table 4 · p. 5, §4.3 · p. 6

• The proposed zero-label procedure uses language models as few-shot example creators to generate high-quality synthetic data without human annotation.Source: §5 · p. 6
• The paper demonstrates that NLP models can obtain strong results without any human-annotated labels.Source: §5 · p. 6
• The authors describe this approach as a promising direction for future transfer-learning research in NLP.Source: §5 · p. 6

• NLA removes a synthetic example when the model predicts a different label with probability above the current threshold.The initial threshold is 0.9 and is gradually annealed toward 1/K, where K is the number of classes.Source: A Noisy Label Annealing · p. 8
• The threshold schedule demands high confidence early in fine-tuning, then lowers the bar as the model becomes better trained.Source: A Noisy Label Annealing · p. 8
• More aggressive final threshold settings often perform better, and regularization is especially helpful on tasks sensitive to noise.Source: A Noisy Label Annealing · p. 8

• Text-classification inputs are truncated to 512 subwords, with learning rate, batch size, and epochs tuned across specified ranges.The learning rates are 1e-5, 2e-5, and 5e-5; batch sizes are 32, 64, and 128; epochs range from 5 to 30.Source: B Finetuning Details · p. 8
• For text classification, the setup also fine-tunes BERT on in-domain unlabeled data before final training, and TSA can be applied alongside NLA.Source: B Finetuning Details · p. 8
• For SuperGLUE, the unsupervised T5-Large pretraining omits the original downstream-task data used for multi-task pretraining.Fine-tuning uses batch size 8 and sequence length 512, with dev scores selecting the checkpoint for test-server submission.Source: B Finetuning Details · p. 8

C.1.1 IMDb

• C.1.1 IMDb samples include a saccharine romance review and a sharply negative review of a remake.Source: §C.1.1 · p. 9

C.1.2 Yelp-2; C.1.3 Yelp-5

• The sample business review praises its burger, truffle fries, sauce, service, and food; a Yelp rating passage states 3/5, while a separate reviewer calls the meal fine after a 35-minute wait.Source: §C.1.2 · p. 9, §C.1.3 · pp. 9–10

C.1.4 Amazon-2; C.1.5 Amazon-5

• The passages include a dog-bed review praising comfortable sleep, a disappointed doll review describing confusing assembly and a hole in the doll’s head, and separate reviews of a book and a game.Source: §C.1.4 · p. 10, §C.1.5 · pp. 10–11

C.2.4 MultiRC; C.2.5 ReCoRD

• The cited examples pair an article with a question and answer, while other article passages include a keyword followed by a relevant fact or a generated-output field.Source: §C.2.4 · p. 12, §C.2.5 · p. 13

C.2.5 ReCoRD; C.2.6 RTE

• The passage describes China’s Moon mission as ambitious: on December 14 it achieved the first lunar soft landing in nearly four decades and aims at a manned mission by 2025.Source: §C.2.5 · p. 13

C.2.7 WiC; C.2.8 WSC

• The excerpts show a “meet” word/example prompt and a “skin” definition/example prompt, alongside pronoun-reference prompts for Sam/he and Ann/She, including a sentence explicitly stating that She refers to Ann.Source: §C.2.7 · p. 13, §C.2.8 · p. 14