Source-linked AI summary
It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Timo Schick, Hinrich Schütze
TL;DR
Large few-shot language models achieve strong performance but require substantial compute and have limited scalability. This paper adapts PET to multiple-token predictions and combines it with ALBERT, showing similar few-shot SuperGLUE performance with models three orders of magnitude smaller. The authors identify pattern combination, labeled-data updates, and the underlying language model as contributors, while leaving generative-task performance for future work.
Problem
GPT-3’s few-shot performance relies on enormous models, high compute, and context windows that limit the number of usable examples.
Method
The paper adapts PET to predict multiple tokens and combines cloze reformulation with gradient-based training, including iterative self-training with unlabeled data.
Results
PET with ALBERT achieves few-shot SuperGLUE performance similar to GPT-3 while using language models with three orders of magnitude fewer parameters.
Takeaways & Limitations
The results indicate that strong few-shot text classification can be achieved with substantially smaller language models, reducing financial cost and environmental impact.
Takeaways & Limitations
The paper leaves open whether PET works for generative tasks when combined with generative language models.
Abstract
from arXiv · showhide
When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.
1 Introduction
Few-shot language modeling can be effective, but GPT-3’s approach depends on enormous models, high compute, and a context window that limits the number of examples. The paper adapts PET for multiple-token predictions and reports comparable or better SuperGLUE performance with ALBERT using far fewer parameters.
- GPT-3 achieves near state-of-the-art results on some SuperGLUE tasks from 32 labeled examples by using demonstrations as context without gradient updates.
- GPT-3’s priming requires a gigantic language model, creating substantial usability and carbon-footprint concerns.
- Priming does not scale beyond a few examples because language-model context windows are limited.
- PET combines cloze-question reformulation with gradient-based finetuning and uses unlabeled data, which is easier to obtain than labeled data.
- PET is adapted for multiple-token tasks, and ALBERT with PET or iPET outperforms GPT-3 on SuperGLUE with 32 training examples while using 0.1% of its parameters.
2 Related Work
Prior work reformulates tasks as cloze questions for zero- or few-shot language-model prediction, while PET combines such reformulations with distillation and self-training. This work also fits within Green AI efforts to reduce the compute demands of language processing.
- Task descriptions and cloze reformulations have been used for zero-shot text classification, knowledge mining, relation classification, and probing language-model knowledge.
- PET addresses difficult task reformulations by combining several patterns through knowledge distillation and self-training.
- Priming provides examples as context for predictions without updating model parameters.
- Reducing compute for few-shot learning is related to Green AI techniques including distillation, pruning, and quantization.
3 Pattern-Exploiting Training
Pattern-exploiting training reformulates tasks as cloze questions and derives output probabilities from masked language model predictions. The paper extends PET to multi-token verbalizers and combines pattern-specific models through distillation and iterative self-training.
- Pattern-verbalizer pairs: PET uses pattern-verbalizer pairs to map inputs into cloze questions and outputs into task-specific vocabulary tokens.A pattern creates a masked cloze question, while a verbalizer represents each output with a token.
- PET training: Multiple PVPs are combined by finetuning masked language models, annotating unlabeled examples with soft labels, and training a sequence classifier by distillation.The ensemble can be processed sequentially, so it does not require more memory than a single model.
- Iterative PET: iPET iteratively trains model generations on increasingly large datasets labeled by earlier generations and confident subsets of other models.Each generation retrains models on examples selected for confidence, repeating the process while increasing dataset size.
- PET with multiple masks: PET generalizes verbalizers from single tokens to token sequences and allows the possible output set to vary by input.The maximum verbalizer length determines how many masks are inserted into the pattern.
- Inference: For multi-token verbalizers, inference autoregressively selects the highest-confidence token, inserts it, and predicts the remaining token.In the restaurant-review example, the verbalizer for “terrible” is split into terri and •ble.
- Training with multiple masks: Training predicts all verbalizer tokens in parallel using the maximum required number of masks, enabling the needed probabilities in a single forward pass.Superfluous mask predictions are ignored, and a multi-class hinge loss enforces a log-probability margin of at least 1 over competing outputs.
4 Experiments
The experiments evaluate PET and iPET with ALBERT on eight SuperGLUE tasks using few labeled examples and additional unlabeled data. PET performs similarly to the largest GPT-3 model while using a much smaller model, though it remains below full-data state-of-the-art performance.
- Experimental setup: PET and GPT-3 are compared on eight SuperGLUE natural language understanding tasks using few-shot training settings.The authors construct new training sets because GPT-3 and PET do not use exactly the same available examples.
- Experimental setup: FewGLUE provides up to 20,000 unlabeled examples per task by removing labels from the original training sets.These unlabeled examples support PET’s training procedure.
- Tasks: The benchmark includes tasks covering causal reasoning, word-sense consistency, question answering, entailment, pronoun resolution, and cloze-style answer selection.The experiments use task-specific patterns and verbalizers across the SuperGLUE task suite.
- Experimental setup: ALBERT-xxlarge-v2 serves as PET’s underlying masked language model and final sequence classifier, with no development set used for hyperparameter optimization.The setup follows the same hyperparameters as the prior PET work.
- Results: PET with ALBERT performs similarly to the largest GPT-3 model, which is 785 times larger, on the reported SuperGLUE comparison.The table compares GPT-3 priming with 32 examples against PET and iPET trained on FewGLUE.
- Results: PET averages 18 points higher than GPT-3 Med, while iPET improves three of five evaluated tasks but slightly lowers MultiRC performance.Despite strong few-shot results, PET remains clearly below a state-of-the-art model trained on the full SuperGLUE training sets.
5 Analysis
The analysis examines how patterns, labeled and unlabeled data, model architecture, decoding, and example selection affect PET’s few-shot performance. Across these factors, suitable cloze formulations, bidirectional language models, and careful training choices are central, while performance remains sensitive to data and configuration.
- 5.1 Patterns: Combining multiple patterns compensates for task-specific weaknesses and further improves average performance over the best individual pattern set.The GPT-3 pattern performs better on RTE, while the initial patterns perform better on MultiRC; their combination addresses both weaknesses.
- 5.2 Unlabeled Data Usage: Unlabeled data is unnecessary for good performance but required to distill PET’s ensemble into a single lightweight classifier.Without distillation, the ensemble performs even better on average across three tasks, although it is larger than the distilled model.
- 5.2 Unlabeled Data Usage: The first-to-second iPET generation yields substantial gains for most tasks, while later generations add little; final distillation further improves results and nearly eliminates run variability.MultiRC is the main exception to these improvements, and later generations show reduced standard deviation.
- 5.3 Labeled Data Usage: PET clearly outperforms supervised and fully unsupervised baselines with 32 labeled examples, and it outperforms priming in the direct XLNet comparison.Priming also scales poorly to many demonstrations because sequence-length and self-attention costs constrain the number of examples.
- 5.4 Model Type: ALBERT is crucial for PET’s strong performance: replacing it with RoBERTa causes an average drop of 8 points, while GPT-2 performs much worse.The authors argue that bidirectionality is important because it enables more flexible pattern creation, while noting other differences between models may also matter.
- 5.5 PET with Multiple Masks: PET’s multiple-mask decoding strategy generally outperforms parallel and left-to-right decoding, with WSC as the exception for parallel decoding.PET also clearly outperforms untrained ALBERT on the three evaluated tasks, while distillation effects vary by task.
- 5.6 Training Examples: Changing the few-shot training examples can cause large performance differences for PET, although its average performance is similar to GPT-3 across all seeds.This sensitivity motivates using identical example sets when comparing few-shot approaches.
6 Conclusion
The paper shows that PET can deliver GPT-3-like few-shot text classification performance with models three orders of magnitude smaller, reducing environmental impact. It identifies several contributors to PET’s performance and outlines generative and multitask extensions for future work.
- PET combined with ALBERT benefits from multiple concurrent patterns, compensation for difficult patterns, labeled-data updates, and the underlying language model.
- PET achieves few-shot text classification performance similar to GPT-3 on SuperGLUE with language models that have three orders of magnitude fewer parameters.
- The smaller models substantially reduce financial cost and environmental impact, including their carbon footprint.
- Future work could test PET for generative tasks with generative language models and pursue improvements in multitask settings.
A Training Details
The training details describe the implementation, preprocessing, task-specific input handling, and computational setup used for PET and related experiments.
- The implementation extends PET using Transformers and PyTorch, with reproduction instructions and dependencies documented in the supplementary material and README.
- Training uses the original PET hyperparameters except that iPET runs three generations, with experiments conducted on one 11GB GPU.A single PET model required approximately 45 minutes for 250 steps, while labeling unlabeled examples took 0.2–1.5 hours per model.
- Task-specific procedures include option or sentence-order randomization, sequence-boundary markers, and specialized formats for WSC, MultiRC, and ReCoRD.MultiRC and ReCoRD use maximum sequence lengths of 512 tokens; ReCoRD examples with many candidates are split into multiple training examples.
- Examples are shortened to the maximum sequence length before patterns are applied, using the Transformers library’s longest-first truncation strategy.
- For candidate-based training, each correct answer is paired with up to nine randomly selected negative examples, producing ten answer candidates.
B Dataset Details
The dataset construction uses fixed 32-example few-shot sets, task-specific handling for certain datasets, and unlabeled data drawn primarily from training examples.
- FewGLUE sets contain the first 32 examples after fixed-seed shuffling, with positive-only WSC examples and 32 selected questions for MultiRC and ReCoRD.For MultiRC and ReCoRD, 32 questions correspond to more than 32 training examples.
- Unlabeled data includes up to 20,000 label-removed training examples per task, supplemented with MNLI examples for the small RTE and CB datasets.Evaluation uses official validation and test sets, and all SuperGLUE datasets are in English.