Source-linked AI summary
Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference
Timo Schick, Hinrich Schütze
TL;DR
Few-shot NLP often has too few labeled examples for standard supervised learning, while task descriptions have mainly been used without training data. PET combines natural-language cloze patterns with semi-supervised labeling, and its iterative variant extends the process; across diverse tasks and languages, they substantially outperform major baselines in low-resource settings. The method's reported evaluation includes a limitation from an excluded underfitting MNLI run and hyperparameters chosen without a large development set.
Problem
Few-shot NLP is challenging because small labeled datasets often make standard supervised learning perform poorly, while task descriptions had mostly been used in zero-shot settings.
Method
PET uses pattern-verbalizer pairs to reformulate examples as cloze phrases, finetunes language models, soft-labels unlabeled data, and trains a standard classifier; iPET repeats this with increasing training set sizes.
Results
Across diverse tasks and multiple languages, PET and iPET substantially outperform unsupervised, supervised, and strong semi-supervised approaches with small to medium labeled datasets.
Takeaways & Limitations
When initial training data is limited, PET gives large improvements over standard supervised training and strong semi-supervised approaches.
Takeaways & Limitations
Hyperparameters were chosen from prior work and practical considerations because the few-shot setting provided no large development set, and one underfitting MNLI run was excluded.
Abstract
from arXiv · showhide
Some NLP tasks can be solved in a fully unsupervised fashion by providing a pretrained language model with "task descriptions" in natural language (e.g., Radford et al., 2019). While this approach underperforms its supervised counterpart, we show in this work that the two ideas can be combined: We introduce Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task. These phrases are then used to assign soft labels to a large set of unlabeled examples. Finally, standard supervised training is performed on the resulting training set. For several tasks and languages, PET outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.
1 Introduction
Few-shot NLP is difficult because small labeled datasets often do not reveal the underlying task, while natural-language task descriptions can make task inference easier. PET combines task descriptions with supervised learning and substantially improves low-resource performance.
- Small labeled datasets often make standard supervised learning perform poorly because many tasks are difficult to grasp from only a few examples.
- Task descriptions make few-shot inference easier by clarifying what the labels represent.The paper illustrates this with examples whose labels are ambiguous until the task is identified as detecting statements about prices.
- Pretrained language models make it feasible to provide task descriptions in natural language, an idea previously used mostly for zero-shot settings.
- PET reformulates examples as cloze-style phrases, finetunes one pretrained language model per pattern, labels unlabeled data softly, and trains a classifier on that dataset.iPET repeats the process with increasing training set sizes.
- On diverse tasks and multiple languages, PET and iPET substantially outperform unsupervised, supervised, and strong semi-supervised approaches with small to medium labeled datasets.
2 Related Work
Prior work uses task descriptions, cloze prompts, related-task examples, and data augmentation for zero- or few-shot NLP. PET differs by combining cloze-based task descriptions with semi-supervised training under limited labeled data.
- Natural-language patterns have been used to provide hints for zero-shot learning and to support unsupervised classification, knowledge mining, and relation classification.
- Cloze-style phrases have also been used to probe pretrained language models' factual, commonsense, linguistic, lexical, and symbolic capabilities.
- Other few-shot approaches exploit examples from related tasks or augment data, with back-translation commonly requiring large amounts of parallel data.
- iPET resembles self-training and bootstrapping methods previously applied across tasks including word-sense disambiguation, relation extraction, parsing, translation, and sequence generation.
3 Pattern-Exploiting Training
PET converts task inputs into cloze questions through pattern-verbalizer pairs, uses finetuned language models to produce soft labels, and trains a classifier on the resulting dataset. iPET reduces dependence on any single pattern by iteratively training new model generations on enlarged datasets.
- Pattern-verbalizer pairs: A pattern maps an input to a phrase containing exactly one mask token, while a verbalizer maps each task label to a vocabulary word.Together they form a pattern-verbalizer pair (PVP).
- Pattern-verbalizer pairs: A PVP converts classification into predicting which verbalizer word is most likely at the masked position.
- PET training: PET finetunes a separate language model for each PVP on the small labeled training set.Because the initial dataset is small, finetuning remains inexpensive even for many PVPs.
- Soft-label construction: The finetuned models combine class scores from multiple PVPs, convert them into a soft probability distribution, and collect the results as a soft-labeled training set.PVPs may be weighted uniformly or by their pretraining accuracy on the training set.
- Classifier training: A standard classifier is finetuned on the soft-labeled dataset to serve as the task classifier.
- Iterative PET (iPET): Because weaker patterns can introduce many mislabeled examples, iPET trains successive PET generations on increasingly larger datasets labeled by random subsets of earlier models.
2. Using this subset, we create a labeled dataset
iPET selects examples for enlarged training sets from labels produced by previous-generation models, preferring examples on which the ensemble is confident.
- Examples are sampled separately for each label from the previous ensemble's labeled pool to enlarge each model-specific training set.The method prefers examples receiving high-confidence predictions because they are typically more likely to be classified correctly.
3. We define T j
This section describes how iPET iteratively trains generations of models and uses later generations to construct training data for a classifier, including a zero-shot adaptation.
- After k generations of PET models, M_k creates T_C, which is used to train classifier C as in basic PET.
- For zero-shot iPET, the initial model set is untrained and M_1 is trained on 10 examples evenly distributed across all labels.
- When label-specific training examples are insufficient, zero-shot iPET samples examples with the highest score for each label, even when that label is not the top prediction.
4 Experiments
The experiments evaluate PET and iPET across English and multilingual tasks, training sizes, pattern variants, and semi-supervised baselines. Results show strong gains in low-resource settings, with benefits generally decreasing as labeled data increases.
- Experimental setup: PET is evaluated on four English datasets and multilingual x-stance using RoBERTa or XLM-R, with multiple training-set sizes and repeated runs.The English tasks are Yelp Reviews, AG’s News, Yahoo Questions, and MNLI; x-stance covers German, French, and Italian.
- English results: Zero-shot iPET outperforms unsupervised baselines across all English datasets and exceeds supervised training with 1000 examples on AG’s News.With 10 examples, standard supervised learning does not outperform chance, whereas PET performs substantially better than fully unsupervised baselines.
- English results: PET considerably outperforms standard supervised training with 50 and 100 examples, while iPET provides further consistent improvements.As training-set size increases, PET and iPET’s performance gains become smaller; with 1000 examples, PET still improves accuracy on all tasks except AG’s News.
- Comparison with SotA: PET substantially outperforms UDA and MixText across all compared tasks despite those methods’ reliance on backtranslation and large labeled resources.The comparison uses RoBERTa base, and the reported PET and iPET results outperform both semi-supervised methods.
- X-Stance: PET brings huge improvements across German, French, and Italian x-stance, including considerable gains in zero-shot cross-lingual performance.Results use macro-averaged F1 for labels 0 and 1; Italian performance averages zero-shot predictions from German and French models.
5 Analysis
The analysis shows that PET benefits from combining patterns, auxiliary language modeling, iterative data expansion, and in-domain unlabeled data, while gradual expansion remains important.
- Combining PVPs: PET compensates for large performance differences among patterns and improves accuracy over the best individual pattern across tasks.Distillation also consistently improves over the undistilled ensemble while reducing the final classifier’s size.
- Auxiliary Language Modeling: Auxiliary language modeling is extremely valuable with 10 examples but becomes less important, and sometimes harmful, as training data increases.Only MNLI receives consistent benefits across training-set sizes.
- Iterative PET: Each additional iPET iteration further improves ensemble performance in the evaluated zero-shot setting.The authors did not test whether more iterations would continue improving performance.
- Iterative PET: Skipping generations 2 and 3 and expanding the training set aggressively produces clearly worse performance on AG’s News and Yahoo.The authors surmise that early annotation of too many examples creates many mislabeled training examples.
- In-Domain Pretraining: In-domain pretraining improves supervised learning but does not eliminate PET’s advantage on Yelp.It also helps PET, suggesting PET uses unlabeled data differently from standard masked language-model pretraining.
6 Conclusion
The conclusion presents PET as a way to combine natural-language task descriptions with supervised training in low-resource settings. It reports large improvements over standard supervised and strong semi-supervised approaches when labeled data is limited.
- Conclusion: PET combines cloze-question patterns and verbalizers with pretrained language models to create large soft-labeled datasets for classifier training.Models are fine-tuned for all pattern-verbalizer pairs before annotating unlabeled data.
- Conclusion: With limited initial training data, PET gives large improvements over standard supervised training and strong semi-supervised approaches.The conclusion states this outcome without restricting it to a single task or language.
A Implementation
The implementation of PET and iPET uses the Transformers library and PyTorch.
- Implementation: PET and iPET are implemented with the Transformers library and PyTorch.The implementation is based on Wolf et al. (2020) and Paszke et al. (2017).
B Training Details
The paper reports hardware, optimization, ensemble, dataset-growth, and comparison settings used to train and evaluate PET and iPET.
- Hardware: Except for in-domain pretraining, experiments use a single NVIDIA GeForce GTX 1080 Ti GPU with 11GB RAM.The in-domain pretraining experiment uses separate hardware described elsewhere in the implementation details.
- Training Setup: Batch size and sequence length are chosen to fit 11GB of GPU memory, with four-step gradient accumulation yielding an effective batch size of 16.The setup accounts for the larger batch sizes used in prior RoBERTa and BERT work.
- Optimization: The supervised learning rate is 1e−5 because 5e−5 often causes unstable training without improving training-set accuracy.Experiments with other learning rates are reported in Appendix D.
- Optimization: Supervised training uses 250 steps, while individual PET models use four times as many steps because each batch combines one labeled and three unlabeled examples.The supervised schedule corresponds to four epochs with 1000 examples.
- Optimization: The auxiliary language-modeling weight α is selected from {1e−3, 1e−4, 1e−5} using Yelp development accuracy and then reused across tasks and training sizes.The selection uses both 90/10 and 50/50 train-development splits.
- Ensembles: Each ensemble contains 3n models for n pattern-verbalizer pairs because three models are trained per pattern.Preliminary experiments found little impact on final performance from this choice.
- iPET: iPET quintuples the number of training examples after each iteration, using d = 5, while each next-generation training set uses 25% of current-generation models, λ = 0.25.These settings seek few generations, diverse training sets, and limited influence from any single model.
- Resources: PET and iPET add no learnable parameters, so their parameter counts match the underlying language model: 355M for RoBERTa large and 270M for XLM-R base.Training a single PET classifier takes approximately 30 minutes for 250 steps on one GPU.
C Dataset Details
The experiments construct balanced few-shot and unlabeled sets, apply standardized preprocessing and task-specific evaluation metrics, and examine PET’s sensitivity to learning rate and training steps on Yelp.
- Dataset construction: Each few-shot training set contains the first t/|L| examples per label, while the unlabeled set contains 10 000 examples per label with labels removed.
- Preprocessing: Examples are shortened to a maximum of 256 tokens using longest-first truncation, with PET sequences truncated before patterns are applied.
- Evaluation: Accuracy is reported for Yelp, AG’s News, Yahoo, and MNLI, while x-stance uses macro-average F1.
- Hyperparameter analysis: The hyperparameter analysis varies learning rates of 1e−5, 2e−5, and 5e−5 and training steps of 50, 100, 250, 500, and 1000.The analysis is performed on Yelp with 100 examples because the full grid yields 30 configurations for one task and training-set size.
- Hyperparameter analysis: For supervised learning on Yelp with 100 examples, the paper’s standard configuration of LR = 1e−5 and 250 steps performs best, whereas PET consistently performs better with fewer steps.
E Automatic Verbalizer Search
Automatic Verbalizer Search (AVS) removes the need to manually assign label words by iteratively evaluating candidate tokens from randomized initial verbalizers. PET with AVS remains stronger than regular supervised training, although handcrafted verbalizers perform better and AVS struggles on some tasks.
- Purpose: AVS automatically finds suitable single-token verbalizers for labels from a training set and language model.
- Candidate scoring: A candidate token is scored for a label by testing whether the model assigns it high probability mainly to training examples carrying that label.
- Iterative search: Because scoring one label can require verbalizers for the others, AVS begins with random assignments and repeatedly recomputes the best verbalization for each label.
- Iterative search: AVS samples multiple verbalizer assignments and updates a probability distribution over candidate tokens, with scores jointly considering all patterns.
- Results: The search uses k = 250, ε = 10−3, i_max = 5, and m = 10, while retaining only frequent alphabetic tokens; AVS struggles on some Yahoo and all MNLI labels.The search space is restricted to the 10 000 most frequent eligible tokens in the unlabeled dataset.
- Results: Carefully handcrafted verbalizers perform much better than AVS, but PET with AVS still considerably outperforms regular supervised training.