Source-linked AI summary
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification
Timo Schick, Helmut Schmid, Hinrich Schütze
TL;DR
Few-shot classification methods based on cloze questions require difficult manual mappings between labels and language-model words. PETAL automatically searches for these verbalizers from limited training data, and its mappings nearly match hand-crafted ones across several tasks while outperforming regular supervised training. The approach uses candidate filtering and multi-verbalizers, but PET includes additional details not covered here and its verbalization-count hyperparameter can be difficult to optimize on small datasets.
Problem
Manually defining PET’s label-to-word mapping requires task-specific knowledge and understanding of which single-token words the language model handles well.
Method
PETAL automatically searches for label verbalizers using candidate sets and supports multiple verbalizations per label.
Results
PETAL substantially outperforms regular supervised training and nearly matches PET with manually defined mappings, especially on some tasks.
Takeaways & Limitations
PETAL identifies suitable label-representing words with as little as 50 examples, reducing reliance on task-specific knowledge for verbalizer selection.
Takeaways & Limitations
PETAL adds the number of verbalizations per label as a hyperparameter that may be difficult to optimize for small training sets.
Abstract
from arXiv · showhide
A recent approach for few-shot text classification is to convert textual inputs to cloze questions that contain some form of task description, process them with a pretrained language model and map the predicted words to labels. Manually defining this mapping between words and labels requires both domain expertise and an understanding of the language model's abilities. To mitigate this issue, we devise an approach that automatically finds such a mapping given small amounts of training data. For a number of tasks, the mapping found by our approach performs almost as well as hand-crafted label-to-word mappings.
1 Introduction
Few-shot text classification remains difficult despite pretrained language models, motivating task reformulation through cloze questions. PET uses this strategy, but its label-to-word mapping is difficult to define manually, so PETAL learns it automatically and nearly matches hand-crafted mappings.
- 1 Introduction: Few-shot learning remains challenging even as pretrained language models improve many NLP tasks.The challenge matters because small datasets are common in real-world NLP applications.
- 1 Introduction: PET reformulates classification inputs as cloze questions that pretrained language models can process directly.
- 1 Introduction: PETAL automatically identifies label-representing words from small amounts of training data, removing the need for expert knowledge.Its mapping approach is designed to replace manually defined label-to-word mappings.
- 1 Introduction: PETAL substantially outperforms regular supervised training and nearly matches PET with a manually defined mapping.
2 Related Work
Prior work applies language-model reformulation across unsupervised, few-shot, and high-resource settings, while PET and AVS address label-word mappings. PETAL differs from AVS by replacing iterative greedy search with a faster closed-form solution that performs better on difficult tasks.
- 2 Related Work: Language-model reformulation has been studied in unsupervised, few-shot, high-resource, and probing settings.
- 2 Related Work: PETAL extends PET and is related to AVS, which automatically searches for mappings from labels to words.
- 2 Related Work: AVS initializes random label-word assignments and iteratively replaces words with better alternatives using greedy updates.
- 2 Related Work: PETAL instead uses a closed-form solution that is conceptually simpler, faster, and less hyperparameter-dependent than AVS.The passage reports that PETAL performs much better, especially on difficult tasks.
- 2 Related Work: PET focuses on automating the label-to-word mapping because that mapping carries much of the expert knowledge required by PET.
3 Pattern-Exploiting Training
PET converts inputs into single-mask cloze questions and uses verbalizers to connect masked-token predictions with task labels. It then fine-tunes pattern-specific models, generates soft labels for unlabeled data, and trains a final classifier, while omitting some auxiliary details here.
- 3 Pattern-Exploiting Training: Each PET pattern maps an input to a token sequence containing exactly one [MASK] token, creating a cloze question.
- 3 Pattern-Exploiting Training: A verbalizer maps each label to one vocabulary token whose suitability as the masked replacement should correspond to label correctness.
- 3 Pattern-Exploiting Training: PET derives each label probability from the masked language model’s probability for that label’s verbalization.
- 3 Pattern-Exploiting Training: PET fine-tunes a separate masked language model for each pattern-verbalizer pair using cross-entropy on the training labels.
- 3 Pattern-Exploiting Training: An ensemble of the fine-tuned models assigns soft labels to unlabeled examples, which train a final sequence-classification model.
- 3 Pattern-Exploiting Training: The presentation omits additional PET details, including an auxiliary language-modeling objective intended to prevent catastrophic forgetting.
4 Likelihood Ratio Verbalizer Search
PETAL automatically searches for label words using small training sets by decomposing verbalizer selection into tractable subproblems. It uses likelihood-ratio objectives, filtered candidate sets, and multi-verbalizers to address imbalance, token-likelihood bias, and labels with multiple aspects.
- Likelihood Ratio Verbalizer Search: PETAL replaces intractable joint verbalizer search with independent one-vs-rest searches for each label.The method creates a binarized training set for each label, distinguishing that label from all others.
- Likelihood Ratio Verbalizer Search: Label imbalance from one-vs-rest reformulation is compensated by reweighting examples according to each label’s frequency.Each binarized dataset contains k − 1 times as many negative as positive examples when the original dataset is balanced.
- Likelihood Ratio Verbalizer Search: Likelihood-ratio training avoids cross-entropy’s tendency to select globally likely tokens by comparing a candidate’s score with scores for other verbalizations.Cross entropy can make negative examples contribute almost nothing because the probability assigned to the negative class is close to 1.
- Empirical Analysis: On Yahoo Questions, LCE selects generally likely substitutes, whereas LLR finds reasonable verbalizers but unrestricted search can produce uncommon tokens.The filtered candidate sets are introduced to address the uncommon-token issue while preserving plausible verbalizations.
- Verbalization Candidates: Candidate sets are reduced to 10,000 frequent real-word tokens and then to 1,000 tokens maximizing positive-example likelihood for each label.The filtering favors tokens that are syntactically and semantically plausible for the pattern, while also introducing a desirable frequency bias.
- Multi-Verbalizers: Multi-verbalizers assign multiple tokens to one label and use their average model score, accommodating labels whose meanings span different aspects.The paper gives “Science” and “Tech” as verbalizations for the AG’s News “Science/Tech” category.
5 Experiments
Experiments evaluate PETAL’s automatically found verbalizers qualitatively and quantitatively across four few-shot classification tasks. With 50 labeled examples, PETAL generally outperforms alternative automatic verbalizer methods and nearly matches manually defined mappings on some tasks.
- Qualitative analysis: The qualitative analysis evaluates automatically identified verbalizers on Yahoo Questions and MNLI using only 50 labeled examples.Yahoo Questions has ten categories with five examples per label; MNLI tests contradiction, entailment, and neutral labels.
- Qualitative analysis: Restrained candidate sets mitigate uncommon verbalizations, while multi-verbalizers combine terms such as “hardware” and “software” to cover broader label meanings.The candidate vocabulary is filtered to frequent real-word tokens, and multi-verbalizers average scores across several verbalizations.
- Quantitative results: Random verbalizers slightly outperform supervised learning, but AVS performs much better than random selection and PETAL outperforms AVS on 3 of 4 tasks.The advantage is especially large on MNLI, and the separate PETAL variant performs slightly better on average than the joint variant.
- Quantitative results: PETAL almost matches manually defined mappings on MNLI while requiring no task-specific knowledge to find verbalizers.The comparison uses manually defined mappings as an upper bound for incorporating task- and model-specific knowledge.
- Hyperparameter analysis: Results remain relatively stable for nv values from 1 to 100, although the best result across all tasks occurs at nv = 3.The number of verbalizations per label is PETAL’s single additional hyperparameter and may be difficult to optimize with small training sets.
6 Conclusion
The paper concludes that PETAL automatically maps labels to words for PET using as few as 50 examples. Its analyses show that the resulting verbalizers are suitable and can nearly match hand-crafted mappings on some tasks.
- Conclusion: PETAL enriches PET with automatic label-to-word mapping using as few as 50 examples.The approach is presented as simple and is supported by both qualitative and quantitative analyses.
- Conclusion: PETAL almost matches the performance of hand-crafted mappings for some tasks without requiring task-specific knowledge to find verbalizers.The paper identifies automatic pattern acquisition as a direction for future work.
A Relation of Maximum Likelihood Estimate and One-Vs-Rest Likelihood Ratio
The appendix relates the maximum-likelihood verbalizer objective to the one-vs-rest likelihood-ratio objective. Their key distinction is whether each label’s score is compared with the maximum or the average score of competing verbalizations.
- Objective reformulation: The analysis studies reframing k-class classification as k one-vs-rest problems, downsampling negative examples, and replacing LCE with LLR.These modifications are analyzed as steps from the intractable maximum-likelihood estimate toward the likelihood-ratio objective.
- Approximation: The derivation uses approximations involving q(P,t)(0) and the tangent-line approximation log(a + b) ≈ log a + b/a.The resulting expression interprets part of each addend as a soft approximation of LogSumExp.
- Likelihood-ratio objective: Under a balanced dataset assumption, each verbalization’s likelihood-ratio loss is independent of verbalizations for other labels.This permits writing the optimization criterion as the sum of likelihood-ratio losses across verbalizations.
- Likelihood-ratio objective: Each training example contributes k times to the one-vs-rest formulation: once positively and k − 1 times negatively.The negative contributions correspond to every label other than the example’s true label.
- Interpretation: Eq. 2 separates the correct verbalization from the maximum competing score, whereas Eq. 7 separates it from the average competing score.This is the appendix’s final verification of the relationship between maximum likelihood and one-vs-rest likelihood ratio.