Source-linked AI summary
Improving and Simplifying Pattern Exploiting Training
Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, Colin Raffel
TL;DR
Few-shot fine-tuning can struggle with limited labeled data, while PET requires task-specific unlabeled data. ADAPET modifies PET to provide denser supervision without unlabeled data and outperforms competing approaches on SuperGLUE.
Problem
Few-shot fine-tuning performance can suffer with very limited labeled data, and PET assumes access to task-specific unlabeled data.
Method
ADAPET modifies PET by decoupling label-token losses and adding label-conditioned masked language modeling over the full original input.
Results
ADAPET outperforms models without unlabeled data, including iPET, and achieves state-of-the-art few-shot performance on SuperGLUE; it uses 0.1% as many parameters as GPT-3.
Takeaways & Limitations
ADAPET shows that denser supervision can support few-shot SuperGLUE learning without task-specific unlabeled data.
Takeaways & Limitations
The setup does not assume task-specific unlabeled data but still assumes access to the full development set.
Abstract
from arXiv · showhide
Recently, pre-trained language models (LMs) have achieved strong performance when fine-tuned on difficult benchmarks like SuperGLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning. However, PET uses task-specific unlabeled data. In this paper, we focus on few-shot learning without any unlabeled data and introduce ADAPET, which modifies PET's objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data. Our code can be found at https://github.com/rrmenon10/ADAPET.
1 Introduction
Few-shot fine-tuning can struggle with limited labeled data, while PET improves performance but relies on task-specific unlabeled examples. ADAPET modifies PET to use more supervision and outperforms iPET on SuperGLUE using only labeled data.
- Limited labeled data can reduce pretrained language-model performance on downstream tasks.
- GPT-3 learns from a few labeled examples, but its 175B-parameter scale makes it impractical to study.
- PET reformulates language-understanding tasks as cloze-style questions and outperforms GPT-3 with few labeled examples using ALBERT.
- PET uses additional task-specific unlabeled data, whereas ADAPET uses 32 labeled examples per task and performs better than iPET.
2 Background
The background defines cloze-style masked-language-modeling inputs and PET’s pattern-verbalizer framework. PET uses task-specific unlabeled data, which can be costly to obtain for benchmarks such as SuperGLUE.
- A cloze task removes text portions and predicts the missing content from surrounding context, motivating masked language modeling.
- Task-specific unlabeled data may be expensive for SuperGLUE because some examples require filtering suitable text pairs before labeling.
- PET: PET converts each example into a cloze-style question using a pattern and maps task classes to output tokens with a verbalizer.
- PET: For entailment, PET can map “Not Entailment/Entailment” to the output tokens “No/Yes” and compute a softmax over those label tokens.
- PET: PET distills knowledge from models trained with different patterns, while iPET iteratively expands training data using labels from previous iterations.
3 ADAPET
ADAPET modifies PET with denser supervision so few-shot learning can use labeled examples without task-specific unlabeled data. Its objective decouples label-token losses and adds label-conditioned masked language modeling over the input.
- ADAPET modifies PET to provide more supervision and learn without task-specific unlabeled data.
- 3.1 Decoupling Label Losses: PET discards gradient information from vocabulary tokens that are not task labels, whereas ADAPET trains against the full vocabulary while specifically suppressing incorrect class tokens.
- 3.1 Decoupling Label Losses: ADAPET encourages correct class tokens and penalizes incorrect class tokens using a binary cross-entropy formulation.
- 3.1.1 Unified Loss for Different Tasks: For multi-token labels, ADAPET splits one loss into separate token-level losses, avoiding the probability-normalization issues encountered by PET.
- 3.2 Label Conditioning: Label conditioning trains the model to predict masked input tokens given a label, maximizing the correct-label probability and minimizing probabilities for incorrect labels.
- 3.2 Label Conditioning: The final ADAPET objective sums the decoupled label loss and the label-conditioned masked-language-modeling loss.
4 Results and Analyses
Experiments evaluate ADAPET on SuperGLUE under few-shot conditions, comparing it with GPT-3, PET, iPET, and sPET while examining objective components. ADAPET performs strongly without task-specific unlabeled data, and label conditioning is especially beneficial for some tasks.
- Experimental setup: Experiments use the SuperGLUE split with 32 labeled examples per task and select checkpoints using the development set.The setup follows PET’s pretrained model and hyperparameters, with 1k training batches for the main comparison.
- Main results: ADAPET outperforms all models without unlabeled data and exceeds iPET by 0.5 points absolute on the development set.The comparison includes GPT-3, PET/iPET, and sPET under the reported experimental setup.
- Main results: On the test set, ADAPET outperforms all other models, including iPET, without access to approximately 9k unlabeled examples per task.The reported result establishes state-of-the-art few-shot learning performance on SuperGLUE.
- Ablation analysis: Label conditioning is extremely beneficial for ADAPET, especially on CB, while its effects vary across BoolQ, RTE, and MultiRC.Removing label conditioning hurts CB F1 but improves RTE and MultiRC relative to the modified decoupled label objective; conditioning only on the correct label hurts BoolQ but helps CB.
5 Conclusion
ADAPET is presented as a few-shot natural language understanding method that replaces task-specific unlabeled data with denser supervision. The conclusion reports strong SuperGLUE performance while identifying random masking as a limitation and future improvement target.
- Conclusion: ADAPET is a new method for few-shot natural language understanding.The method is designed for settings where labeled examples are scarce.
- Conclusion: ADAPET does not use unlabeled data and instead leverages more supervision to train the model.Its objective is applied to pattern-based inputs and label-conditioned masked language modeling.
- Conclusion: Using the same data budget, ADAPET outperforms GPT-3 on SuperGLUE with 0.1% as many parameters.The conclusion states this comparison at the model level rather than for an individual task.
- Limitations and future work: The paper uses a naive random masking strategy that might not make sense for label conditioning.Future work is suggested on masking important tokens using gradients of the logits.
A.2 Results on Individual Patterns
The individual-pattern analysis compares sPET and ADAPET across SuperGLUE patterns using the same experimental setup. ADAPET outperforms sPET on most evaluated patterns.
- Pattern-level results: ADAPET outperforms sPET on 25 out of 29 patterns across SuperGLUE datasets.Results are reported on the validation dataset after training across all patterns.
B (More) Experiment Details
Additional experiment details specify masking and pattern choices, while Tables 4 and 5 compare sPET and ADAPET across PET’s patterns after different training durations. The tables are intended to show pattern-specific performance rather than only the best-pattern aggregate.
- Training details: The experiments mask out up to 10.5% of tokens for all datasets.COPA uses a different pattern for label conditioning because its standard pattern contains both the correct and incorrect choice.
- Training details: MultiRC and ReCoRD use a maximum text length of 512 tokens.The footnote identifies this as an exception to the general maximum-length setting.
- Pattern comparisons: Table 4 reports sPET and ADAPET validation performance for different patterns after 1000 batches, with Table 1 selecting the best pattern for each model.The same PET patterns are used, and the table marks the best model for each pattern.
- Pattern comparisons: Table 5 reports sPET and ADAPET validation performance for different patterns after 250 batches.It uses the same PET patterns and marks the best model for each pattern.
C.1 Duration of Training
Training duration affects the two pattern-based methods differently: longer training benefits ADAPET, while sPET performs best earlier.
- C.1 Duration of Training: ADAPET improves with longer training, especially on ReCoRD.The comparison evaluates 250 versus 1k batches.
- C.1 Duration of Training: sPET peaks at 250 batches rather than benefiting from the longer schedule.
C.2 Multi-Task Multi-Pattern Training
The ablations examine multi-pattern training, alternative label objectives, masking strategies, and ensembles. Results generally favor the original ADAPET design choices over these alternatives.
- C.2 Multi-Task Multi-Pattern Training: Multi-task multi-pattern training generally hurts ADAPET performance, is mixed for sPET, and helps ADAPET without label conditioning.
- C.3 Replacement Token Detection (RTD): The decoupled label objective outperforms the RTD objective overall, while ELECTRA initialization produces the same results as the tested binary-head setup.The paper attributes RTD’s weaker performance partly to discarding the pretrained softmax classifier.
- C.4 Label Conditioning with Important Words Masked Out: Random masking performs better overall than masking tokens selected as important by TFIDF.
- C.5 Ensembles: Ensembling across patterns and seeds produces mixed results, so the authors do not ensemble models for their final results.
- C.6 Masking Ratio: 10.5% VARIABLE masking provides the best trade-off across models and is selected for the final experiments.VARIABLE masking samples between 1 and the specified maximum number of masked tokens.
C.7 What if we had unlabeled data?
The paper tests whether unlabeled-data pretraining can improve ADAPET. TAPT helps regular fine-tuning but significantly hurts both sPET and ADAPET, leaving pattern-based use of unlabeled data open.
- C.7 What if we had unlabeled data?: TAPT improves regular fine-tuning on three of four datasets but significantly hurts sPET and ADAPET on all datasets.The models are pretrained for 2500 batches on FewGLUE unlabeled data before fine-tuning.
- C.7 What if we had unlabeled data?: The authors speculate that TAPT hurts pattern-based models because the model never sees the pattern during pretraining.
- C.7 What if we had unlabeled data?: Improving pattern-based few-shot methods such as ADAPET when unlabeled data is available remains an open challenge.