Source-linked AI summary
Few-Shot Text Generation with Pattern-Exploiting Training
Timo Schick, Hinrich Schütze
TL;DR
Few-shot text generation needs instruction designs that pretrained models understand and safeguards against overfitting. The paper introduces GENPET, which combines instructions with labeled examples for generative finetuning, and reports consistent improvements over regular finetuning in few-shot settings. Its evaluation is limited by the lack of comparisons with other few-shot approaches.
Problem
Few-shot text generation requires effective task instructions and overfitting controls, while prior instruction-based gradient training was limited to classification.
Method
GENPET adapts Pattern-Exploiting Training to generative language models by combining textual instructions with labeled examples.
Results
GENPET-trained PEGASUS outperforms regular finetuning across a broad set of generative tasks and training-set sizes, including few-shot settings.
Takeaways & Limitations
GENPET enables data-efficient finetuning of PEGASUS with as little as 10 or 100 training examples.
Takeaways & Limitations
The paper does not compare GENPET with other few-shot approaches because their assumptions or generative applicability differ.
Abstract
from arXiv · showhide
Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion. Moreover, when combined with regular learning from examples, this idea yields impressive few-shot results for a wide range of text classification tasks. It is also a promising direction to improve data efficiency in generative settings, but there are several challenges to using a combination of task descriptions and example-based learning for text generation. In particular, it is crucial to find task descriptions that are easy to understand for the pretrained model and to ensure that it actually makes good use of them; furthermore, effective measures against overfitting have to be implemented. In this paper, we show how these challenges can be tackled: We introduce GenPET, a method for text generation that is based on pattern-exploiting training, a recent approach for combining textual instructions with supervised learning that only works for classification tasks. On several summarization and headline generation datasets, GenPET gives consistent improvements over strong baselines in few-shot settings.
1 Introduction
Few-shot task reformulation with natural-language instructions can make pretrained models more data-efficient, but existing instruction-based gradient training was limited to classification. GENPET extends this idea to generative tasks and outperforms standard finetuning across tasks and training-set sizes.
- Few-shot downstream settings remain difficult because methods that reduce data requirements often perform poorly with only a handful of examples.
- Natural-language task descriptions reformulate downstream inputs to resemble pretraining objectives and help pretrained models understand the task.
- Instruction-based gradient training had shown strong few-shot potential but was limited to text classification before GENPET.
- GENPET finetunes generative language models using both instructions and labeled examples, including with as little as 10 or 100 training examples.
- Across six English headline-generation and summarization tasks, PEGASUS trained with GENPET outperforms regular finetuning in zero-shot and few-shot settings.
- The paper introduces GENPET, demonstrates its broad performance gains, and analyzes the contribution of all its components.
2 Related Work
Related work frames GENPET as an extension of prompt- and cloze-based few-shot learning, while distinguishing it from approaches requiring related-task data, consistency training, or specialized high-resource methods.
- Masked-language-model research commonly reformulates tasks as cloze questions, and PET combines this idea with gradient-based learning for efficient few-shot classification.
- Other few-shot NLP methods may require large related-task example sets, parallel consistency-training data, or task-specific techniques.
- GENPET requires no additional labeled data and offers an interface for incorporating task-specific human knowledge.
- PEGASUS is used as the strong data-efficiency baseline against which GENPET is evaluated.
- Prompt- and keyword-based controllable generation methods use specialized pretraining procedures and have been studied mainly in high-resource settings.
3 PEGASUS Pretraining
PEGASUS is a Transformer encoder-decoder pretrained for summarization with gap-sentence generation. It masks informative sentences and trains the decoder to generate the removed sentences as a pseudo-summary.
- PEGASUS is a Transformer encoder-decoder model pretrained with gap-sentence generation, an objective tailored to text summarization.
- Its preprocessing selects informative sentences, replaces them with mask tokens, and concatenates the removed sentences into a pseudo-summary.
- The model encodes the partially masked document and generates the removed sentences as the training target.
- The experiments use PEGASUS-large because it is the publicly available variant not finetuned on a downstream task.
4 Pattern-Exploiting Training
Pattern-Exploiting Training makes classification data-efficient by converting inputs into cloze questions and mapping labels to verbalizers whose masked-token probabilities define label probabilities.
- By converting inputs into cloze questions, PET reduces the number of examples required for text classification.
- PET converts each classification input into a cloze question containing exactly one mask token.
- A verbalizer maps each label to a single vocabulary token representing that label’s meaning in the pattern.
- PET derives each label probability from the pretrained masked language model’s probability for its verbalizer at the masked position.
- Finetuning minimizes cross-entropy between the derived label distribution and the true label.
5 Generation with Instructions
GENPET adapts pattern-exploiting training to few-shot text generation by combining natural-language instructions with labeled examples. It addresses instruction placement, instruction variability, and overfitting through decoder prefixes, instruction combination, and regularization-oriented training procedures.
- 5 Generation with Instructions: GENPET represents each input-output task with a pattern P that modifies the input and produces a sequence containing one mask token, without requiring a verbalizer.Unlike classification PET, generated outputs are natural-language sequences rather than labels mapped to individual vocabulary tokens.
- 5 Generation with Instructions: The method targets three challenges: making encoder-decoder models use instructions effectively, ensuring they understand instruction variants, and preventing few-shot overfitting.These challenges concern instruction placement, sensitivity to pattern wording, and memorization from very small training sets.
- 5 Generation with Instructions: The model assigns pM(y | z) to a candidate output as a plausible replacement for the mask, decomposing output probability autoregressively across encoder-processed input and decoder-processed prefixes.Known output prefixes can already be processed by the decoder when scoring the remaining sequence.
- 5.1 Using a Single Instruction: Decoder-processed instruction tokens influence predictions more strongly than encoder-processed tokens, so GENPET supplies each pattern with a decoder prefix.For PEGASUS, encoder-only partial prompts may be ignored in favor of starting an unrelated new sentence.
- 5.2 Combining Instructions: GENPET combines multiple pattern-and-prefix pairs because a small development set may not reliably distinguish helpful instructions from poor ones.The combination mechanism uses candidate generations and likelihood-based scores to train one model from unlabeled examples, avoiding multiple models at inference.
- 5.2 Combining Instructions: The combined model is trained with a mask-prepending pattern to reduce the mismatch between masked-language-model pretraining and fine-tuning.Prepending a mask also biases generation toward text likely to precede the input, such as a headline or summary.
- 5.3 Preventing Overfitting: GENPET addresses few-shot memorization through unsupervised scoring and joint training, with joint training forcing one model to work across instructions as a proposed regularizer.Unsupervised scoring removes candidate outputs ranked below a threshold by the untrained model; joint training also reduces training and inference overhead.
6 Experiments
GENPET is evaluated on six headline-generation and summarization tasks using PEGASUS, few-shot training sets, and multiple baselines. Across few-shot settings, GENPET consistently outperforms regular finetuning, while analyses examine instruction combinations, scoring, and qualitative outputs.
- Results: GENPET consistently outperforms regular PEGASUS finetuning across all tasks in few-shot settings.The comparison includes regular PEGASUS, PEGASUS-M with a trivial pattern, and GENPET with multiple patterns and decoder prefixes.
- Results: 7.20 R1 points separate GENPET from PEGASUS with 10 examples, while the gap is 2.58 points with 100 examples.The reported averages are 31.63 versus 24.43 for 10 examples and 34.45 versus 31.87 for 100 examples.
- Results: GENPET also outperforms all baselines on average in zero-shot learning, although it falls short on individual tasks.PEGASUS-M improves over regular finetuning, but remains clearly worse than GENPET.
- Analysis: Combining instructions and applying unsupervised scoring with joint training improves performance relative to corresponding variants.Removing joint training hurts all three evaluated tasks, while supervised scoring hurts two of three tasks.
- Qualitative analysis: Qualitatively, GENPET produces fluent, factual headlines, unlike regular PEGASUS copying input text and PEGASUS-M hallucinating unsupported information.The qualitative comparison is illustrated on a selected Gigaword input.
7 Conclusion
The paper studies whether pretrained language models can use simple instructions to make text generation more data-efficient. GENPET addresses instruction use, instruction usefulness, and overfitting through decoder prefixes, instruction distillation, unsupervised scoring, and joint training, and improves few-shot finetuning with PEGASUS.
- The conclusion identifies three challenges: making language models use instructions, ensuring instructions are useful, and preventing overfitting.
- GENPET addresses these challenges with decoder prefixes, knowledge-distillation-based instruction combination, unsupervised scoring, and joint training.
- A PEGASUS model finetuned with GENPET clearly outperforms regular finetuning in few-shot settings.
A Analysis
The analysis tests whether GENPET’s instruction effects persist for longer outputs and whether unsupervised scoring reduces overfitting to training examples. It also considers variance across training sets.
- Sequence Length: Increasing the maximum output length from 32 to 128 reduces GENPET’s average gain over PEGASUS by only 0.10 R1 points.This indicates that decoder-prefix instructions continue influencing generated tokens across longer sequences.
- Sequence Length: GENPET remains beneficial for generating long text sequences, despite the potential for decoder-prefix influence to diminish with distance.
- Unsupervised Scoring: Unsupervised scoring downweights training examples that appear overly likely under the trained model but less likely under an untrained model.The method is motivated by PEGASUS overfitting to its small training set.
- Unsupervised Scoring: 92% versus 24% illustrates how unsupervised scoring changes an example’s relative likelihood under trained and untrained models.The first three examples are removed from the final training set when their unsupervised rank falls below τ = 0.2.
- Variance: Performance variability is assessed using standard deviations of R1, R2, and RL across three training sets for each task.