Source-linked AI summary
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Robert L. Logan, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, Sebastian Riedel
TL;DR
Few-shot learning depends heavily on prompt design, creating a difficult search problem. The paper evaluates prompt-based finetuning with simpler prompts and lightweight updates, finding that null prompts remain competitive and bias-only finetuning preserves accuracy while updating 0.1% of parameters.
Problem
Prompt design can substantially affect few-shot accuracy, but selecting patterns, examples, and verbalizers requires difficult combinatorial search and hard-to-replicate intuition.
Method
The paper compares prompt-based finetuning and in-context prompt strategies using null prompts, automated prompt tuning, and bias-only finetuning.
Results
Null prompts achieve comparable accuracy to manually written patterns, while BitFit achieves competitive or better accuracy than standard finetuning with 0.1% of parameters updated.
Takeaways & Limitations
Prompt-based finetuning is more accurate and robust across prompts, and BitFit makes it nearly as memory-efficient as using frozen language models.
Takeaways & Limitations
The experiments use relatively small masked language models, so the findings may differ for very large or left-to-right language models.
Abstract
from arXiv · showhide
Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced: finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs.
1 Introduction
Few-shot prompting can require difficult prompt engineering, but prompt-based finetuning makes simpler prompts effective and lightweight bias updates reduce its memory cost.
- Prompt engineering requires searching over pattern wording, training-example inclusion, and verbalizer design, often relying on difficult-to-replicate human intuition.
- Null prompts omit task-specific patterns and training examples while retaining only input concatenation, a [MASK] token, and label names.
- Null prompts achieve comparable accuracy to manually written patterns across many masked-language-model tasks.
- Automated prompt tuning for in-context learning generally fails to match manually engineered prompts, whereas lightweight prompt-based finetuning is more successful.
- Lightweight prompt-based finetuning updates only masked-LM bias terms, achieving competitive or better accuracy than standard finetuning while updating 0.1% of parameters.
2 Prompting Language Models
The paper contrasts frozen-LM in-context learning with prompt-based finetuning, then studies how prompt construction and parameter updates affect accuracy and efficiency.
- 2 Prompting Language Models: Few-shot prompting uses a masked language model, limited labeled examples, a pattern producing a single [MASK] token, and a verbalizer mapping labels to vocabulary tokens.
- 2 Prompting Language Models: The pattern and verbalizer together constitute the prompt, whose construction and model-parameter updates define the methods studied.
- 2.1 Constructing the Prompt: Prompt accuracy can range from near chance to near state of the art, motivating searches over patterns, verbalizers, and training-example placement.
- 2.1 Constructing the Prompt: Prompt-based finetuning reduces prompt importance because the model is finetuned, unlike settings where the model remains frozen.
- 2.2 Finetuning the LM: In-context learning keeps the language model frozen and conditions predictions on the prompt, working best with very large models.
- 2.2 Finetuning the LM: Prompt-based finetuning constructs masked examples and trains the language model to generate the correct verbalizer token at the masked position.
- 2.2 Finetuning the LM: Prompt-based finetuning usually improves accuracy, especially for relatively small models, but prevents reusing one model across tasks and reduces memory efficiency.
3 Experimental Setup
The experiments evaluate few-shot methods on GLUE and Super-GLUE tasks using cross-validation, repeated sampling, held-out development sets, and a significance-based # Wins comparison.
- The evaluation covers BoolQ, CB, MNLI, MRPC, QNLI, QQP, RTE, and SST-2 from GLUE and Super-GLUE.
- Few-shot datasets sample 2K examples per label and use 4-fold cross-validation for model selection instead of separate K-shot training and development splits.
- Examples are sampled with 10 random seeds, performance is reported as mean and variance, and final evaluation uses each dataset’s original development set.
- WiC and WNLI results are omitted because all evaluated models achieved near-random accuracy.
- The study uses RoBERTa large with 330M parameters and ALBERT xxl-v2 with 223M parameters as masked language models.
- The # Wins metric counts datasets where a method performs significantly better than all other methods, using pairwise Welch’s t-tests and allowing ties.
4 Simplifying Prompt Engineering
Prompt-based finetuning can achieve competitive few-shot performance with null prompts, reducing prompt-design requirements. Null prompts also offer a simpler selection process than prompt tuning while retaining strong performance.
- Prompt Ablations: The study compares six prompt types, including prior manual prompts, manually written prompts, prompt tuning, null prompts, null verbalizers, and their combination.All variants are evaluated with prompt-based finetuning, alongside traditional [CLS] finetuning.
- Prompt Comparisons: Prior-work manual prompts perform best on average for both models, but their validation-set tuning procedure is unclear.Manually written prompts without engineering perform noticeably worse and are outperformed by several alternatives.
- Prompt Comparisons: Null prompts and prompt tuning often perform comparably to manually written prompts, especially for RoBERTa, without requiring prompt-pattern tuning.Both methods outperform the manually written prompts in terms of # Wins in the reported comparison.
- Prompt Comparisons: Combining null prompts with null verbalizers significantly outperforms standard [CLS] finetuning on numerous tasks.The improvement is significant for 3 tasks with RoBERTa and 5 with ALBERT at p = 0.05.
- Prompt Selection: Null prompts require choosing only the concatenation order of input fields and the [MASK] token, whereas prompt tuning requires additional embedding choices.The concatenation order can be selected by trying possible options on a validation set.
5 Achieving Simplicity and Efficiency
The paper compares simplifying frozen-model prompting with reducing the parameter cost of prompt-based finetuning. Prompt-only tuning is generally inferior, whereas BitFit offers the strongest accuracy-efficiency tradeoff and combines effectively with null prompts.
- 5.1 Simplifying In-Context Learning With Prompt-Only Tuning: Automated prompt-only methods search discrete tokens or learned prompt embeddings while keeping the masked LM fixed.The evaluated variants include AUTOPROMPT and short or long prompt tuning.
- 5.1 Simplifying In-Context Learning With Prompt-Only Tuning: Prompt-only tuning generally fails to match manually designed prompts and performs substantially worse than prompt-based finetuning.On CB, the gap reaches 40% absolute F1.
- 5.2 Memory-Efficient Finetuning: The study reduces prompt-based finetuning costs using adapters, bias-only updates, LM-head tuning, or calibration.BitFit updates only Transformer bias terms, while the other methods update smaller task-specific components or inserted layers.
- 5.2 Memory-Efficient Finetuning: BitFit provides the best accuracy-efficiency tradeoff and can outperform updating all parameters in the number of dataset wins.The results show diminishing returns, marginal gains, or decreased performance as the parameter count increases beyond BitFit.
- 5.3 Putting Everything Together: Combining null prompts with BitFit yields competitive accuracy, simple setup, and small memory costs for each downstream task.The paper’s final comparison reports prompt-based finetuning with null prompts against representative few-shot methods.
6 Conclusion and Future Work
The conclusion presents null prompts and BitFit as a simple, efficient, and competitive few-shot approach. It also identifies open questions about why prompt-based finetuning works and whether the findings extend beyond the evaluated masked LMs.
- 6 Conclusion and Future Work: Null prompts make prompt-based finetuning robust to prompt choices without degrading performance relative to manually written or learned prompts.They can be applied flexibly across different tasks.
- 6 Conclusion and Future Work: BitFit updates only bias terms, achieving comparable or better accuracy than full finetuning while being 1000x more memory efficient.The approach updates 0.1% of the parameters.
- 6 Conclusion and Future Work: The findings suggest that prompt-based finetuning gains are partly associated with predicting a [MASK] token using a pretrained MLM head.The paper calls for further analysis of how small few-shot design changes affect accuracy.
- 6 Conclusion and Future Work: The experiments use relatively small masked LMs, so extension to very large and left-to-right LMs remains future work.The authors explicitly identify this as a boundary of the current results.