PaperlayerView exact revision on arXiv

Source-linked AI summary

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

Robert L. Logan, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, Sebastian Riedel

arXiv:2106.13353v2cs.CLcs.LG

TL;DR

Few-shot learning depends heavily on prompt design, creating a difficult search problem. The paper evaluates prompt-based finetuning with simpler prompts and lightweight updates, finding that null prompts remain competitive and bias-only finetuning preserves accuracy while updating 0.1% of parameters.

  • Problem

    Prompt design can substantially affect few-shot accuracy, but selecting patterns, examples, and verbalizers requires difficult combinatorial search and hard-to-replicate intuition.

    Source: §1 · p. 1

  • Method

    The paper compares prompt-based finetuning and in-context prompt strategies using null prompts, automated prompt tuning, and bias-only finetuning.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Null prompts achieve comparable accuracy to manually written patterns, while BitFit achieves competitive or better accuracy than standard finetuning with 0.1% of parameters updated.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Prompt-based finetuning is more accurate and robust across prompts, and BitFit makes it nearly as memory-efficient as using frozen language models.

    Source: §1 · p. 2, §6 · p. 8

    The experiments use relatively small masked language models, so the findings may differ for very large or left-to-right language models.

    Source: §5.1 · p. 7, §6 · p. 8

Abstract

from arXiv · show

Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced: finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs.

1 Introduction

Few-shot prompting can require difficult prompt engineering, but prompt-based finetuning makes simpler prompts effective and lightweight bias updates reduce its memory cost.

  • Prompt engineering requires searching over pattern wording, training-example inclusion, and verbalizer design, often relying on difficult-to-replicate human intuition.Source: §1 · p. 1
  • Null prompts omit task-specific patterns and training examples while retaining only input concatenation, a [MASK] token, and label names.Source: §1 · p. 1, Figure 1 · p. 2
  • Null prompts achieve comparable accuracy to manually written patterns across many masked-language-model tasks.Source: §1 · p. 1
  • Automated prompt tuning for in-context learning generally fails to match manually engineered prompts, whereas lightweight prompt-based finetuning is more successful.Source: §1 · pp. 1–2
  • Lightweight prompt-based finetuning updates only masked-LM bias terms, achieving competitive or better accuracy than standard finetuning while updating 0.1% of parameters.Source: §1 · p. 2, Figure 1 · p. 2

2 Prompting Language Models

The paper contrasts frozen-LM in-context learning with prompt-based finetuning, then studies how prompt construction and parameter updates affect accuracy and efficiency.

2 Prompting Language Models

  • Few-shot prompting uses a masked language model, limited labeled examples, a pattern producing a single [MASK] token, and a verbalizer mapping labels to vocabulary tokens.Source: §2 · p. 2
  • The pattern and verbalizer together constitute the prompt, whose construction and model-parameter updates define the methods studied.Source: §2 · p. 2

2.1 Constructing the Prompt

  • Prompt accuracy can range from near chance to near state of the art, motivating searches over patterns, verbalizers, and training-example placement.Source: §2.1 · p. 2
  • Prompt-based finetuning reduces prompt importance because the model is finetuned, unlike settings where the model remains frozen.Source: §2.1 · p. 2

2.2 Finetuning the LM

  • In-context learning keeps the language model frozen and conditions predictions on the prompt, working best with very large models.Source: §2.2 · pp. 2–3
  • Prompt-based finetuning constructs masked examples and trains the language model to generate the correct verbalizer token at the masked position.Source: §2.2 · p. 3
  • Prompt-based finetuning usually improves accuracy, especially for relatively small models, but prevents reusing one model across tasks and reduces memory efficiency.Source: §2.2 · p. 3

3 Experimental Setup

The experiments evaluate few-shot methods on GLUE and Super-GLUE tasks using cross-validation, repeated sampling, held-out development sets, and a significance-based # Wins comparison.

  • The evaluation covers BoolQ, CB, MNLI, MRPC, QNLI, QQP, RTE, and SST-2 from GLUE and Super-GLUE.Source: §3.1 · p. 3
  • Few-shot datasets sample 2K examples per label and use 4-fold cross-validation for model selection instead of separate K-shot training and development splits.Source: §3.1 · p. 3
  • Examples are sampled with 10 random seeds, performance is reported as mean and variance, and final evaluation uses each dataset’s original development set.Source: §3.1 · p. 3
  • WiC and WNLI results are omitted because all evaluated models achieved near-random accuracy.Source: §3.1 · p. 3
  • The study uses RoBERTa large with 330M parameters and ALBERT xxl-v2 with 223M parameters as masked language models.Source: §3.2 · p. 4
  • The # Wins metric counts datasets where a method performs significantly better than all other methods, using pairwise Welch’s t-tests and allowing ties.Source: §3.3 · p. 4, Figure 2 · p. 4

4 Simplifying Prompt Engineering

Prompt-based finetuning can achieve competitive few-shot performance with null prompts, reducing prompt-design requirements. Null prompts also offer a simpler selection process than prompt tuning while retaining strong performance.

Prompt Ablations

  • The study compares six prompt types, including prior manual prompts, manually written prompts, prompt tuning, null prompts, null verbalizers, and their combination.All variants are evaluated with prompt-based finetuning, alongside traditional [CLS] finetuning.Source: §4 · p. 4

Prompt Comparisons

  • Prior-work manual prompts perform best on average for both models, but their validation-set tuning procedure is unclear.Manually written prompts without engineering perform noticeably worse and are outperformed by several alternatives.Source: §4 · p. 4, Figure 3 · p. 5
  • Null prompts and prompt tuning often perform comparably to manually written prompts, especially for RoBERTa, without requiring prompt-pattern tuning.Both methods outperform the manually written prompts in terms of # Wins in the reported comparison.Source: §4 · p. 4, Figure 3 · p. 5
  • Combining null prompts with null verbalizers significantly outperforms standard [CLS] finetuning on numerous tasks.The improvement is significant for 3 tasks with RoBERTa and 5 with ALBERT at p = 0.05.Source: §4 · p. 5

Prompt Selection

  • Null prompts require choosing only the concatenation order of input fields and the [MASK] token, whereas prompt tuning requires additional embedding choices.The concatenation order can be selected by trying possible options on a validation set.Source: §4 · p. 5

5 Achieving Simplicity and Efficiency

The paper compares simplifying frozen-model prompting with reducing the parameter cost of prompt-based finetuning. Prompt-only tuning is generally inferior, whereas BitFit offers the strongest accuracy-efficiency tradeoff and combines effectively with null prompts.

5.1 Simplifying In-Context Learning With Prompt-Only Tuning

  • Automated prompt-only methods search discrete tokens or learned prompt embeddings while keeping the masked LM fixed.The evaluated variants include AUTOPROMPT and short or long prompt tuning.Source: §5.1 · p. 6
  • Prompt-only tuning generally fails to match manually designed prompts and performs substantially worse than prompt-based finetuning.On CB, the gap reaches 40% absolute F1.Source: §5.1 · p. 6

5.2 Memory-Efficient Finetuning

  • The study reduces prompt-based finetuning costs using adapters, bias-only updates, LM-head tuning, or calibration.BitFit updates only Transformer bias terms, while the other methods update smaller task-specific components or inserted layers.Source: §5.2 · p. 7
  • BitFit provides the best accuracy-efficiency tradeoff and can outperform updating all parameters in the number of dataset wins.The results show diminishing returns, marginal gains, or decreased performance as the parameter count increases beyond BitFit.Source: §5.2 · p. 7

5.3 Putting Everything Together

  • Combining null prompts with BitFit yields competitive accuracy, simple setup, and small memory costs for each downstream task.The paper’s final comparison reports prompt-based finetuning with null prompts against representative few-shot methods.Source: Table 2 · p. 7, §5.3 · p. 8

6 Conclusion and Future Work

The conclusion presents null prompts and BitFit as a simple, efficient, and competitive few-shot approach. It also identifies open questions about why prompt-based finetuning works and whether the findings extend beyond the evaluated masked LMs.

6 Conclusion and Future Work

  • Null prompts make prompt-based finetuning robust to prompt choices without degrading performance relative to manually written or learned prompts.They can be applied flexibly across different tasks.Source: §6 · p. 8
  • BitFit updates only bias terms, achieving comparable or better accuracy than full finetuning while being 1000x more memory efficient.The approach updates 0.1% of the parameters.Source: §6 · p. 8
  • The findings suggest that prompt-based finetuning gains are partly associated with predicting a [MASK] token using a pretrained MLM head.The paper calls for further analysis of how small few-shot design changes affect accuracy.Source: §6 · p. 8
  • The experiments use relatively small masked LMs, so extension to very large and left-to-right LMs remains future work.The authors explicitly identify this as a boundary of the current results.Source: §6 · p. 8

Related papers

Loading 2106.13353v2…