PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel

arXiv:2205.05638v2cs.LGcs.AIcs.CL

TL;DR

Few-shot ICL is flexible but repeatedly processes labeled examples, creating computational and memory costs, while evidence comparing PEFT in very low-data settings has been limited. The paper compares the paradigms and introduces T-Few, a T0-based recipe using (IA)3 and additional losses. T-Few achieves higher accuracy than ICL at lower computational cost and reaches super-human performance on RAFT.

  • Problem

    Evidence has been limited on whether PEFT works well with very little labeled data, despite ICL’s computational costs and performance and prompt-sensitivity drawbacks.

    Source: §1 · p. 2

  • Method

    The paper develops T-Few, a fixed T0-based recipe using (IA)3 activation rescaling, unlikelihood and length-normalized losses, and pre-training of PEFT parameters.

    Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 6

  • Results

    T-Few attains higher accuracy than few-shot ICL at lower computational cost, and achieves 75.8% accuracy on RAFT versus 73.5% for the human baseline.

    Source: §5 · p. 9, §4.3 · p. 9

  • Takeaways & Limitations

    T-Few can be applied as-is to novel real-world tasks without task-specific hyperparameter tuning or other changes.

    Source: §4.3 · p. 9, §5 · p. 9

    ICL inference costs grow with the number of in-context examples, while PEFT adds a one-time fine-tuning cost and can incur method-dependent inference overhead.

    Source: §2.1 · p. 3, §2.2 · p. 4

Abstract

from arXiv · show

Few-shot in-context learning (ICL) enables pre-trained language models to perform a previously-unseen task without any gradient-based training by feeding a small number of training examples as part of the input. ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made. Parameter-efficient fine-tuning (PEFT) (e.g. adapter modules, prompt tuning, sparse update methods, etc.) offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task. In this paper, we rigorously compare few-shot ICL and PEFT and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs. Along the way, we introduce a new PEFT method called (IA)$^3$ that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters. We also propose a simple recipe based on the T0 model called T-Few that can be applied to new tasks without task-specific tuning or modifications. We validate the effectiveness of T-Few on completely unseen tasks by applying it to the RAFT benchmark, attaining super-human performance for the first time and outperforming the state-of-the-art by 6% absolute. All of the code used in our experiments is publicly available.

1 Introduction

The paper contrasts few-shot ICL with PEFT and proposes T-Few, a fixed recipe using T0, (IA)3, and additional loss terms for unseen tasks.

  • ICL performs new tasks from prompted examples without gradient-based training, but requires processing those examples for every prediction.Source: §1 · pp. 1–2
  • (IA)3 rescales attention keys, values, and feed-forward activations with learned vectors while introducing relatively few parameters.Source: Figure 1 · p. 2, §1 · p. 2
  • PEFT updates or adds only a small fraction of model parameters, supporting efficient adaptation and mixed-task batches.Source: §1 · p. 2
  • The paper proposes T-Few, a T0-based recipe combining (IA)3 with unlikelihood and length-normalized losses for novel tasks.Source: §1 · p. 2, Figure 1 · p. 2
  • T-Few is evaluated against strong ICL baselines and designed for application without task-specific tuning or modifications.Source: §1 · p. 2

2 Background

The background compares ICL and PEFT in terms of task adaptation, computation, memory, and storage, highlighting ICL’s inference costs and PEFT’s efficiency trade-offs.

Few-shot in-context learning (ICL)

  • ICL feeds prompted input-target examples and an unlabeled query to an autoregressive model, enabling immediate few-shot task performance.Source: §2.1 · pp. 2–3
  • k-shot ICL increases computational cost by approximately k + 1 times relative to processing the unlabeled example alone.Source: §2.1 · p. 3
  • ICL memory costs scale approximately linearly with k, while storing 32 prompted 512-token examples requires about 66 kilobytes on disk.Source: §2.1 · p. 3
  • Example ordering strongly influences ICL predictions, and performance can remain strong when in-context labels are swapped incorrectly.Source: §2.1 · p. 3

Parameter-efficient fine-tuning

  • PEFT updates or adds relatively few parameters, reducing memory and storage requirements while supporting some mixed-task batch configurations.Source: §2.2 · pp. 3–4
  • PEFT methods differ in inference overhead and mixed-task support, and their one-time fine-tuning cost is amortized during inference.Source: §2.2 · p. 4

3 Designing the T-Few Recipe

T-Few combines T0 with few-shot PEFT, loss terms tailored to rank classification, and the new (IA)3 activation-rescaling method. Across held-out tasks, these choices improve accuracy while targeting low parameter, memory, storage, and mixed-task inference costs.

3.1 Model and Datasets

  • T-Few uses T0 as its pretrained model and evaluates generalization on held-out sentence-completion, natural-language-inference, coreference, and word-sense-disambiguation tasks.The evaluation uses few-shot subsets, prompt templates, rank classification, and median accuracy across datasets, prompts, and subsets.Source: §3.1 · pp. 4–5

3.2 Unlikelihood Training and Length Normalization

  • 63.3% accuracy results when both length normalization and unlikelihood loss are added to full-model fine-tuning, improving over 60.7% without length normalization.Length normalization alone raises accuracy to 62.71%; the combined losses add no hyperparameters and are retained in the recipe.Source: §3.2 · p. 5

3 Designing the T-Few Recipe

  • The recipe is designed to minimize updated parameters, retain strong few-shot accuracy, and support mixed-task batches without task-specific model modifications.Activation-modifying methods are preferred because task-specific changes can be applied independently to examples in a batch.Source: §3 · p. 4, §3.3 · pp. 5–6

3.3 Parameter-efficient fine-tuning with (IA)3

  • (IA)3 rescales selected attention and feed-forward activations with learned vectors initialized to one, adding L(2d_k + 2d_v + d_ff) decoder parameters.The vectors are applied to keys and values in self-attention and encoder-decoder attention, plus intermediate feed-forward activations.Source: §3.3 · p. 6
  • Results for prompt tuning differ from prior reports, potentially because this study uses a different model and datasets, with validation performance fluctuating during training.The authors describe this as a hypothesis and note possible optimization issues for prompt tuning.Source: §3.3 · p. 6

4 Outperforming ICL with T-Few

T-Few uses one fixed recipe across tasks and outperforms few-shot ICL on held-out tasks while substantially reducing inference costs. It also performs strongly on RAFT, where it exceeds the human baseline, while ablations show benefits from its training components.

Held-out T0 tasks

  • T-Few outperforms GPT-3 175B few-shot ICL by 6% accuracy while using a model about 16× smaller.It also exceeds zero-shot T0, few-shot ICL with T5+LM, and smaller GPT-3 variants.Source: §4.1 · p. 8

Computational costs

  • 1,000× lower computational cost is reported for T-Few than for GPT-3 175B ICL in the held-out-task comparison.The table caption summarizes the accuracy and computational-cost comparison.Source: Table 1 · p. 8

Inference costs

  • 1.1e12 FLOPs are required for a T-Few prediction, versus 1.4e15 FLOPs for GPT-3 175B ICL.The GPT-3 ICL estimate is more than three orders of magnitude larger.Source: §4.2 · p. 8

Training costs

  • 20 GPT-3 175B ICL examples cost about as much FLOPs as training T-Few.Training T-Few requires approximately 2.7e16 FLOPs, while fine-tuning on one dataset takes about half an hour on one NVIDIA A100 GPU.Source: §4.2 · pp. 8–9

Storage costs

  • 4.2 MB per task for (IA)3 is larger than ICL’s 16 kB example storage, but 10,000 adaptations occupy about one T0 checkpoint.The T0 checkpoint is 41.5 GB.Source: §4.2 · p. 9

RAFT benchmark

  • 75.8% accuracy on RAFT exceeds the 73.5% human baseline and is 6% above the next-best method.GPT-3 175B achieves 62.7%, while T-Few is applied using the standard prompts without task-specific changes.Source: §4.3 · p. 9

Ablation experiments

  • Removing unlikelihood training and length normalization decreases average accuracy by 4.1%.Removing pre-training decreases accuracy by 1.6%, while removing both pre-training and the additional losses decreases it by 2.5%.Source: §4.4 · p. 9

5 Conclusion

The paper concludes that T-Few combines higher accuracy with lower computational cost than few-shot ICL. It also reports strong RAFT performance and identifies generative tasks as future work.

Conclusion

  • T-Few attains higher accuracy than few-shot ICL at lower computational cost using the (IA)3 PEFT method.(IA)3 rescales inner activations with learned vectors while adding only a tiny number of parameters.Source: §5 · p. 9
  • T-Few exceeds the human baseline on RAFT and outperforms prior submissions by a large margin without task-specific tuning.The recipe is applied as-is to the benchmark.Source: §5 · p. 9

Future work

  • The experiments cover classification tasks, and applying T-Few to summarization and question answering remains future work.Source: §5 · p. 9

A Compute resources used

The reported experiments used several GPU configurations and training durations for T0 models and PEFT methods. Pre-training (IA)3 required about one day on four GPUs.

Compute resources

C Full Unlikelihood Training and Length Normalization Results

Table 3 reports results using both unlikelihood training and length normalization.

D Full PEFT Results

The full PEFT evaluation compares multiple parameter-efficient methods under specified training configurations, alongside full-model fine-tuning and loss ablations.

E Full Pre-training Results

Table 8 reports per-dataset results for pre-training with (IA)3.

F Full Main Results

The main results compare T-Few with zero-shot T0, T5+LM in-context learning, and GPT-3 in-context learning baselines. T0 itself was unable to benefit from few-shot in-context examples in preliminary experiments.

G Full Ablation Results

Table 10 presents the T-Few ablation results.

H RAFT Experiment Details

The RAFT experiment applies T-Few across 11 tasks with task-specific handling for Banking 77. Results are reported per dataset alongside other top-performing methods.

Related papers

Loading 2205.05638v2…