PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, Yanai Elazar

arXiv:2305.16938v2cs.CL

TL;DR

Prior comparisons of few-shot fine-tuning and in-context learning used models of different sizes, leaving their out-of-domain generalization comparison confounded. This paper evaluates both methods with matched models, parameters, and examples, finding similar but highly variable generalization and showing that robust task adaptation remains challenging.

  • Problem

    Prior comparisons used different model sizes for fine-tuning and in-context learning, leaving the fairness of their out-of-domain generalization comparison unresolved.

  • Method

    The paper compares few-shot pattern-based fine-tuning and in-context learning with the same OPT models, matched parameter counts, and matched example counts.

  • Results

    Both approaches often achieve comparable results, with fine-tuning frequently generalizing better than in-context learning for the evaluated models and datasets.

  • Takeaways & Limitations

    Generalization varies substantially with model size, examples, data seeds, and pattern or verbalizer choices, so robust task adaptation remains an open challenge.

  • Takeaways & Limitations

    The study evaluates covariate-shift OOD generalization on specific challenge datasets, so different datasets might yield different conclusions.

Abstract

from arXiv · show

Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning due to its simplicity and improved out-of-domain generalization, and because extensive evidence shows that fine-tuned models pick up on spurious correlations. Unfortunately, previous comparisons of the two approaches were done using models of different sizes. This raises the question of whether the observed weaker out-of-domain generalization of fine-tuned models is an inherent property of fine-tuning or a limitation of the experimental setup. In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B. Our results show that fine-tuned language models can in fact generalize well out-of-domain. We find that both approaches generalize similarly; they exhibit large variation and depend on properties such as model size and the number of examples, highlighting that robust task adaptation remains a challenge.

1 Introduction

Fine-tuning and in-context learning are competing strategies for adapting pretrained language models, but prior comparisons confounded adaptation method with model size. This paper compares them under matched conditions to reassess out-of-domain generalization.

  • Fine-tuning updates model parameters using supervised target-task data, whereas in-context learning prompts the unchanged model with input–label pairs.
  • In-context learning offers reusable task behavior and natural-language control, but context limits constrain the number of demonstrations.
  • Prior studies often compared in-context learning in much larger models with fine-tuning in smaller models, leaving the source of their generalization difference unresolved.
  • The paper compares both strategies using the same OPT models, parameter counts from 125M to 30B, and the same number of examples.
  • Both approaches can be unstable, with fine-tuning affected by training instability and in-context learning affected by prompt choice.

2 Background

Pattern-based fine-tuning adapts language models through task-specific patterns and verbalizers, while in-context learning conditions predictions on demonstrations without changing weights. The paper challenges claims that in-context learning generally provides better out-of-domain performance.

  • Pattern-based fine-tuning: Pattern-based fine-tuning reuses the pretrained language-modeling head and requires an input pattern plus a verbalizer mapping vocabulary tokens to task labels.
  • Pattern-based fine-tuning: Pattern-based fine-tuning is efficient for few-shot learning, but its advantage over vanilla fine-tuning disappears when many examples are available.
  • In-context learning: In-context learning adapts a model by conditioning it on demonstrations followed by a transformed test input, without updating pretrained weights.
  • Recent claims that in-context learning generalizes better out of domain than fine-tuning often do not hold under the paper’s comparison.

3 A fair comparison of FT and ICL

The study evaluates fine-tuning and in-context learning under matched few-shot conditions using shared OPT models, controlled demonstrations, and challenge datasets. Its design varies model size, examples, patterns, and seeds to measure adaptation and generalization fairly.

  • Evaluation: In-domain generalization is measured by validation-set accuracy, while out-of-domain generalization uses challenge datasets under covariate shift.
  • Models and tasks: The experiments use seven OPT models from 125M to 30B parameters, all trained on the same data, isolating model-size effects.
  • Models and tasks: The study covers English natural language inference and paraphrase identification, evaluating MNLI and RTE against HANS and QQP against PAWS-QQP.
  • Few-shot setup: Few-shot runs sample 2, 16, 32, 64, or 128 training examples, use 10 example sets and three patterns, and produce 30 runs per adaptation method and example count.
  • Scope: The largest evaluated model is OPT-30B, which was the largest model the researchers could fit with their available resources.
  • Few-shot setup: The main ICL experiments use 16 demonstrations because OPT models have a fixed 2048-token context size.

4 Results

Across MNLI, RTE, and QQP, few-shot ICL and FT show dataset- and model-size-dependent in-domain and OOD performance. Fine-tuning can generalize OOD as well as or better than ICL, while model selection, training data, and fine-tuning strategy materially affect results.

  • In-domain performance: For MNLI and RTE, both ICL and FT usually exceed the majority baseline in-domain, with performance generally improving as model size increases.For ICL, the OPT 30B model reaches 71.4% average and 74.9% maximum on MNLI, and 61.7% average and 66.8% maximum on RTE.
  • In-domain performance: With 16 examples, FT achieves in-domain performance similar to ICL across datasets and model sizes, although QQP remains weak for most models.Only 10 of 70 fine-tuned QQP models outperform the majority baseline.
  • Out-of-domain performance: Most ICL models perform near the majority baseline OOD on MNLI and QQP, while only 4 of 10 OPT 30B MNLI runs show good OOD generalization.On RTE, OPT 30B in-domain and OOD performance mostly overlap, but comparisons with the smallest FT models can misleadingly favor ICL.
  • Out-of-domain performance: As model size increases, fine-tuned models’ OOD performance increases, and larger MNLI and RTE models often match or exceed their in-domain performance.This pattern is especially evident for MNLI models of 6.7B, 13B, and 30B, and is strong on RTE even for the 1.3B model.
  • Overall comparison: Overall, FT generalizes OOD as well as or better than ICL, underscoring the need to compare adaptation strategies with models of equal size.The comparison uses matched models, parameter counts, and example counts, with statistical tests reported for RTE and MNLI.
  • Model selection: Selecting FT checkpoints by OOD performance lets smaller models generalize well, whereas selecting by in-domain performance yields good OOD performance mainly for the largest models.For 2.7B RTE, 7 of 10 models have OOD performance equal to or better than in-domain performance under OOD-based selection.
  • Training data and fine-tuning approaches: For larger models, increasing the FT training set improves OOD performance, while for the smallest models it raises in-domain performance but leaves OOD performance low.Using 50 randomly selected examples to estimate OOD performance correlates very highly with evaluation on all examples: Pearson correlation 0.99.
  • Training data and fine-tuning approaches: Pattern-based FT performs similarly with LoRA or full-parameter training on RTE, and Pythia experiments reproduce the broader model-size and OOD trends.For most Pythia sizes, FT achieves significantly better OOD performance than ICL, while both in-domain and OOD performance improve with more data.

5 Discussion

The results show that fine-tuning and in-context learning can be more similar than previously thought, but both exhibit substantial instability and sensitivity to adaptation choices.

  • What signal to learn from?: Both ICL and fine-tuning results exhibit large variance across in-domain and out-of-domain evaluations.Fine-tuning can show different OOD behavior when only the data seed changes.
  • What signal to learn from?: ICL generalization depends heavily on the selected pattern and verbalizer, making training-data and pattern choices important for task adaptation.Prior work also found that misleading patterns can yield strong pattern-based fine-tuning performance.
  • Advances in task adaptation: Vanilla ICL and pattern-based fine-tuning achieve similar performance in-domain and out-of-domain, suggesting that ICL improvements may also apply to PBFT.The paper expects methods such as calibration and chain-of-thought prompting to achieve similar results when applied to PBFT.
  • Analyzing the fine-tuning loss surface: Some fine-tuning runs undergo dramatic OOD-performance changes after 75 steps, and the relationship between the loss surface and OOD generalization remains for future study.The authors report heavy OOD fluctuations and changes in generalization strategy during fine-tuning.

6 Comparing FT and ICL

Fine-tuning and in-context learning involve different usability, resource, and inference trade-offs. Both can perform strongly, improve with model size, and differ in how they use data and computation.

  • Fine-tuning requires model-training expertise, whereas ICL uses natural language and is more accessible to non-experts.
  • ICL is highly reusable because it leaves pretrained model weights unchanged, while fine-tuning is generally less reusable except with parameter-efficient methods.
  • ICL requires large models, whereas fine-tuning works well with small models, limiting ICL applicability for low-resource languages.Training billion-parameter models requires amounts of data unavailable for many languages, where fine-tuning remains dominant.
  • Fine-tuning requires potentially expensive training, but its inference is faster than ICL because test inputs need not include demonstrations.ICL inference processes all demonstrations with each test instance.
  • ICL demonstrations are limited by fixed context size, whereas fine-tuning permits unlimited training examples.Some methods allow infinite context, but most current successful language models have limited context sizes.
  • Both approaches achieve strong in-domain and OOD performance and improve with model size, while fine-tuning benefits more from additional samples.
  • Because both methods remain relatively recent and poorly understood, further research is needed to clarify their strengths and weaknesses.

7 Related work

Prior comparisons often used unequal model sizes or training settings, complicating conclusions about the relative generalization of fine-tuning and ICL. Related findings are mixed across tasks, architectures, and scales.

  • Brown et al. found similar question-answering results for few-shot ICL and fully supervised fine-tuning, but compared ICL with smaller fine-tuned models.For SuperGLUE, their smaller-model experiments found that fine-tuning largely outperformed ICL.
  • Si et al. reported better OOD performance for ICL than fine-tuning, but compared GPT-3 with RoBERTa-large and used different training settings.GPT-3 had 500 times as many parameters as RoBERTa-large in that comparison.
  • Earlier work reported that increasing model size had little effect on generalization up to 1.5B parameters, whereas these findings suggest that the in-domain–OOD relationship depends on model size.
  • Utama et al. found that encoder-only models using pattern-based few-shot fine-tuning adopted inference heuristics and performed poorly OOD, especially with more training data.This contrasts with the paper’s finding that pattern-based few-shot fine-tuning can lead to stronger OOD generalization.

8 Conclusion

The paper fairly compares few-shot ICL and fine-tuning across matched OPT models, examples, and parameter scales. It finds that both improve with scale, while performance remains variable and robust adaptation remains unresolved.

  • The study compares few-shot ICL and fine-tuning for in-domain and OOD generalization using matched OPT models and experimental conditions.
  • The comparison spans OPT models from 125M to 30B parameters, three classification datasets, and two tasks.
  • For OPT-30B, fine-tuning outperforms ICL on both in-domain and OOD performance and improves further with more training data.
  • Both approaches improve as models become larger, but their performance also exhibits high variance.
  • The results show that truly robust task adaptation remains an open challenge.

9 Limitations

The study is limited to covariate-shift OOD generalization, OPT-family models, English, and basic adaptation methods. It also does not empirically test larger models or GPT-3.

  • The experiments evaluate only covariate-shift OOD generalization on the specific challenge datasets used in the study.
  • The comparison focuses on OPT decoder-only models because comparable encoder-only models lack strong ICL abilities and suitable encoder–decoder models are limited.
  • GPT-3 is excluded because its fine-tuning procedure through the API is unclear, making a fair comparison difficult.
  • The study cannot empirically test similarly large models such as OPT-175B and evaluates only English models.
  • Only basic fine-tuning and ICL methods are evaluated, excluding advanced techniques such as calibration.

A Experimental details

The experiments compare ICL and pattern-based fine-tuning across models, datasets, patterns, examples, seeds, and checkpoint-selection strategies. Results show sensitivity to these choices and generally comparable adaptation performance, with fine-tuning sometimes stronger OOD.

  • Implementation: Experiments use Hugging Face Transformers with DeepSpeed on 8x A100 GPUs with 80GB memory each.
  • Datasets: The study evaluates NLI and paraphrase identification, using MNLI, RTE, QQP, HANS, and PAWS-QQP with majority-class baselines reported for each dataset.
  • Adaptation methods: ICL uses minimal, GPT-3, and eval-harness patterns, while PBFT reuses the minimal patterns and requires a verbalizer.
  • ICL results: On QQP, the largest OPT model’s ICL results are much worse than prior results and close to the majority-class classifier.
  • Significance tests: RTE results show that ICL outperforms FT mainly when comparing large models with smaller models; at matched sizes, FT performs at least equally well and sometimes significantly better.
  • Significance tests: On MNLI, ICL outperforms FT in-domain for models of 6.7B parameters and larger, while OOD advantages again occur only against much smaller models.
  • Sensitivity analyses: Across patterns and datasets, models are sensitive to demonstration count and do not necessarily improve with more demonstrations; only the largest model benefits from instructive patterns on MNLI and RTE.

B.3 Fine-tuning

Fine-tuning outcomes depend on training-example count and checkpoint selection. For MNLI, merging neutral with contradiction produces a better in-domain–OOD relationship than removing neutral examples.

  • Fine-tuning results: Figures 11–13 compare fine-tuning results across training-example counts and model-selection strategies for MNLI, RTE, and QQP.
  • Fine-tuning results: Model selection is important for both in-domain and out-of-domain performance.
  • MNLI binarization: Merging MNLI’s neutral and contradiction classes yields a better relationship between in-domain and OOD performance than removing neutral examples.

C Additional results for Pythia models

Additional Pythia experiments reproduce key OPT patterns: ICL is highly pattern-sensitive, while fine-tuning benefits from model scale and can outperform ICL OOD. Generalization also varies substantially across seeds and training.

  • Experimental setup: Pythia experiments cover 410M–12B parameter models on RTE, using 16 examples for ICL and 16 or 128 examples for FT.
  • ICL results: All Pythia models perform poorly with the minimal ICL pattern.
  • ICL results: With the GPT-3 pattern, Pythia model size clearly affects both in-domain and out-of-domain performance, whereas with eval-harness only in-domain performance improves.
  • Fine-tuning results: With 16 samples and OOD-based checkpoint selection, almost all fine-tuned models perform significantly better OOD than ICL-adapted models.
  • Fine-tuning results: With 128 examples, model size has a clear effect on both in-domain and out-of-domain fine-tuning performance.
  • Training stability: Out-of-domain performance varies considerably across random seeds and during fine-tuning on MNLI and RTE.
  • Figures: The figures organize performance by model size, training-data amount, pattern or selection strategy, and random seed, with axes indicating majority-class baselines where specified.
  • Figures: Figure 16 reports ICL and FT results for Pythia models across patterns, example counts, checkpoint strategies, and 10 random seeds.
Loading 2305.16938v2…