PaperlayerView exact revision on arXiv

Source-linked AI summary

On the generalization of language models from in-context learning and finetuning: a controlled study

Andrew K. Lampinen, Arslan Chaudhry, Stephanie C. Y. Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, James L. McClelland

arXiv:2505.00661v3cs.CLcs.AIcs.LG

TL;DR

Fine-tuning can generalize narrowly from factual examples, whereas in-context learning may support more flexible deductions. The paper compares both modes on controlled datasets and augments fine-tuning with in-context reasoning traces, finding improved fine-tuning generalization and complementary benefits.

  • Problem

    The paper asks how factual generalization and deductive reasoning differ between in-context learning and fine-tuning, because fine-tuned models can fail on simple reversals and logical deductions.

  • Method

    The authors construct controlled synthetic datasets isolated from pretraining knowledge, expose models to matched information through in-context learning or fine-tuning, and augment fine-tuning data with in-context reasoning traces.

  • Results

    Across datasets, in-context learning generally generalizes better than fine-tuning, while reasoning-trace augmentation improves fine-tuning and can bridge the gap.

  • Takeaways & Limitations

    The results indicate that the complementary benefits of in-context learning and fine-tuning can be combined to improve generalization.

  • Takeaways & Limitations

    The main experiments rely on nonsense words and implausible operations, which may interfere with model performance and limit transfer to more plausible tasks.

Abstract

from arXiv · show

Large language models exhibit exciting capabilities, yet can show surprisingly narrow generalization from finetuning. E.g. they can fail to generalize to simple reversals of relations they are trained on, or fail to make simple logical deductions based on trained information. These failures to generalize factual information from fine-tuning can significantly hinder the reasoning capabilities of these models. On the other hand, language models' in-context learning (ICL) shows different inductive biases and deductive reasoning capabilities. Here, we explore these differences in generalization and deductive reasoning between in-context- and fine-tuning-based learning. To do so, we constructed several novel datasets to evaluate and improve models' abilities to make generalizations over factual information from novel data. These datasets are designed to create clean tests of generalization, by isolating the knowledge in the dataset from that in pretraining. We expose pretrained large models to controlled subsets of the information in these datasets -- either through ICL or fine-tuning -- and evaluate their performance on test sets that require various types of generalization. We find overall that in data-matched settings, ICL can generalize several types of inferences more flexibly than fine-tuning (though we also find some qualifications of prior findings, such as cases when fine-tuning can generalize to reversals embedded in a larger structure of knowledge). We build on these findings to propose a method to enable improved generalization from fine-tuning: adding in-context reasoning traces to finetuning data. We show that this method improves generalization across various splits of our datasets and other benchmarks. Our results have implications for understanding the generalization afforded by different modes of learning in language models, and practically improving their performance.

Introduction

This paper compares how pretrained language models generalize factual knowledge through in-context learning and fine-tuning, then uses in-context reasoning to improve fine-tuning generalization.

  • The authors use controlled synthetic datasets designed to avoid overlap with pretrained knowledge and test multiple generalization types.
  • The study examines distinct patterns of factual generalization from in-context learning and fine-tuning.
  • In-context learning over entire training datasets often generalizes better than fine-tuning on reversals, syllogistic inferences, and other logical deductions.
  • They propose augmenting fine-tuning data with reasoning traces generated by prompting a language model to reason in context.
  • Dataset augmentation improves fine-tuning generalization and can bridge the gap with in-context learning.
  • A fine-tuning method that breaks correlations among sentences amplifies the benefits of augmentation.

Datasets

The paper evaluates generalization using controlled synthetic datasets containing reversals, syllogisms, and hierarchical semantic knowledge, with nonsense terms reducing pretraining overlap.

  • The evaluation spans prior and newly constructed datasets designed to isolate different aspects of generalization.
  • The reversal dataset trains models on fictional-celebrity descriptions and tests whether they infer names from descriptions.
  • Simple reversals and syllogisms: Simple reversal tests contrast a trained relation with its reordered contradictory alternative, discouraging bag-of-words strategies.
  • Simple reversals and syllogisms: Simple syllogism examples contain two statements, and tests assess whether models make the corresponding logical inference.
  • Semantic structure benchmark: The semantic-structure benchmark represents 110 categories of animals and objects in a relational hierarchy supporting reversals, syllogisms, and longer deductions.
  • Semantic structure benchmark: Nonsense terms replace nouns, adjectives, and verbs to prevent overlap with pretraining data, though they may introduce tokenization and familiarity concerns.
  • Semantic structure benchmark: Training uses 2200 synthetic documents containing hierarchical facts, varied formatting, and some question-answer examples.
  • Semantic structure benchmark: Evaluation includes trained-fact rephrasings, relation reversals, syllogisms over trained facts, and category holdouts requiring potentially longer chains.

Methods

The methods compare likelihood-scored answers after in-context exposure or fine-tuning, then augment fine-tuning data with local and global reasoning traces.

  • Models are evaluated with multiple-choice likelihood scoring without providing answer choices in context.
  • Full-dataset in-context evaluation concatenates training documents into the prompt, with subsampling on the largest datasets because longer contexts can cause interference.
  • Fine-tuning experiments primarily tune Gemini 1.5 Flash for 200-1000 steps using batch sizes of 8 or 16 and learning rate 3 · 10^-4.
  • Dataset augmentation: Dataset augmentation uses in-context inference to expand fine-tuning coverage so models generalize out of context at test time.
  • Dataset augmentation: Local augmentation targets individual sentences with rephrasings and reversals, while global strategies relate information across documents.
  • Dataset augmentation: Global document augmentation concatenates the training dataset and prompts the model to generate inferences linking one document with the others.
  • Splitting linked documents into multiple sentence-level fine-tuning examples improves performance even after accounting for dataset size and gradient steps.

Experiments

Across controlled factual and process-generalization experiments, in-context learning generally outperforms standard fine-tuning, while reasoning-trace augmentation improves fine-tuning and can sometimes surpass ICL. Performance advantages vary by task structure, with category holdouts remaining difficult.

  • Reversal and syllogism tasks: In-context learning nearly reaches ceiling on reversal-curse reversals, whereas simple fine-tuning achieves near-zero accuracy; augmented fine-tuning performs similarly to ICL.Pretrained models perform near chance, indicating little dataset contamination.
  • Semantic structure benchmark: On the richer semantic structure benchmark, ICL generally outperforms fine-tuning, with larger benefits for reversals and syllogisms than for rephrased trained facts.Category-level holdouts remain difficult, and ICL improvements there are minimal.
  • Semantic structure benchmark: Augmenting fine-tuning data with in-context inferences improves performance across semantic-structure splits and often outperforms ICL.Most improvements are statistically significant, except for ICL on syllogisms and both methods on category holdouts, which show marginal improvements.
  • Semantic structure benchmark: Fine-tuning alone can generalize above chance to reversals when tested facts are embedded in a broader coherent knowledge structure, though ICL and augmented fine-tuning remain substantially better in most cases.The broader structure can provide information supporting reversed conclusions.
  • Reversal and syllogism tasks: In-context learning outperforms fine-tuning on simple reversal and syllogism datasets, while dataset augmentation produces strong fine-tuning improvements.Pretrained models perform near chance on these datasets.
  • Process knowledge: Preliminary derivatoid experiments show the same overall pattern: ICL outperforms fine-tuning, especially in low-shot settings, and augmentation improves fine-tuning.The authors note that process knowledge may involve distinct evaluation methods and factors from semantic knowledge.

Related work

Prior work studies in-context learning, out-of-context factual use, factual generalization failures, data augmentation, and synthetic data. This paper connects these literatures through controlled tests of factual generalization and model-generated augmentation.

  • In-context learning: Research on in-context learning examines its empirical, mechanistic, and theoretical learning and generalization patterns.This literature includes studies of scaling to hundreds or thousands of in-context examples.
  • Out-of-context learning: Out-of-context learning studies whether models can flexibly use information not directly included in the prompt at test time.The paper relates its partial fine-tuning generalization results to this literature but generally does not find reliable use of such information as in-context learning.
  • Factual learning and generalization: Factual-learning research has examined how language models acquire and generalize facts during pretraining and fine-tuning, including failures such as the Reversal Curse.Other work also studies hallucination changes during pretraining and fine-tuning.
  • Data augmentation: Data-augmentation research uses language models or targeted transformations to improve performance from small or narrow datasets, including attempts to address reversal failures.A closely related approach proposes deductive closure training to improve coverage.
  • Synthetic data: Synthetic-data research spans hand-designed datasets for domain-specific generalization and newer approaches that generate data directly from language models.The cited literature includes applications across linguistics, mathematics, and broader language-model performance improvement.

Discussion

The paper finds distinct generalization patterns for in-context learning and finetuning, with in-context reasoning often more flexible, and uses this difference to improve finetuning. It also frames the approach as increasing access to information through computation while noting important experimental limitations.

  • Findings: Controlled experiments show that in-context learning generally generalizes better than finetuning across several inference dimensions.
  • Implications: Adding in-context reasoning traces to finetuning data combines complementary benefits and improves finetuning generalization.
  • Scope: The study extends prior work beyond typical input-output tasks to examine factual knowledge and logical inferences such as reversals and syllogisms.
  • Interpretation: The authors interpret these gains as making information implicit in the data more explicit and accessible at test time.
  • Implications: The augmentation approach scales train-time computation through in-context inference to improve some aspects of model generalization.
  • Limitations: The main experiments use nonsense words and implausible operations, and the study does not test other language models.These choices limit how broadly the results can be generalized; plausible entities may yield greater benefits from in-context learning.

A.3 Compute resources

The experiments used substantial TPU resources, with costs varying by model evaluation mode and dataset size.

  • Reproducing all main-text experiments is estimated to require around 2000 TPU-hours.Individual finetuning passes took about 10–20 minutes on 128 devices, while ICL evaluation took about 1.5–2 hours on 64 devices.

B Supplemental experiments and ablations

The supplemental experiment tests how sentence-level splitting affects the semantic structure benchmark, both with and without augmentation.

  • The experiment compares sentence and document augmentation methods under sentence-level splitting on the semantic structure benchmark.Sentence augmentation rephrases each sentence, whereas document augmentation generates augmentation at the document level.

B.2 Non-sensification and Long-context ICL

The experiments examine how nonsensical names, sentence splitting, model size, and training choices affect ICL and fine-tuning on the semantic-structure and reversal datasets. Augmentation and sentence-level splitting generally improve fine-tuning, while nonsensification substantially reduces long-context ICL performance.

  • Nonsensification and long-context ICL: Nonsensification of celebrity names substantially reduces full-dataset ICL performance on the reversal dataset.The result suggests that unfamiliar nonsense words can hinder effective use of long contexts; the authors leave systematic study of this hypothesis to future work.
  • Sentence splitting: Sentence-level splitting consistently improves fine-tuning performance except for augmented dataset variants.The analysis compares document-level training with independent sentence splitting on the semantic-structure benchmark.
  • Model size: Augmented fine-tuning performs better across both the smaller Flash-8B and larger Gemini 1.5 Flash models.For Flash-8B, full-dataset ICL is worse than vanilla fine-tuning on some splits, including syllogisms.
  • Hyper-parameter checks: Changing batch size or training steps does not significantly improve the fine-tuning baseline or explain the main-paper gains from ICL and augmentation.Batch size 16 is slightly better and is used for the main results; baseline performance remains fairly stable across training steps.
  • Augmentation: Combining augmentation methods provides complementary benefits because different methods perform better across different holdouts.The comparison includes sentence-level, document-level, and global augmentation methods.

B.6 Process benchmark

The process benchmark is an exploratory test of whether models can apply a novel procedure to inputs rather than learn semantic facts. Preliminary results indicate that data augmentation benefits low-shot fine-tuning, leaving data-efficient fine-tuning as an open challenge.

  • Benchmark scope: The process benchmark tests applying a novel procedure to inputs, distinct from learning semantic factual data.The benchmark is exploratory and targets process knowledge rather than the semantic knowledge tested by the other benchmarks.
  • Preliminary findings: Preliminary results show benefits from data augmentation for low-shot fine-tuning.The authors present these findings as exploratory and identify data-efficient fine-tuning as a remaining open challenge.

Process benchmark description: “Derivatoids”

The derivatoids benchmark evaluates compositional generalization for a deliberately unfamiliar, derivative-like transformation over math-like expressions. It compares ICL and fine-tuning using controlled shot counts while measuring output similarity to the correct transformation.

  • Transformation rules: Primitive expressions follow specified transformation rules, while combinations of primitives follow separate composition rules.These rules are documented in the primitive and combination transformation tables.
  • Benchmark design: Derivatoids are a novel derivative-like transformation of math expressions, designed to test learning a process rather than facts or semantics.The benchmark uses familiar words and symbols while making the process unfamiliar to pretrained models.
  • Evaluation split: The benchmark tests combinatorial generalization by evaluating unseen combinations of primitives whose individual components appeared during training.For example, combinations involving exponentials and logarithms can be held out even when each primitive was trained in other combinations.
  • Data-efficiency setup: An ICL k-shot dataset places k process examples in context, whereas an SFT k-shot dataset contains k training examples before a final query is evaluated.The setup supports direct data-efficiency comparisons between in-context and supervised learning.
  • Evaluation and training: Model outputs are evaluated with ROUGE-L against the correct derivatoid output.Fine-tuning generally uses batch size 8 and learning rate 1 · 10^-5, with checkpoints selected using validation loss.

Experiments

Experiments compare ICL and SFT data efficiency on held-out combinations, then test whether train-time reasoning-style augmentations improve low-shot SFT. ICL generally leads at low shot counts, while augmentation improves 8-shot SFT under specific ordering and content choices.

  • Data efficiency: ICL generally outperforms SFT at low k, while SFT performance improves as the number of unique training examples increases.Here k denotes examples shown in context for ICL and unique training examples for SFT.
  • Evaluation target: The process experiments evaluate combinatorial generalization on held-out primitive combinations whose components were seen during training.All results in this section use multiplication-based combinations.
  • Augmented SFT: Augmentation is studied for k = 4 and k = 8 because SFT is generally outperformed by ICL in this low-shot regime.The experiments test whether additional reasoning-style training data can improve SFT.
  • Reflection augmentation: Programmatically generated reflection augmentations improve 8-shot SFT performance.The reflections describe the primitive categories and combination before presenting the answer, providing an upper bound on augmentation effects independent of generation quality.
  • Augmentation ablations: Reversing the answer-reflection order, training longer, or replacing nonsense words with real words does not exceed baseline non-augmented performance.The ordering manipulation did not reproduce the improvement, and the authors suggest the augmentation benefit may be primarily a test-time effect.
Loading 2505.00661v3…