PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaICL: Learning to Learn In Context

Sewon Min, Mike Lewis, Luke Zettlemoyer, Hannaneh Hajishirzi

arXiv:2110.15943v2cs.CLcs.AI

TL;DR

In-context learning avoids parameter updates but can lag behind supervised finetuning, vary substantially, and require task-specific templates. MetaICL meta-trains a pretrained language model across tasks to recover unseen task semantics from a few examples, outperforming strong baselines across disjoint tasks while remaining complementary to instructions.

  • Problem

    Ordinary in-context learning can underperform supervised finetuning, show high variance, and require difficult task-specific templates.

  • Method

    MetaICL meta-trains a pretrained language model on many tasks using example-conditioned inputs so it learns to in-context learn new tasks without parameter updates or task-specific templates.

  • Results

    MetaICL outperforms in-context and multi-task zero-shot baselines across disjoint tasks, often approaches or exceeds supervised finetuning, matches models with 8x more parameters, and benefits from diverse meta-training tasks.

  • Takeaways & Limitations

    MetaICL supports few-shot transfer across dissimilar tasks, and combining it with human-written instructions achieves the best performance.

  • Takeaways & Limitations

    The paper leaves systematic study of how to select meta-training tasks and relate them to particular target tasks for future work.

Abstract

from arXiv · show

We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set of training tasks. This meta-training enables the model to more effectively learn a new task in context at test time, by simply conditioning on a few training examples with no parameter updates or task-specific templates. We experiment on a large, diverse collection of tasks consisting of 142 NLP datasets including classification, question answering, natural language inference, paraphrase detection and more, across seven different meta-training/target splits. MetaICL outperforms a range of baselines including in-context learning without meta-training and multi-task learning followed by zero-shot transfer. We find that the gains are particularly significant for target tasks that have domain shifts from the meta-training tasks, and that using a diverse set of the meta-training tasks is key to improvements. We also show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task, and outperforms much bigger models with nearly 8x parameters. Finally, we show that MetaICL is complementary to human-written instructions, and the best performance can be achieved by combining both approaches.

1 Introduction

MetaICL addresses the limitations of ordinary in-context learning by meta-training a pretrained language model across tasks to learn new tasks from examples alone. Across seven disjoint task splits, it outperforms major baselines and is especially effective under task or domain shifts.

  • Approach: MetaICL meta-trains a pretrained language model on many tasks to recover a new task’s semantics from k examples without parameter updates or task-specific templates.Meta-training uses the same example-conditioning setup expected at test time.
  • Experimental scope: 142 datasets across seven settings provide evaluation with no overlap between meta-training and target tasks, yielding 52 unique target tasks.The collection spans classification, question answering, natural language inference, and paraphrase detection.
  • Results: MetaICL consistently outperforms in-context learning without meta-training and multi-task learning followed by zero-shot transfer.These comparisons cover a range of strong baselines.
  • Results: Gains over multi-task zero-shot transfer are particularly significant when meta-training and target tasks differ in formats, domains, or required skills.The authors attribute this pattern to recovering task semantics in context despite dissimilarity.
  • Results: MetaICL often approaches or exceeds supervised target-task finetuning and performs as well as models with 8x more parameters.The paper also identifies the number and diversity of meta-training tasks as important ingredients.
  • Results: MetaICL without templates surpasses recent instruction-based work, while combining MetaICL with human-written instructions achieves the best performance.Instructions therefore remain complementary to the learned in-context capability.

2 Related Work

Prior work establishes in-context learning and multi-task transfer as routes to learning new tasks without parameter updates, but their effectiveness is limited by task mismatch, model scale, or formatting requirements. MetaICL explicitly trains this in-context capability and reports improvements with a much smaller model.

  • In-context learning: In-context learning conditions a language model on concatenated training examples, but can perform poorly on tasks unlike language modeling, especially with smaller models.It also exhibits high variance and poor worst-case accuracy.
  • MetaICL: MetaICL explicitly trains on an in-context learning objective and improves few-shot performance even with smaller language models.The approach retains conditioning on training examples as its core mechanism.
  • Multi-task transfer: Earlier multi-task zero-shot methods either restrict target tasks to shared formats or require task reformatting, limiting their generality.Examples include methods centered on question-answering formats.
  • MetaICL: MetaICL learns new task semantics without manual effort and significantly outperforms zero-shot transfer methods.Its task interface is based on examples rather than task-specific templates.
  • MetaICL: MetaICL demonstrates improvements with a 770M-parameter model, whereas prior findings reported meta-training benefits only at 68B parameters or more.The comparison highlights a difference in the model scale at which gains were observed.
  • Positioning: Relative to concurrent work, MetaICL removes human-written templates or instructions and evaluates more diverse tasks with stronger baselines and broader experiments.The cited distinctions concern design requirements and experimental scale.

3 MetaICL

MetaICL uses multi-task meta-training to make inference on unseen tasks match the training-time in-context learning setup. The model predicts target labels or answers from concatenated demonstrations, with a channel variant that reverses the modeled direction.

  • MetaICL: MetaICL trains across many tasks so a model conditions on a small set of examples, recovers task semantics, and predicts an unseen task’s output.The same setup is used during meta-training and inference.
  • Instructions: MetaICL without instructions is better than instruction-based zero-shot transfer, while adding instructions further improves MetaICL performance.This comparison motivates treating instructions as complementary rather than necessary.
  • Meta-training: During meta-training, k examples condition the model while a final example supplies the supervised target, simulating inference on a test example.The model receives x1, y1, …, xk, yk, xk+1 and generates yk+1 using negative log likelihood.
  • Inference: At inference, the model concatenates k labeled examples with a test input, scores each candidate label or answer, and returns the maximum-probability candidate.Candidate sets support classification and question answering.
  • Channel MetaICL: Channel MetaICL reverses the input-output direction, training the model to generate xk+1 from y1, x1, …, yk, xk, yk+1-style context.At inference it scores candidates through P(x|y) with a uniform candidate prior.
  • Experimental settings: The evaluation uses seven settings with no overlap between meta-training and target tasks, and Table 2 records high- versus low-resource conditions.The accompanying statistics define the split structure and task counts.

4 Experimental Setup

The experiments evaluate MetaICL across diverse, non-overlapping NLP tasks and compare it with in-context, zero-shot, and fine-tuning baselines under controlled few-shot settings. The setup also removes task-specific templates and measures performance across task formats, resources, and domains.

  • Datasets: The benchmark contains 142 English tasks spanning classification, question answering, natural language inference, and paraphrase detection.
  • Datasets: The seven experimental settings use 52 unique target tasks with no overlap between meta-training and target tasks.
  • Datasets: The settings test high-resource-to-low-resource transfer, shared task formats, and generalization across different formats and required capabilities.
  • Datasets: Each setting includes target subsets with no domain overlap from meta-training, including finance, poetry, climate, and medical domains.
  • Baselines: The comparison includes raw-LM zero-shot and in-context methods, PMI and noisy-channel variants, multi-task zero-shot transfer, and target-task fine-tuning.
  • Experiment Details: MetaICL uses GPT-2 Large, 16 randomly sampled target examples across five seeds, Macro-F1 for classification, and Accuracy for non-classification tasks.

5 Experimental Results

MetaICL consistently outperforms strong in-context and zero-shot baselines, with especially large gains under low-resource and distribution-shift settings. Its performance improves with more and more diverse meta-training tasks, complements instructions, and can approach supervised fine-tuning and much larger models.

  • Main Results: Channel MetaICL achieves the best performance in 6 out of 7 settings, with 6–15% absolute gains in HR→LR, non-NLI→NLI, and non-Para→Para.These settings represent low-resource learning or substantial shifts in task formats, domains, or required skills.
  • Main Results: In QA→QA, MetaICL only marginally outperforms Multi-task 0-shot because the meta-training and target tasks are relatively similar.This is identified as an exception to the larger gains observed in most settings.
  • Main Results: MetaICL is better or comparable to raw-LM baselines across unseen-domain settings, whereas Multi-task 0-shot becomes less competitive under the harder generalization.The supplied results describe stronger gains over Multi-task 0-shot on target tasks from unseen domains.
  • Main Results: MetaICL matches or sometimes outperforms models fine-tuned without meta-training, although fine-tuning with meta-training performs better than both alternatives.The comparison shows promise for learning without target-task parameter updates while leaving room for improvement.
  • Ablations: Increasing the number of in-context training examples generally helps, while performance tends to saturate near k = 16 because of sequence-length limits.The ablation varies k from 0, 4, 8, 16 to 32; k = 0 is equivalent to zero-shot learning.
  • Ablations: Performance generally increases with more meta-training tasks, but substantial variance across task selections remains; diverse task sets outperform non-diverse sets by a substantial margin.The task-count ablation uses 7, 15, 30, and 61 tasks, while the diversity study compares 13-task sets.
  • Ablations: Combining MetaICL with human-written instructions yields the best performance, and MetaICL without instructions already outperforms Multi-task 0-shot with instructions.MetaICL benefits more from increasing instructions per task than the Multi-task 0-shot baseline, although a reduced-task instruction setting underperforms Channel MetaICL without instructions.

6 Conclusion

MetaICL meta-trains language models to recover task semantics from examples, outperforming several baselines across diverse unseen tasks while complementing natural instructions. The approach remains limited by context length, model scope, and evaluated task formats.

  • MetaICL meta-trains a language model to condition on training examples, recover task semantics, and predict outputs without target-task parameter updates.
  • Across 142 tasks and 52 target tasks in seven settings, MetaICL outperforms in-context and multi-task zero-shot baselines and matches or exceeds models eight times larger.
  • Meta-training task number and diversity are identified as ingredients for MetaICL success.
  • MetaICL and human-written instructions are complementary, with the best performance achieved by integrating them.
  • In-context learning requires longer contexts and is less efficient, while the experiments use a modest 770M-parameter causal model and focus on classification and multi-choice tasks.
  • Future work includes freeform generation, outperforming supervised models with meta-training, identifying useful meta-training tasks, and improving instruction integration.

A Dataset List

The dataset inventory covers 142 unique tasks and 52 unique target tasks across multiple task families and seven non-overlapping meta-training/target settings. The appendix documents how these settings and datasets are assembled.

  • The HR→LR setting is determined by training-data size, while Classification, NLI, and Paraphrase settings come from CROSSFIT.
  • QA settings combine question-answering datasets from CROSSFIT and UNIFIEDQA.
  • The dataset inventory is larger than those of several recent related works, whose task counts range from 32 to 62.

B Implementation Details

Implementation choices standardize example formatting, manage long concatenated contexts, and compare training and inference costs across baselines. Additional details specify preprocessing, instruction selection, and model-size ablations.

  • Preprocessing details: Raw GPT-2 uses spaces between inputs and outputs, whereas meta-trained models and raw GPT-J use one newline and three newlines between examples.
  • Preprocessing details: When contexts are too long, examples are capped at 256 tokens and earlier concatenated tokens are truncated; extractive-QA passages retain the ground-truth answer.
  • Comparison with baselines in training and inference cost: MetaICL takes 4.5 hours to train versus 3 hours for multi-task zero-shot baselines, which are roughly 4x more efficient at inference because their sequences are shorter.
  • Ablations in using instructions: Instruction selection prioritizes names containing multiple_choice, most_correct, or most_suitable after excluding no_option, then randomly samples otherwise.
  • Comparison with baselines in training and inference cost: Attention-layer memory scales as O(L^2), while feed-forward-layer memory scales as O(L); empirical memory usage tends to be linear for large transformers.
  • Model-size and baseline settings: Table 10 reports average and worst-case accuracy across seeds, and Table 11 reports the GPT-2 size ablation on HR→LR.

C.1 GPT-J results

The appendix reports raw GPT-J baselines and GPT-2 model-size comparisons. MetaICL consistently outperforms baselines across tested GPT-2 sizes and can let a smaller model exceed a much larger baseline.

  • GPT-J results: GPT-J raw-LM baselines use a 6B-parameter model, with average and worst-case accuracy reported across seeds for target training examples.
  • Model-size ablation: GPT-2 ablations compare small, medium, large, and XL models with 124M, 355M, 774M, and 1.5B parameters.
  • Model-size ablation: For all model sizes, Channel MetaICL significantly outperforms baselines.
  • Model-size ablation: 46.2 vs. 43.5: Channel MetaICL with GPT-2 Small outperforms the GPT-2 XL baseline despite being 12x smaller.

C.3 Which meta-training tasks are more helpful?

Meta-training task usefulness varies substantially across selections. Diverse, high-quality datasets can help, while usefulness depends on the target-task set and remains difficult to explain systematically.

  • 50 random seven-task meta-training sets were evaluated to correlate included tasks and task combinations with Channel MetaICL performance.The analysis measures how task inclusion relates to average performance across different random selections.
  • High-quality datasets with diverse domains, including the GLUE family, were often helpful for meta-training.
  • Adversarially collected datasets and tasks notably dissimilar from other tasks, such as WikiSQL semantic parsing, were often unhelpful.
  • Which tasks help depends largely on the choice of target tasks, so early conclusions about universally helpful or unhelpful tasks are unwarranted.The authors call for more systematic investigation of these effects.

C.4 Does MetaICL generalize when semantic hints from label words are removed?

The paper tests whether MetaICL can learn tasks when label words no longer provide semantic hints. Raw language models and models trained on original labels perform near random, while MetaICL is meta-trained with changing random-label mappings, although overall performance remains relatively poor.

  • Original dataset labels often contain semantic hints about the intended class meaning, such as entailment/not_entailment or positive/negative.
  • Labels were replaced with random English words sampled from 61,569 common words to remove semantic information during evaluation.
  • Raw language models and models trained on original data achieved near-random performance with replaced labels, indicating semantic label hints were necessary for them.
  • MetaICL was meta-trained with independently remapped random labels at each iteration, preventing overfitting to a specific label mapping.
  • Overall performance remained relatively poor, motivating future work on models that can in-context learn any task.
Loading 2110.15943v2…