Source-linked AI summary
General-Purpose In-Context Learning by Meta-Learning Transformers
Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, Luke Metz
TL;DR
The paper asks whether black-box models with minimal inductive bias can learn general-purpose in-context algorithms that improve from examples across different datasets. It meta-trains Transformers and characterizes when learning generalizes, memorizes, or fails; the results identify algorithmic transitions and accessible memory as important constraints.
Problem
Method
The authors meta-train black-box Transformers on tasks generated from existing datasets to produce test predictions from in-context examples.
Source: §1 · p. 2, §3.2 · p. 4Results
Black-box models can learn general-purpose in-context algorithms, with behavior shifting from task memorization to task identification to general learning-to-learn as task diversity and model size increase.
Source: §1 · p. 2, §4.1 · p. 6, §6 · p. 10Takeaways & Limitations
Accessible state size significantly determines how well architectures learn across tasks, while optimizer, hyperparameter, and biased-distribution interventions can improve meta-training.
Source: §6 · p. 10The discovered algorithms are currently limited in applicability beyond random-projection input and output sizes, and processing very long input sequences remains difficult in the black-box setting.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Modern machine learning requires system designers to specify aspects of the learning pipeline, such as losses, architectures, and optimizers. Meta-learning, or learning-to-learn, instead aims to learn those aspects, and promises to unlock greater capabilities with less manual effort. One particularly ambitious goal of meta-learning is to train general-purpose in-context learning algorithms from scratch, using only black-box models with minimal inductive bias. Such a model takes in training data, and produces test-set predictions across a wide range of problems, without any explicit definition of an inference model, training loss, or optimization algorithm. In this paper we show that Transformers and other black-box models can be meta-trained to act as general-purpose in-context learners. We characterize transitions between algorithms that generalize, algorithms that memorize, and algorithms that fail to meta-train at all, induced by changes in model size, number of tasks, and meta-optimization. We further show that the capabilities of meta-trained algorithms are bottlenecked by the accessible state size (memory) determining the next prediction, unlike standard models which are thought to be bottlenecked by parameter count. Finally, we propose practical interventions such as biasing the training distribution that improve the meta-training and meta-generalization of general-purpose in-context learning algorithms.
1 INTRODUCTION
The paper seeks general-purpose in-context meta-learners with minimal inductive bias, avoiding design effort and restrictions associated with hand-engineered constraints. It investigates how Transformers can learn from training examples and generalize to different datasets, and studies conditions and interventions affecting that behavior.
1 INTRODUCTION
- Prior approaches encourage generalization through inductive biases such as gradient-based rules, symbolic graphs, or parameter sharing, but require more design effort and may restrict discoverable algorithms.The paper cites gradient-based approaches (Kirsch et al., 2019; Oh et al., 2020), symbolic graphs, and parameter sharing (Kirsch & Schmidhuber, 2020; Kirsch et al., 2021).Source: §1 · p. 1
- Black-box sequence models can take training data and produce test-set predictions without explicitly specified inference models, losses, or optimization algorithms.The paper considers LSTMs and Transformers as candidates for this in-context learning approach.Source: §1 · p. 1
- The study investigates training in-context meta-learners to generalize to datasets different from those used in meta-training.It characterizes transitions among memorization, task identification, and general learning-to-learn, examines accessible state size, and proposes interventions to improve meta-training.Source: §1 · p. 2
2 BACKGROUND
The background defines supervised in-context learning as learning predictions from examples without explicit gradient computation at meta-test time. It frames general-purpose learning as improving across varied tasks and unseen datasets.
2 BACKGROUND
- A learning algorithm is a function that improves its query predictions as its training set grows; meta-learning seeks such functions through meta-optimization.In the paper’s supervised setting, the function maps a training set and query input to a query prediction, with no explicit gradients computed at meta-test time.Source: §2 · p. 2
- A general-purpose learning algorithm improves on a wide range of tasks, including entirely different datasets such as MNIST, Fashion MNIST, and CIFAR10.The paper contrasts this with meta-learners that often generalize poorly to entirely new datasets.Source: §2 · p. 2
- The paper classifies neural-network algorithms along two dimensions: how much they learn and how much they generalize.The table caption identifies these dimensions but does not specify individual classifications.Source: Table 1 · p. 2
3 GENERAL-PURPOSE IN-CONTEXT LEARNING
The section presents GPICL as a Transformer-based in-context learner and distinguishes task memorization, task identification, zero-shot generalization, and learning that improves across seen and unseen tasks. It describes generating augmented tasks, training on query predictions after dataset prefixes, and evaluating without gradient-based learning at meta-test time.
3 GENERAL-PURPOSE IN-CONTEXT LEARNING
- GPICL transitions from memorization to learning-to-learn, which improves with examples on both seen and significantly different unseen tasks.The figure caption contrasts this transition with an MLP, which does not generalize to unseen tasks.Source: §3 · p. 3, Figure 2 · p. 4
- The section distinguishes task memorization, task identification, zero-shot generalization, and general-purpose learning by how models learn and generalize.Task identification improves with examples only on similar trained tasks; zero-shot generalization applies immediately to unseen tasks.Source: §3 · pp. 2–3
3.1 GENERATING TASKS FOR LEARNING-TO-LEARN
- Because commonly available tasks are scarce relative to datapoints, prior approaches add structure or generate new tasks, though diverse, structured task generation is difficult.The paper generates tasks by augmenting a base dataset with transformations and output-label permutations.Source: §3.1 · p. 3, §3 · p. 3
3.2 META-LEARNING AND META-TESTING
- Meta-training samples augmented datasets and minimizes query-prediction losses after observing dataset prefixes.For classification, the loss is cross-entropy; training uses backpropagation and Adam.Source: §3.2 · p. 4
- At meta-test time, GPICL predicts by evaluating its network on a dataset and query, with no gradient-based learning.Evaluation uses test-split datapoints from the same or a different base dataset, including Fashion MNIST or CIFAR10.Source: §3.2 · p. 4
4 EXPERIMENTS ON THE EMERGENCE OF GENERAL LEARNING-TO-LEARN
The experiments show that black-box Transformers can learn general-purpose in-context learning algorithms when trained with sufficient task diversity and model size. Their behavior depends on task distribution, accessible state, and meta-optimization, with interventions and pretrained features offering ways to improve learning.
4 EXPERIMENTS ON THE EMERGENCE OF GENERAL LEARNING-TO-LEARN
- With sufficient task diversity and model size, a Transformer can learn in context and generalize to unseen tasks and datasets without meta-test gradient updates.The authors report needing at least 213 = 8192 tasks to reach the learning-to-learn regime.Source: §4 · p. 5
4.1 LARGE DATA: GENERALIZATION AND ALGORITHMIC TRANSITIONS
- Performance improves with more test-time examples, and meta-trained algorithms transfer across datasets, with no observed generalization gap from MNIST training to FashionMNIST.Transfer from FashionMNIST to MNIST also occurs, but with a larger generalization gap.Source: §4.1 · p. 5
- GPICL comes close to VSML without its associated inductive bias and generalizes across many datasets, though learning CIFAR10 and SVHN from 99 examples is difficult.Table 2 evaluates meta-test generalization after augmented-MNIST meta-training; its caption reports GPICL is competitive with approaches requiring more inductive bias.Source: §4.1 · pp. 5–6, Table 2 · p. 6
- As the number of meta-training tasks increases, behavior shifts from task memorization to seen-task identification and then learning that generalizes across tasks and datasets.The phases are distinguished by within-sequence performance gains on seen tasks, unseen tasks, and tasks from a different base dataset.Source: §4.1 · p. 6, Figure 4 · p. 6
4.2 ARCHITECTURE: LARGE MEMORY (STATE) IS CRUCIAL FOR LEARNING
- Across architectures, accessible state size predicts learning-to-learn performance more consistently than parameter count.The authors describe state as the information bottleneck that must retain what has been learned so far to make predictions.Source: §4.2 · p. 7, Figure 5 · p. 7
4.3 CHALLENGES IN META-OPTIMIZATION
- Meta-training can remain on a loss plateau for long periods, during which generalization to unseen tasks worsens before loss drops sharply.In the reported setting, the drop occurs around 35 thousand steps; the authors suggest initial memorization may enable the subsequent learning-to-learn phase.Source: §4.3 · p. 7, Figure 6 · p. 7
- Larger meta-batches shorten plateaus, smaller Adam ϵ more than halves their length, and biasing the task distribution eliminates the plateau.Increasing the number of tasks can lengthen the plateau; distribution biasing yields a smooth path from memorizing to learning, while relying on parallelizability is a constraint of the batch-size intervention.Source: §4.3 · p. 8
4.4 DOMAIN-SPECIFIC AND GENERAL-PURPOSE LEARNING
- Using pretrained ImageNet features accelerates learning on matching domains: after 100 examples, MNIST-meta-trained GPICL reaches about 45% accuracy on CIFAR10.The paper also reports that CLIP embeddings can further improve learning efficiency.Source: §4.4 · p. 9
5 RELATED WORK
Prior work has pursued more general-purpose meta-learners through architectural and algorithmic inductive biases, while black-box sequence models can learn in context with little such bias. This work positions GPICL among those approaches and studies generalization across datasets.
In-context learning with black-box models
- Pre-trained features can accelerate in-context learning on matching-domain datasets; with 100 examples, the algorithm achieves about 45% accuracy on CIFAR10.The algorithms still generalize to a wide range of datasets.Source: Figure 9 · p. 9
Meta-learning: Inductive biases and general-purpose learning algorithms
- Meta-learning research has also pre-wired learning algorithms, optimizers, or loss functions, searched over hyperparameters, and used fast-weight programmers or hypernetworks.These approaches commonly draw on existing human-engineered learning algorithms.Source: §5 · p. 9
- Prior methods sought generality using biases such as gradient-based learning rules, symbolic graphs, parameter sharing, symmetries, or evolutionary approaches.These methods aim to encourage learning rather than memorization.Source: §5 · pp. 9–10
In-context learning with black-box models
- Black-box sequence models can learn-to-learn through activations with little architectural or algorithmic bias, using feedback or demonstrations in their inputs.Transformer-based in-context learning has also been studied in language models, where task demonstrations support language-task inference.Source: §5 · p. 10
General-purpose in-context learning
- GPICL investigates meta-generalization of black-box learning algorithms directly across datasets including MNIST, Fashion MNIST, and CIFAR10, unlike prior approaches relying on fine-tuning for complex classification settings.PFNs demonstrated learning to learn on small tabular datasets from synthetic problems; experiments on settings such as Omniglot relied on fine-tuning.Source: §5 · p. 10
6 DISCUSSION AND CONCLUSION
The paper demonstrates that black-box models, including Transformers, can learn general-purpose in-context algorithms from tasks generated from existing datasets. It identifies task diversity, model state, and meta-optimization as important conditions, while noting limits in input/output flexibility and long-sequence processing.
6 DISCUSSION AND CONCLUSION
- Generating tasks from existing datasets enables black-box models such as Transformers to meta-learn general-purpose in-context learning algorithms.The observed behavior transitions from task memorization through task identification to general learning-to-learn as task count or model size scales.Source: §6 · p. 10
- The size of model state significantly determines learning-to-learn capability across architectures, while meta-optimization remains challenging and can benefit from optimizer, hyperparameter, and data-distribution interventions.The paper also demonstrates combining domain-specific learning with general-purpose learning.Source: §6 · p. 10
- Applicability to arbitrary input and output sizes remains limited beyond random projections, and black-box models still struggle to process the millions of inputs some learning algorithms require.Recurrent models may accumulate errors, while Transformer computational complexity grows quadratically with sequence length.Source: §6 · p. 10
A.1 SUMMARY OF INSIGHTS
The insights emphasize that general-purpose learning-to-learn can emerge in black-box models, that data distributions shape generality, and that transitions and state size characterize the resulting capabilities.
A.1 SUMMARY OF INSIGHTS
- Effective in-context learning algorithms can emerge in black-box models with few inductive biases when meta-training uses sufficiently diverse tasks and large models.The reported transition required at least 2^13 = 8192 tasks.Source: §A.1 · p. 14
- Simple augmentations promoting permutation and scale invariance can improve the generality of discovered learning algorithms when large task distributions are unavailable.The data distribution can control algorithm generality.Source: §A.1 · p. 15
- As task count increases, meta-learned behavior transitions from task memorization to task identification and then general learning-to-learn.Source: §A.1 · p. 15
- State size is a stronger driver of learning-to-learn capability than parameter count, with self-attention giving Transformers increased state size.The specific inductive biases of each architecture matter to a smaller degree in this setting.Source: §A.1 · p. 15
A.2 LIMITATIONS
The discovered algorithms have limitations in handling arbitrary input/output sizes and very long sequences, both of which constrain practical application to broader problems.
Varying input and output sizes
- Arbitrary input and output sizes are supported only through random projections, which may make applying the algorithm to an unseen problem more difficult.Tokenizing inputs into compatible, unified representations is identified as a possible approach.Source: §A.2 · p. 15
Processing large datasets
- Processing millions of inputs remains difficult for black-box learning algorithms because recurrent models may accumulate errors and Transformer complexity grows quadratically with sequence length.Additional work is needed to build models that can process and train on long sequences.Source: §A.2 · p. 15
A.3 THE TRANSITION TO GENERAL LEARNING-TO-LEARN
Across runs near the task-count boundary, GPICL solutions separate into memorization and generalization modes. At a threshold, optimization begins finding generalizing solutions, which become consistent as task count grows.
A.3 THE TRANSITION TO GENERAL LEARNING-TO-LEARN
- At 6000 tasks, some runs first reach the lower-training-loss generalization cluster; with more tasks, runs settle there consistently.At lower task counts, fixed-capacity models find it harder to memorize or identify every task.Source: §A.3 · p. 15
- Across seeds and task counts near the transition, almost all final training-loss solutions fall into either a memorization cluster or a generalization cluster.Figure 10 plots final training loss for runs with different seeds and task counts around the transition boundary.Source: Figure 10 · p. 15, §A.3 · p. 15
A.4 ARCHITECTURAL DETAILS AND HYPER-PARAMETERS
The appendix specifies default Transformer dimensions and implementation variants for LSTM- and VSML-based learners, including their training configurations.
A.4 ARCHITECTURAL DETAILS AND HYPER-PARAMETERS
- The default Transformer uses 4 layers, 8 heads, key, value, and query sizes of 32, and model size 256.Its MLP expands the model size to a hidden representation of 4 × NM.Source: §A.4 · p. 16
- The outer-product LSTM replaces the context state with an outer-product update and uses an inner-product read-out.Source: §A.4 · p. 16
- The VSML variant uses one layer, self-messages of size 8, hidden size 16 per LSTM, and two micro-ticks per update.It is trained on 225 tasks with a 90% biased permutation distribution and task batch size 8.Source: §A.4 · p. 16
- The VSML variant without symmetries projects inputs and logits linearly and uses grids of k × k LSTMs, with k from 1 to 24.Each LSTM has hidden size 64; training uses 225 tasks, a 90% biased permutation distribution, and batch size 128.Source: §A.4 · p. 16
- The Table 2 LSTM uses hidden size 256 and 105 optimization steps, trained on 225 tasks with batch size 128.Larger hidden sizes were harder to optimize; the task distribution uses 90% biased permutations.Source: §A.4 · p. 16
A.5 EXPERIMENTAL DETAILS
The appendix documents experimental settings and additional tests of algorithmic transitions, embedding-based learning, memory, sequence length, and optimization behavior.
A.6 ADDITIONAL EXPERIMENTS
- Across datasets, Transformers show transitions from task memorization through task identification to learning-to-learn; ImageNet embeddings show similar transitions on CIFAR10 and other datasets.Figure 11 describes the three phases, while Figure 12 shows transitions with ImageNet embeddings.Source: Figure 11 · p. 17, §A.6 · pp. 17–18, Figure 12 · p. 17
- Meta-test loss drops at a task-count threshold on MNIST and KMNIST, with runs clustering into two modes; FashionMNIST’s transition is smoother.Source: §A.6 · p. 18, Figure 13 · p. 19
- Pre-trained CLIP embeddings support strong, generalizable in-context learning without meta-training on unseen datasets, and accelerate learning except on SVHN.Experiments use mini-ImageNet and a 10-way 10-shot setting.Source: §A.6 · p. 18, Figure 14 · p. 20
- Accessible state size is particularly important for storing learning progress, and predicts general-purpose learning performance more strongly than parameter count.Figure 15 compares state size and parameter count across architectures.Source: §A.6 · p. 18, Figure 15 · p. 20
- Plateau behavior varies with data-distribution bias; larger task batches or task distributions increase the number of tasks processed before leaving a plateau.The appendix also reports that reducing Adam’s ϵ or using gradient-sign normalization shortens plateaus, deeper VSML architectures make meta-optimization more difficult, and grokking was not elicited in memorization settings.Source: §A.6 · pp. 18, 21
- Longer sequences consistently improve final-query accuracy, although generalization to sequences longer than those used in meta-training remains a future direction.Main-paper experiments used sequences of 100 examples.Source: §A.6 · p. 18, Figure 16 · p. 20