In-context Learning and Induction Heads

Key Concepts

• The paper measures in-context learning as decreasing loss at increasing token indices, using the 500th-token loss minus the average 50th-token loss.The 50th and 500th positions are chosen somewhat arbitrarily, and different choices do not change the result.Source: In-context Learning · p. 4
• Induction heads implement [A][B] … [A] → [B] by matching prior occurrences of the current token and copying the following token.In two-layer attention-only models, one head copies previous-token information so a later head can match prefixes and promote the predicted continuation.Source: Induction Heads · p. 5
• Unlike fixed n-gram memorization, induction heads apply the same copying rule to arbitrary tokens and can operate out of distribution when local statistics generalize.Their behavior also supports wholesale sequence repetition, such as continuing “The cat …” with “sat on the mat.”Source: Induction Heads · p. 5
• This paper tests whether induction heads play a major role in general in-context learning, not merely literal copying or behavior limited to small attention-only models.Because large models with MLPs complicate mechanistic and eigenvalue analyses, the paper defines heads empirically by sequence-copying behavior before testing broader functions.Source: Induction Heads · p. 5
• Formally, an induction head must show prefix matching and copying on repeated random token sequences, increasing the likelihood of [B] after [A][B] … [A].Prefix matching attends to tokens followed by the current or recent tokens, while copying increases the logit of the attended token.Source: Induction Heads · p. 6
• The proposed broader role is analogical sequence copying or in-context nearest-neighbor behavior when induction heads operate on sufficiently abstract representations.This would connect sequence repetition to a simple form of few-shot learning and potentially explain broader in-context behavior.Source: Induction Heads · p. 6
• Induction heads may complete sequences by matching similar token pairs, potentially supporting behaviors such as word-by-word translation, although this general mechanism is not yet mechanistically proven.The proposed extension adapts literal copying to embedding-space similarity, and empirical examples plus mechanistic plausibility are deferred to later arguments.Source: Induction Heads · p. 6

Per-Token Loss Analysis

• Per-token loss vectors collect models’ log-likelihoods on 10,000 tokens across training snapshots and use PCA to visualize shared output trajectories.The first two principal components provide a high-level visualization, while omitting directions not captured by those components.Source: Per-Token Loss Analysis · pp. 6–7
• The overall case is strongest for small attention-only models, where ablations and mechanistic reverse engineering support induction heads as the majority mechanism, but larger MLP models rely mainly on potentially confounded correlations.Across all models, the evidence supports some role for induction heads; establishing that they account for the majority becomes harder as models grow.Source: Per-Token Loss Analysis · p. 7
• Induction heads form abruptly in exactly the same training window as in-context learning develops, across models of different sizes and datasets.The co-occurrence is consistent with induction heads explaining much of in-context learning, but the evidence remains correlational and could be confounded.Source: Per-Token Loss Analysis · pp. 8–10
• In-context learning, measured as the 50th-token loss minus the 500th-token loss, rises abruptly from less than 0.15 to roughly 0.4 nats between about 2.5 and 5 billion training tokens, then remains constant.Source: Per-Token Loss Analysis · p. 9
• After the phase change, loss continues improving beyond token 50, whereas before it largely stops there, confirming the in-context-learning increase beyond the chosen 50th-versus-500th-token measure.The derivative of loss with respect to logarithmic context position provides a complementary view of improvement per relative increase in context length.Source: Per-Token Loss Analysis · p. 10

Looking at the Phase Change More Closely

• During the phase change, repeated token sequences become easier to predict on later occurrences, whereas tokens followed by different continuations become harder, matching expected induction-head behavior.Examples include improved prediction of the repeated sequence “The Dursleys” and worsened prediction of the first “Mrs Potter” occurrence.Source: Looking at the Phase Change More Closely · pp. 13–14

Assessing the Evidence

• The evidence suggests induction heads contribute most in-context learning, but it does not establish them as the primary mechanism because their formation merely coincides with the improvement.More than 75% of final in-context learning forms in the window, yet other mechanisms or a shared latent variable could explain the co-occurrence, especially given limited time resolution in large models.Source: Looking at the Phase Change More Closely · p. 14, Assessing the Evidence · p. 15
• Changing the transformer architecture shifts the timing or possibility of induction-head formation and shifts the dramatic in-context-learning improvement in precisely the same way.This architectural intervention is presented as stronger evidence than merely observing co-variation, though the supplied passage introduces the argument without its detailed results.Source: Assessing the Evidence · p. 16
• A smeared-key architecture enables in-context learning in one-layer models and causes it to form earlier in deeper models.The modification interpolates each key between the current and previous token, making induction heads expressible at any depth.Source: Assessing the Evidence · pp. 16–17
• Knocking out induction heads greatly decreases in-context learning, with almost all such learning in small attention-only models appearing to come from these heads.This conclusion holds from the start of the phase change through its end, while no ablations are available for full-scale models.Source: Assessing the Evidence · pp. 18–19
• The authors conclude that induction heads are the primary mechanism for in-context learning in small attention-only models, but only suggestive evidence for models with MLPs.In MLP models, interactions between MLP and attention layers and masking from redundant heads complicate interpretation of marginal head ablations.Source: Assessing the Evidence · p. 19
• These larger-model examples show that induction heads can implement surprisingly abstract forms of in-context learning, but cover only a small number of tasks.Thus, the evidence applies to models up to 12B parameters but remains suggestive regarding in-context learning overall.Source: Assessing the Evidence · p. 19
• Induction heads can perform copying, translation, and a specific type of abstract pattern matching while meeting a strict empirical definition, including in models studied up to 12B parameters.Source: Assessing the Evidence · pp. 19–20

Behavior 1: Literal sequence copying

• A basic induction head literally copies repeated text by attending to earlier occurrences and attributing their contribution to the current prediction.The visualization presents attention in red and direct-path logit attribution in blue for repeated text.Source: Behavior 1: Literal sequence copying · p. 20
• For literal sequence copying, the head predicts repeated names, phrases, and sentences by attending to earlier occurrences where those sequences appeared.These predictions exemplify prefix-matching induction behavior: the head copies what followed a previous matching phrase.Source: Behavior 1: Literal sequence copying · p. 21

Behavior 2: Translation

• An induction head in a 40-layer model translates between English, French, and German by attending to semantically corresponding earlier tokens despite differing word order and token lengths.Its net logit attributions contribute to the correct translation, though they are not perfectly sharp.Source: Behavior 2: Translation · pp. 21–22

Behavior 3: Pattern matching

• Induction heads match abstract patterns by attending to earlier instances of the correct category, allocating about 65% of colon attention to correct positions across similar problems.They often skip examples with a matching word but the wrong overall pattern, although the behavior is imperfect.Source: Behavior 3: Pattern matching · p. 23

What’s going on with more abstract heads that are also induction heads?

• The same induction heads that copy literal sequences can also implement abstract behaviors, suggesting they contribute to in-context learning and may be its primary driver in very small models.The evidence establishes that induction heads contribute to some in-context learning, but does not determine what fraction they perform in general or establish the same mechanism in models with MLP layers.Source: What’s going on with more abstract… · pp. 25–26

Summary of Reverse Engineering Induction Heads

• Induction heads match preceding-token prefixes through QK-circuit K-composition, then copy the associated token through the OV circuit.More sophisticated variants can also shift query information and match sequences longer than one preceding token.Source: Summary of Reverse Engineering… · pp. 26–27
• In the small models studied previously, all induction heads used the described QK term to drive attention and a positive-eigenvalue circuit to copy.The copying circuit is characterized by a copying matrix with positive eigenvalues.Source: Summary of Reverse Engineering… · pp. 26–27
• GPT-2 showed evidence of a second pointer-arithmetic mechanism that uses positional embeddings and Q-composition rather than the basic mechanism.An earlier attention head attends to previous copies of the current token and copies their positional information.Source: Summary of Reverse Engineering… · p. 27

What About More Complex Induction Heads?

• Induction heads may explain most in-context learning in large models, extrapolating from stronger evidence in small transformers to weaker evidence in large transformers.The authors describe this as an extended inference from Arguments 1–5 and emphasize that the inference is a matter of judgment.Source: What About More Complex Induction Heads? · p. 28
• Small attention-only, small MLP, and full-scale models show analogous phase changes, sharp in-context-learning increases, similar PCA trajectories, and induction-head formation.These measurements are described as similar across all model cases with more than one layer.Source: What About More Complex Induction Heads? · p. 28
• Larger models may also form non-induction composition heads that contribute more than induction heads to phase-change and in-context-learning improvements.The proposed alternative is that larger models have enough capacity for additional Q- and K-composition mechanisms unavailable to small models.Source: What About More Complex Induction Heads? · p. 28

Model Analysis Table

• The analysis examines 34 decoder-only Transformer language models across small attention-only, small MLP, full-scale, and smeared-key series, including small models trained on another dataset.Source: Model Analysis Table · pp. 28–29

Model Details

• The models exhibit a “phase change” at approximately 1-3 billion tokens, unrelated to a later change and occurring during learning-rate warm-up over the first 1.5e9 tokens.Source: Model Details · p. 39

Architecture

• Unlike the small models, full-scale models use both dense attention heads that attend to any earlier token and local heads restricted to a fixed relative-position window.The context window and vocabulary remain 8192 tokens and 216 tokens, respectively.Source: FULL-SCALE MODELS · p. 39

Training snapshots

• Analyses use 15 exponentially spaced snapshots from steps 2^5 through 2^17 plus later saves, except the 40L model, which has 14 snapshots.Training uses a consistent token count across models through 2^11 steps, after which the 24L and 40L schedules increase tokens per step.Source: FULL-SCALE MODELS · p. 39

Model properties

• The provided full-scale model rows span 4 to 10 layers, 13M to 200M non-embedding parameters, activation dimensions from 512 to 1280, and 8 to 20 attention heads per layer.The rows list 4-layer, 13M-parameter models with activation dimension 512 and 8 heads; 6-layer, 42M with 768 and 12; and 10-layer, 200M with 1280 and 20.Source: FULL-SCALE MODELS · p. 40

SMEARED KEY MODELS

• Smeared-key models interpolate each token’s key with the previous token’s key using a trainable parameter, while otherwise matching the small-model training setup.These models are presented only in one-layer and two-layer sizes.Source: SMEARED KEY MODELS · p. 40

Seemingly Constant In-Context Learning Score

• The in-context-learning score is nearly constant across models after the phase change, regardless of model size or additional training.Changing the token indices used to define the score produces only small differences between models.Source: Seemingly Constant In-Context Learning… · p. 40
• Large models outperform small models at every index, but most of their advantage appears within the first ten context tokens before later loss decreases become roughly fixed.The authors suggest this may reflect large models extracting more information from early context and starting from a lower baseline loss.Source: Seemingly Constant In-Context Learning… · p. 41

Phase Change Effect on Loss Derivatives

• At the phase change, model-size rankings in loss-derivative curves invert: small models improve more slowly beforehand, whereas large models improve more slowly afterward.The inversion is presented as further evidence that the phase change marks an important transition in transformer training.Source: Phase Change Effect on Loss Derivatives · pp. 41–42

Additional Curiosities

• Additional curiosities include non-induction heads with phase-change-like ablation effects, unusually non-peaky four-layer MLP ablations, loss spikes, and anti-copying prefix-search heads.A six-layer MLP model also contains an induction head whose ablation has the opposite effect on the in-context-learning score.Source: Additional Curiosities · p. 42

Safety Implications

• The authors frame phase changes and in-context learning as safety-relevant because abrupt capability shifts and inference-time behavioral changes may complicate preparation and enable unwanted off-distribution generalization.They report that, at least in small models, induction heads appear to be the primary in-context-learning mechanism, with no observed evidence of mesa-optimization.Source: Safety Implications · p. 43

Linking Learning Dynamics, Scaling Laws, and Mechanistic Interpretability

• The in-context-learning phase change is proposed as a “Rosetta stone” connecting mechanistic interpretability, learning dynamics, scaling laws, and phase-change phenomena.It offers a concrete, localized training transition linked to an important capability and suitable for study in small models.Source: Linking Learning Dynamics, Scaling… · p. 43

Related Work

• Related work connects in-context learning to token-index loss analyses and word-order preservation, while some retraining-based noun-only experiments appear inconsistent with the induction-head hypothesis.The paper builds on its earlier transformer-circuits framework and focuses here on connections to machine-learning research.Source: Related Work · pp. 43–44
• Retraining-based comparisons are less directly related to natural-data models and difficult to interpret because run-to-run loss variation may exceed their small measured differences.These limitations concern both relevance to the present setting and the reliability of the comparison.Source: Related Work · p. 44
• Xie et al. find LSTMs outperform Transformers on synthetic HMM-generated data, possibly because that structure favors simulating HMMs rather than Transformer induction heads.The authors expect Transformers to outperform LSTMs on natural text, with induction heads as a major explanation, but regard the synthetic-data result as structurally specific.Source: Related Work · p. 44
• The paper uses “in-context learning” broadly for rapid adaptation to or recognition of context, rather than narrowly for few-shot task learning described by Brown et al.This includes cases where the context is not naturally understood as multiple shots of a task.Source: Related Work · p. 44

SCALING LAWS

• The induction-head phase change links a macroscopic loss phenomenon to an explanation in terms of underlying circuits.The authors describe this as the first bridge they are aware of between these abstraction levels.Source: SCALING LAWS · p. 45
• Induction heads may explain previously observed exceptions to scaling laws.Source: SCALING LAWS · p. 45
• One-layer transformers differ from deeper models in scaling behavior, plausibly because they lack induction heads, as previously observed by Kaplan et al.Source: SCALING LAWS · p. 45

• Other abrupt or non-smooth behaviors include sudden arithmetic changes, grokking, and double descent’s reversal from worsening to improving performance.Double descent can vary with parameter size, dataset size, or training amount, whereas its loss need not be discontinuous.Source: PHASE CHANGES & DISCONTINUOUS MODEL… · p. 45
• Induction-head phase changes are most analogous to Power et al.’s “grokking,” because both occur over the course of training.Grokking describes a discontinuous jump from random chance to perfect generalization during training.Source: PHASE CHANGES & DISCONTINUOUS MODEL… · p. 45
• The main contribution is linking observed behavioral changes to induction-head formation and providing a parameter-level understanding of the circuits involved.Source: PHASE CHANGES & DISCONTINUOUS MODEL… · p. 45
• The work claims induction heads are the first case with a mechanistic account of a phase change in machine learning.Source: PHASE CHANGES & DISCONTINUOUS MODEL… · p. 45

• Induction heads offer a preliminary bridge between understanding neural networks as circuits and understanding learning as circuits changing.The authors view this connection as a source of optimism for future work.Source: LEARNING DYNAMICS · p. 46
• Saxe et al. derive closed-form learning dynamics for linear neural networks, suggesting that independent paths aligned with data principal components can be viewed as circuits.Follow-up work also explores connections between this framework and models learning semantic representations.Source: LEARNING DYNAMICS · p. 46
• The paper speculates that loss-surface geometry may connect to circuit formation, with its described phase change corresponding to a large feature in the transformer loss landscape.This connection is presented as a concrete but still superficial direction for future investigation.Source: LEARNING DYNAMICS · p. 46

UNIVERSALITY

• Induction heads are presented as a universal attention pattern, with QK and OV analyses extending universality claims from attention behavior to circuit-level structure.The OV circuit computes roughly the token embedding following a previous copy of the current token.Source: UNIVERSALITY · p. 47

[46, 47, 48] [49]

• Prior translation-like tasks also show attention attending to the token about to be translated or the audio segment about to be transcribed.These patterns may look less induction-like when decoder visualizations are organized by predicted output tokens rather than input tokens.Source: [46, 47, 48] [49] · p. 47

REPLICATION

• Two-layer attention-only transformers reliably exhibit previous-token heads in layer 0 and induction heads in layer 1, often one of each, with expected attention behaviors.With exact previous-token attention, 99% of the loss difference was recovered; the induction-head case recovered about 65%.Source: REPLICATION · p. 48
• Including [A][B][C]...[A][B]→[C] matches recovered an additional 10% of loss difference, while the OV circuit copied tokens and the QK circuit used K-composition.The QK pathway copies information into and back from a separate subspace, and the heads also showed fuzzy semantic matching.Source: REPLICATION · p. 48
• Applying the paper’s induction-head criterion to public models found potential heads in GPT2 and GPT-Neo, including examples in GPT2-XL and GPT-Neo-2.7B.In the examined repetitions, nearly every token attended to its following token in the original paragraph.Source: REPLICATION · p. 48
• On a synthetic dataset, induction was learned with context length 4, suggesting that usefulness depends on the data distribution rather than the model alone.The replication notes uncertainty about how useful this observation is for interpretability overall.Source: REPLICATION · p. 48

Change Log

• The change log corrected the Prefix matching definition and clarified that copying-head logit effects are calculated using a ReLU.Footnotes also state that induction heads require multilayer composition and exhibit their defining properties imperfectly on a continuum.Source: Change Log · p. 49, Footnotes · p. 49

Footnotes

• Induction heads are defined using repeated copies of random sequences, distinguishing induction from heads that merely attend heuristically to previous tokens.A clear subset of heads exhibits the relevant properties at much greater than random chance.Source: Footnotes · p. 49
• Ablating most other attention heads can increase in-context learning because it may remove normal-prediction contributions, allowing in-context learning to predict additional tokens.This counterintuitive effect is explicitly presented as an observed result in the footnotes.Source: Footnotes · p. 50
• In attention-only models, in-context learning must originate with attention heads, making frozen-pattern head ablations a principled way to measure their contributions.The direct path depends only on the present token, while logits are approximately a sum of attention-head terms.Source: Footnotes · p. 50
• Ablation interpretation becomes harder with MLPs because in-context learning depends nonlinearly on the combination of ablated heads rather than their additive contributions.Ablations can shift MLP statistics or disrupt mechanisms that redundantly depend on multiple heads.Source: Footnotes · p. 50
• Basic copying heads are an alternative contender, but they also occur in one-layer models without the large in-context-learning increase seen in two-layer models.The footnote therefore characterizes induction heads as conceptually more powerful, without establishing a quantitative comparison.Source: Footnotes · p. 50

Citation Information

• The cited work is Olsson et al., “In-context Learning and Induction Heads,” published in the Transformer Circuits Thread in 2022.The BibTeX entry gives the title, authorship, year, venue, and URL.Source: Citation Information · p. 54

Where induction heads form

• In attention-only models, induction heads form in the last layer.Source: Where induction heads form · p. 54

Small models with MLPs:

• In small models with MLPs, induction heads form predominantly in the second-to-last layer by the 5L and 6L sizes.Source: Small models with MLPs · p. 55
• In the full-scale sweep, induction heads form earlier than in smaller models.Source: Small models with MLPs · p. 55
• In 24L and 40L models, most induction heads form before halfway through model depth.Source: Small models with MLPs · p. 55
• The reported head plots show a randomized sample of 100 heads, biased toward high prefix-matching scores, with most points translucent for readability.Source: Small models with MLPs · p. 55

What is the distribution of heads’ scores on our head activation evaluators?

• Heads with positive prefix-matching scores are more likely to have positive copying scores, with sharper distributions in small models and stronger correlation among the highest-scoring heads.For models with more than 100 heads, only a subset of heads is plotted.Source: What is the distribution of heads’… · p. 56

Validating head activation evaluators

• In small attention-only models, the copying evaluator correlates with the OV matrix, while the prefix-matching evaluator correlates with the previous-token QK-circuit eigenvalue trace.The copying comparison uses the signed fraction of OV eigenvalues, measured by their sum divided by the sum of absolute values.Source: Validating head activation evaluators · pp. 56–57

Data Collection

• Across training, the study measures per-token losses with individual-head ablations, activation-based estimates of copying and prefix matching, and mathematical previous-token QK-circuit measurements across model snapshots.This covers 200 snapshots each for twelve small models and 14 or 15 snapshots each for six full-scale models.Source: Data Collection · pp. 57–58

(1) Per-token losses (unaltered models)

• Each snapshot is evaluated on the same 10,000 randomly selected 512-token examples, with individual losses saved for every token.Using the same examples across snapshots and models enables direct comparison.Source: (1) Per-token losses (unaltered models) · p. 58
• Per-token losses are summarized by one consistent random token per example and by averages over each context index.The first produces a length-10,000 vector per snapshot, while the second produces a length-512 position-wise vector.Source: (1) Per-token losses (unaltered models) · p. 58

(2) Per-token losses (attention head ablations)

• Individual attention heads are studied by zeroing one head’s residual-stream contribution and comparing the resulting model behavior with the complete model.This functions as a knockout-style intervention for probing each head’s role.Source: (1) Per-token losses (unaltered models) · p. 58
• Pattern-preserving ablation zeros a head’s result vector while forcing later attention patterns to match a saved first run, isolating downstream V effects.Unlike full ablation, it preserves the ablated head’s contribution to later attention patterns and does not affect later Q and K calculations.Source: (1) Per-token losses (unaltered models) · p. 58

(3) Head activation evaluators

• The study constructs three empirical attention-head measurements from observed activations: copying tests same-token logit increases, prefix matching tests attention to earlier matching prefixes, and previous-token attention tests attention to the immediately preceding token.Source: (3) Head activation evaluators · p. 59
• The evaluators measure copying, prefix matching, and previous-token attention because these properties support induction heads’ prediction of repeated random-token sequences.Copying depends on the OV circuit, whereas prefix matching and previous-token attention depend on the QK circuit; copying plus prefix matching defines an induction head.Source: (3) Head activation evaluators · p. 59
• The heuristic tests use 25-token random sequences, with prefix matching repeating the sequence four times and averaging attention to tokens preceding earlier occurrences.Copying applies mean-subtracted ReLU-transformed logits, while prefix matching and previous-token scores are computed from attention patterns.Source: (3) Head activation evaluators · p. 59
• The additional QK measure computes the trace of a weighted pure K-composition matrix linking heads to earlier previous-token heads, and is defined only beyond the first layer.Earlier heads are weighted by their previous-token attention, and the measure captures positive same-matching eigenvalue structure.Source: (3) Head activation evaluators · pp. 59–60
• The appendix validates the evaluators’ connection to attention-head weight-matrix properties and finds that the eigenvalue measure correlates well with heuristic prefix matching.It also presents joint copying–prefix-matching score distributions for heads in small and large models.Source: (3) Head activation evaluators · pp. 59–60

2D in-context learning plot

• The 2D in-context learning plot shows loss across token positions and training time, with horizontal slices giving training-time loss curves and vertical slices giving performance by context index.It compares earlier and later token positions and supports reading both training dynamics and context-dependent prediction ability.Source: Per-token losses · p. 60

2D loss derivative plot

• The loss-derivative view measures how much token loss decreases when the model receives more context by differentiating context-index-average loss with respect to log(context index).It captures the loss reduction associated with seeing an additional percentage of context.Source: Per-token losses · p. 60

In-context learning score

• The in-context learning score is the average 500th-token loss minus 50th-token loss for length-512 contexts, summarizing how later-context prediction improves over training.This statistic captures the trajectory of the model’s ability to predict later tokens better than earlier ones.Source: Per-token losses · p. 60

PCA

• PCA projects concatenated random-token-per-example losses into two principal components to trace each model’s trajectory through function space during training.The analysis uses 200 snapshots for each of 12 small models and 15 snapshots for each of 6 full-scale models.Source: Per-token losses · p. 61

• The measurements track prefix-matching head activation scores and prefix-matching QK-circuit eigenvalue traces over training time.Source: Attention head measurements · p. 61
• For small models, all attention heads are shown; for full-scale models, the plots show a selected subset of 100 heads.Source: Attention head measurements · p. 61
• The weight-decay change at approximately 5 billion tokens creates a visible bend in QK-circuit trajectories, but occurs after and is unrelated to the phase change.Source: Attention head measurements · p. 61

Ablation scope

• Only pattern-preserving attention-head ablations are shown for small models, while full ablation results are deferred to the appendix.The final two rows of Figure 1 derive from per-token loss data; ablation experiments were not conducted for full-scale models.Source: Ablation attribution to phase change · p. 61

Before-and-after attribution

• Removing induction heads produces a negative dot product with the before-and-after vector, indicating that ablation reverses phase-change behavior.The before-and-after vector is the normalized change in 10,000-dimensional random-token-per-example loss between phase-change start and end; the ablation vector is the corresponding ablated-minus-unablated change.Source: Ablation attribution to phase change · p. 61

In-context learning attribution

• The analysis also attributes ablations to in-context learning using the loss difference between predicting token 500 and token 50.The score is defined as Loss(ntrain, icontext= 500) - Loss(ntrain, icontext= 50).Source: Ablation attribution to phase change · p. 61