Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

1 Introduction

• Induction heads use prefix matching to find earlier instances of the current token, then copy the following token to support sequence repetition.The paper investigates whether this mechanism underlies few-shot ICL.Source: §1 · p. 1
• Earlier work proposed induction heads as a mechanism for abstract pattern matching, but did not empirically test their contribution to ICL performance on such tasks.Related NLP work found some overlap between important heads and heads with high induction scores, without directly assessing their ICL contribution.Source: §1 · pp. 1–2
• The study tests Llama-3-8B and InternLM2-20B on abstract pattern recognition and NLP tasks using head ablations and token-level attention knockout.It selects heads by prefix-matching scores and also uses semantically unrelated labels in NLP tasks to encourage reliance on ICL.Source: §1 · p. 2
• Disabling induction heads reduces ICL performance more than equivalent random-head ablations, while blocking their induction attention pattern produces drops comparable to or worse than full head ablation.The attention knockout results support a specific role for prefix matching and copying, not only the importance of the heads themselves.Source: §1 · p. 2

2.1 Factors that influence ICL performance

• ICL performance varies with pretraining data distributions, model scale, and demonstration properties such as label space, input distribution, and input-label pairing.Prior studies also report effects of label choice and adaptation to flipped or unrelated labels in larger models.Source: §2.1 · p. 2

2.2 Mechanistic Interpretability

• Mechanistic interpretability studies internal components and circuits; mean ablation tests component importance by replacing activations with reference-distribution averages.Attention knockout instead disables selected attention weights; Figure 1 illustrates induction-head prefix matching and copying on a repeated word sequence.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.3 Induction heads & ICL

• Prior induction-head work linked the heads to a heuristic loss-reduction measure of ICL, while this study evaluates few-shot task accuracy directly.Bansal et al. found overlap between important heads and high-prefix-matching heads; this work extends that evidence with direct ablations across tasks.Source: §2.3 · p. 3

3 Background

• A multi-head attention layer processes a sequence of token embeddings through H attention heads, with each token embedding formed from its token and the vocabulary embedding matrix.The displayed formulation presents the operations of the MHA layer in this sequence-based framework.Source: §3 · p. 3
• The causal attention mask restricts each position to preceding tokens, while each head’s Query-Key circuit calculates its attention pattern.The Output-Value circuit determines the head’s independent output for the current token.Source: §3 · p. 3
• For a sequence pattern like [A] [B] ... [A], an induction head attends to the earlier [B] after matching [A], then raises the output logit for [B].These operations are called prefix matching and copying, respectively.Source: §3 · pp. 3–4

4.1 Models

• The study evaluates Llama-3-8B, with 32 layers and 32 heads each, and InternLM2-20B, with 48 layers and 48 heads each.InternLM2-20B was selected in part for its Needle-in-the-Haystack retrieval performance.Source: §4.1 · p. 4

4.2 Identifying Induction Heads

• Prefix matching scores are computed on repeated random-token sequences and averaged across five sequences to identify candidate induction heads.The method averages attention to tokens that followed the same token in earlier repeats; scores serve as a behavioral proxy, not mechanistic verification.Source: §4.2 · p. 4
• ~3% of Llama-3-8B heads score at least 0.3 on prefix matching, with some reaching 0.98.High-scoring heads are distributed across layers; Figure 2 shows Llama-3-8B scores.Source: §4.2 · p. 4, Figure 2 · p. 4

4.3 Head Ablations

• Mean ablation replaces activations of the highest-scoring heads with their mean activations, while random-head ablations matched their layer distribution as a control.The induction-head ablations target 1% and 3% of heads; random controls use the same percentages and layer distribution.Source: §4.3 · pp. 4–5

4.4 Attention Knockout

• Blocking induction-pattern attention produces performance drops comparable to or worse than full head ablation, testing the pattern's specific role in ICL.The knockout blocks attention to tokens that previously followed similar tokens.Source: §4.4 · p. 5

• Evaluations used balanced example/class sampling, averaged accuracy over three example-selection seeds, and tested five- or ten-shot binary versus 20-shot four-way tasks.Source: §5 · pp. 5–6
• Letter tasks test repetition, recursion, or centre-embedding against random sequences, while word tasks classify uncommon semantic-category pairs in binary or four-way settings.Source: §5 · p. 5, Figure 3 · p. 5
• Llama-3-8B induction-head ablation caused performance declines up to 31.6% across tasks/settings, while InternLM2-20B’s 3% induction-head ablation caused up to 36%, versus 5.8% for 3% random ablation.Source: Table 1 · p. 6, §5 · pp. 6–7
• At 3% induction-head ablation, letter-sequence performance fell near random; ten-shot binary word tasks still declined by up to 18.3%.Source: §5 · p. 6
• Few-shot prompting beat a close-to-random zero-shot baseline across all tasks for both models.Source: §5 · pp. 6–7

Experimental setup

• The study measured ICL benefit as the ten-shot versus zero-shot performance difference and tested SUL by replacing “Yes”/“No” labels with “Foo”/“Bar.”The SUL label change removes semantic label priors, requiring reliance on input-label mappings.Source: §6 · p. 7

Llama-3-8B results

• For Llama-3-8B, a 1% induction-head ablation reduced SUBJ ICL benefit by 63.8% versus 11.8% for random ablation, and reduced SST-2 benefit by 113%.The SST-2 decline put ten-shot performance below zero-shot; the comparison excludes WSC, where the model showed no ICL benefit.Source: §6 · p. 7, Figure 4 · p. 8

Ablation results

• Induction-head ablations affected ten-shot performance more than zero-shot performance across tasks except ETHOS, a pattern absent from random-head ablations.For InternLM2-20B, targeted ablations showed this pattern across the analyzed tasks.Source: §6 · pp. 7–8

SUL results

• In SUL, Llama-3-8B achieved above-random performance only on ETHOS, SST-2, and SUBJ; 1% induction-head ablation caused declines of around 20% across tasks.Increasing the ablation to 3% did not reduce performance further, consistent with the responsible heads being among those with the highest prefix scores.Source: §6 · pp. 7–8, Figure 5 · p. 8

InternLM2-20B results

• For InternLM2-20B, 1% targeted ablation reduced ICL benefit more than 1% random ablation, and increasing targeted ablation to 3% reduced benefit further except on ETHOS.The 3% random ablation appeared more damaging than targeted ablation for WSC partly because it increased zero-shot performance.Source: §6 · p. 8, Figure 4 · p. 8

SUL results

• In SUL, InternLM2-20B showed induction-ablation declines of up to 20% in all tasks except WSC, generally larger than random-ablation declines.The differences were smaller for SST-2 and WIC; above-random performance occurred in all tasks except WSC.Source: §6 · p. 8, Figure 5 · p. 8

Attention pattern analysis

• In a binary word-sequence example, the final colon token predominantly attended to label tokens, especially the target label “Foo.”This pattern was particularly evident in later-layer induction heads, but was not universal.Source: §7 · p. 9, Figure 6 · p. 9
• For the “ape” token from “grape,” the head attended to tokens that had previously followed fruit tokens, a behavior observed across token categories, heads, and samples.The authors suggest induction heads may capture the underlying pattern as well as input-label mappings.Source: §7 · p. 9, Figure 7 · p. 9

Experimental setup

• The knockout blocked attention to tokens that directly followed same-category tokens; for letter-sequence tasks, blocking was limited to labels, newlines, and colons.For other tasks, the strategy also covered word categories.Source: §7 · p. 9

Llama-3-8B results

• For Llama-3-8B, blocking induction patterns in 1% of induction heads caused declines comparable to full head ablation; at 3%, declines remained substantial and exceeded 17% in both ten-shot binary word-sequence tasks.These results support the role of the blocked attention pattern in the ablation effects.Source: §7 · p. 9

InternLM2-20B results

• For InternLM2-20B, blocking the pattern in 1% of induction heads produced declines within 4.1% of full head ablation, and blocking 3% caused still larger declines.Pattern blocking consistently reduced performance more than full ablation in this model.Source: §7 · p. 9

Conclusion

• Ablating just 1% of induction heads substantially decreased ICL, and blocking induction patterns caused declines comparable to full head ablations.The study also found that induction-head ablations generally affected few-shot performance more severely than zero-shot performance.Source: §8 · p. 9

Limitations

• Induction heads were identified using prefix-matching scores as a proxy rather than a definitive mechanistic basis, leaving their relation to grouped-query attention circuits unclear.Attention knockout was tested only on abstract-pattern recognition tasks; its findings remain unverified across NLP tasks.Source: Limitations · p. 10
• Targeted ablation and attention knockout do not test whether induction heads are the only ICL-relevant process, so additional contributing mechanisms or circuits may exist.The authors note that disabling only a targeted part of the computation cannot establish exclusivity.Source: Limitations · p. 10

A.1 Prefix Matching Scores IntermLM-20B

• Figure 8 presents prefix-matching scores for InternLM2-20B.Source: Figure 8 · p. 13

A.3 Prompt Examples

• The supplied prompt passage specifies five-shot examples for all binary tasks, while its statement about other tasks is incomplete.The passage does not give the full prompting condition for those other tasks.Source: §A.3 · p. 13

A.5.2 Zero Ablations: Full NLP ablation experiments

• The NLP ablation tables compare 1% and 3% induction-head ablations with random-head ablations for both models, including settings with semantically unrelated labels.The captions describe performance differences relative to the full model but provide no cell values.Source: Table 3 · p. 23, Table 5 · p. 25

A.5 Zero Ablations

• The zero-ablation procedure masks the output matrix of the top 1% or 3% of heads by prefix-matching score, with random ablations of the same sizes as controls.Randomly selected control heads came from any layer except the final layer.Source: §A.5 · p. 24

A.5.1 Zero Ablations: Abstract Pattern Recognition Tasks

• The abstract-pattern ablation table compares full-model performance with induction-head ablations, induction-pattern blocking, and random-head ablations for Llama-3-8B and InternLM2-20B.Its caption states that full-model zero-shot performance is approximately 50% across tasks, but supplies no ablation cell values.Source: Table 4 · p. 24

A.6.1 WordSeq1 (Binary) / A.6.2 WordSeq 1 (4-way)

• The supplied captions show attention patterns for tokens 62 and 147, each in head 22 of layer 2 of Llama-3-8B, with both labeled Example 1/5.Source: Figure 11 · p. 27, Figure 59 · p. 39

A.6.2 WordSeq 1 (4-way)

• The plots cover examples and continuations 3/5–5/5, with attention patterns shown for selected tokens in heads across layers 2, 8, 15, and 16.Source: Figure 80 · p. 49, Figure 89 · p. 54, Figure 100 · p. 59