Attention Heads of Large Language Models: A Survey

1 Introduction

• Attention heads are a focus for investigating LLM reasoning because Transformers perform strongly across tasks yet retain the black-box nature of deep neural networks.The survey consolidates research on specific attention-head functions and the methods used to study them.Source: §1 · p. 1
• The survey categorizes methods for investigating special attention heads as Modeling-Free or Modeling-Required, according to whether they require additional modeling.Source: §1 · pp. 2–3
• The survey summarizes evaluation tasks and benchmarks, compiles work on FFNs and Mechanical Interpretability, and discusses future research directions.Source: §1 · p. 2
• The review primarily focuses on recent research into popular LLMs such as LLaMA and GPT, rather than conclusions from earlier BERT studies that it describes as outdated.Source: §1 · p. 2
• The survey applies four human-inspired stages—Knowledge Recalling, In-Context Identification, Latent Reasoning, and Expression Preparation—as an organizing framework for LLM reasoning.It uses the stages to classify head functions and examine collaboration across stages.Source: §1 · pp. 2–3

2 Out-of-scope topics

• The survey focuses on attention heads in current mainstream decoder-only LLM architectures and excludes early Transformer studies such as work on BERT-based models.Source: §2 · p. 4
• It does not cover other Transformer components in its main overview and discovery sections, mentioning them only briefly in Additional topics.Source: §2 · p. 4

3.1 Mathematical representation of LLMs

• A decoder-only LLM is described as an embedding layer, L transformer layers, and an unembedding layer; the embedding transforms one-hot tokens into the first layer’s input.The paper omits layer normalization here because it is outside the section’s focus.Source: §3.1 · p. 4

3.2 Glossary of key terms

• The glossary groups specialized terms into conceptual frameworks for modeling LLM reasoning and empirical methods for probing and validating them.It includes circuits and residual streams as conceptual frameworks, and activation patching, ablation, and logit lens as empirical methods.Source: §3.2 · p. 6, §3.2.1 · p. 6, §3.2.2 · pp. 6–7

3.2.1 Conceptual frameworks

• Circuits model reasoning as computational-graph subgraphs, while residual streams carry accumulated information and QK and OV matrices respectively read from and write to them.Circuit nodes may represent latent features or model components, with transitions or component interactions as edges.Source: §3.2.1 · p. 6

3.2.2 Empirical analysis methods

• Activation patching substitutes activations to assess effects on final decisions, whereas ablation removes components; the logit lens maps intermediate representations to vocabulary logits for comparison.The effects considered with activation patching include direct, indirect, and total effects.Source: §3.2.2 · pp. 6–7, Figure 5 · p. 7

3.3 Existing related surveys

• The paper distinguishes its focus from earlier surveys that covered older attention computation methods or internal-structure methods but overlooked operational-mechanism findings.The cited comparison notes that current LLMs still use scaled-dot-product attention, making many derived attention forms outdated.Source: §3.3 · p. 7, Table 1 · p. 7

4.1 How does the human brain & attention head think?

• The proposed stages are Knowledge Recalling (KR), In-Context Identification (ICI), Latent Reasoning (LR), and Expression Preparation (EP), and can transition between one another.The framework is described as a graph-like process rather than a strictly linear sequence.Source: §4.1 · p. 8, Figure 6 · p. 9

4.2 Knowledge Recalling (KR)

• KR heads retrieve parametric knowledge through context-linked guesses or associations, with Memory Heads writing related content into the residual stream.Associative memories can store and retrieve knowledge during training; shallow FFNs enrich entity semantics before Memory Heads recall associated attributes.Source: §4.2 · pp. 9–10

4.3.1 Overall Structural Information Identification

• Structural ICI heads attend to token positions, rare or repeated words, and specific tokens in long texts [64,65] [66–68] [67] [22] [69,70,97].Previous and Positional Heads capture token-position relationships; Global Retrieval Heads locate specified tokens in long texts.Source: §4.3.1 · p. 11

4.3.2 Syntactic Information Identification; 4.3.3 Semantic Information Identification

• Syntactic heads label roles and merge subwords [66,71], while mover heads transfer or suppress names and letters at [END] [22,73,74]; semantic heads extract task cues and relations [61,77,78].The [END] position is the final token position in the sentence being decoded; semantic heads also summarize sentiment near [SUM] [76].Source: §4.3.2 · pp. 11–12, §4.3.3 · p. 12

4.4 Latent Reasoning (LR)

• LR synthesizes gathered information and performs logical reasoning, with QK matrices reading residual-stream information and OV matrices writing reasoning results back.The authors characterize LR as the core of problem-solving.Source: §4.4 · p. 12

4.4.1 In-context Learning

• Task Recognition uses pretrained priors to interpret demonstrations, whereas Task Learning induces novel input-output mappings; Summary Reader, Function Vector, and Induction Heads support these processes [76] [79] [64,80,81,83].Induction Heads identify patterns such as “. . . [A][B] . . . [A]” and predict [B] after the later [A].Source: §4.4.1 · p. 12

4.4.2 Effective Reasoning; 4.4.3 Task Specific Reasoning

• Studies identify reasoning-related Truthfulness and Accuracy Heads that support truthful and correct QA answers, a Consistency Head for internal consistency across differently phrased questions, and a Vulnerable Head sensitive to specific input forms [86,87] [88,89] [90] [91].In MCQA, Correct Letter Heads match answer text to option letters [62]; in sequential tasks, Iteration Heads infer the next intermediate state from the current state and input [92]. For arithmetic, Successor Heads increment ordinal numbers [93], while Inhibition Heads aggregate outputs and suppress information in syllogistic reasoning and information extraction [22,94].Source: §4.4.2 · p. 13, §4.4.3 · p. 13

4.5 Expression Preparation (EP)

• EP heads can aggregate outputs from ICI and LR heads, write them to the residual stream, and ultimately route them through the unembedding layer to vocabulary logits.The Mixed Head is described as linearly combining information from heads such as Subject, Relation, and Induction Heads.Source: §4.5 · pp. 13–14
• Amplification and Correct Heads can strengthen the correct multiple-choice answer signal near [END] until its letter has the highest probability after unembedding and softmax.The survey also describes EP heads that read contextual or reasoning information from the residual stream and write an amplified signal back.Source: §4.5 · p. 14
• Coherence Heads support output-language consistency with the user’s query, while Faithfulness Heads are associated with consistency between generated CoT and internal reasoning.The survey reports that enhancing Faithfulness Head activation can make CoT results more robust and consistent.Source: §4.5 · p. 14
• For some simple tasks, information from ICI and LR is already suitable for output, so the model can skip EP and select the highest-probability token.EP is therefore not required in every task setting.Source: §4.5 · p. 14

4.6 How do attention heads work together?

• In multiple-choice tasks, a Content Gatherer Head moves inferred answer text to [END], where a Correct Letter Head uses it to identify the matching option.The query represents the question “Are you the correct label?” while keys represent the choice letters and their associated text.Source: §4.6 · p. 15
• Across KR, ICI, LR, and EP, attention heads cooperate to align and propagate information, with the IOI example tracing a sequence from recalling relevant facts through suppressing “John” and boosting “Mary.”This stage pattern describes the majority of cases; some models return to KR or ICI in deeper layers.Source: §4.6 · pp. 15–16

5 Unveiling the discovery of attention heads

• Discovery methods are divided into Modeling-Free and Modeling-Required approaches, based on whether new models must be constructed.The survey further distinguishes Modeling-Free methods by latent-state alteration and Modeling-Required methods by whether training is needed.Source: §5 · p. 16, §5.2 · p. 18

5.1 Modeling-Free

• Modeling-Free methods alter a latent state and use Logit Lens to map intermediate results to logits or probabilities, then infer effects from the change.The survey divides these methods into Modification-Based and Replacement-Based approaches.Source: §5.1 · pp. 16–17
• Modification-Based methods retain some original state information while adding or removing a direction, enabling tests of whether a head summarizes sentiment or whether removed information is backed up or fixed.Directional Addition and Directional Subtraction are the two described forms.Source: §5.1 · p. 17
• Replacement-Based methods replace the original latent state with zero, a dataset mean, or a corrupted-prompt state to test a head’s importance or function.Naive Activation Patching can also replace a corrupted-run state with the original state to observe a shift back toward original-prompt performance.Source: §5.1 · pp. 17–18

5.2 Modeling-Required

• Training-Required methods train probes on head activations labeled by function or train a simplified transformer on a clean task dataset to investigate head behavior.The simplified model is described as reducing training and analysis costs while remaining controllable.Source: §5.2 · p. 18
• Training-Free methods use scores such as Retrieval Score and NAS to quantify head behavior, while an Information Flow Graph selects important token-transfer edges into a reasoning subgraph.A high Retrieval Score indicates a strong “Needle in a Haystack” ability; a high NAS indicates greater focus on negative tokens and a tendency toward negative signals.Source: §5.2 · pp. 18–19

6 Evaluation

• Mechanism Exploration Evaluation tests the working mechanisms of specific heads, whereas Common Evaluation tests whether enhancing or suppressing head functions improves overall LLM performance.The survey presents these as the two evaluation-goal categories.Source: §6 · p. 19

6.1 Mechanism exploration evaluation

• Mechanism-exploration datasets standardize irrelevant task features and simplify answers from paragraphs to tokens, with sentiment-head studies using positive- and negative-sentiment activation differences to identify candidates.ToyMovieReview and ToyMoodStory use prompt templates with adjective and verb substitutions.Source: §6.1 · p. 19, Figure 11 · p. 20

6.2 Common evaluation

• Common benchmarks assess changes in response accuracy and usefulness across five areas: knowledge reasoning, logic reasoning, sentiment analysis, long-context retrieval, and text comprehension.The survey groups these benchmarks by evaluation focus.Source: §6.2 · pp. 19–20

7 Additional topics

• Additional topics cover FFN interpretability and Machine Psychology, including work that investigates LLM mechanisms without identifying new special attention heads.The section is presented as another perspective on LLM interpretability.Source: §7 · p. 20

7.1 FFN interpretability

• FFNs are described as storing knowledge sparsely, with lower layers associated with syntactic and semantic information and higher layers with more concrete facts.Prior work likens GPT FFN neurons to key-value pairs that retrieve corresponding knowledge.Source: §7.1 · pp. 20–21
• FFNs complement attention heads by making local updates to single representations while heads aggregate global information, supporting breadth and depth in textual processing.The survey also describes their collaboration in enriching subject representations and extracting related attributes for factual retrieval during KR.Source: §7.1 · p. 21

7.2 Machine Psychology

• Machine Psychology applies psychological paradigms and behavioral analysis to LLMs, with proposed approaches examining abilities such as heuristics, biases, and learning from environmental interaction.The survey presents this as a perspective for analyzing LLM behavior and potentially informing intelligent-AI development.Source: §7.2 · p. 21

8.1 Limitations in existing studies

• Reasoning circuits lack validation across other tasks, and special-head mechanisms remain insufficiently studied across different model series.The discussion cites task-specific circuit studies and discoveries limited to a few LLMs or toy models.Source: §8.1 · pp. 21–22
• Most studies examine individual heads, leaving coordinated functioning across attention heads without a comprehensive framework.Only a few researchers study collaboration among multiple heads.Source: §8.1 · p. 22
• Experimental validation of circuit hypotheses cannot establish their theoretical soundness or determine whether observed mechanisms are coincidental.The discussion describes hypotheses based on observed phenomena and validated through experiments.Source: §8.1 · p. 22

8.2 Future directions and challenges

• Future work could investigate whether special attention-head functions appear in open-ended question answering, math problems, and tool-using tasks.These are proposed examples of more complex tasks.Source: §8.2 · p. 22
• Future work could examine attention-head mechanisms under prompt changes and propose ways to mitigate prompt sensitivity.The discussion notes that slight prompt changes can potentially produce opposite responses.Source: §8.2 · p. 22
• New experiments could test whether a particular mechanism is indivisible or universally applicable.Source: §8.2 · p. 22
• A comprehensive interpretability framework should cover independent and collaborative mechanisms of most attention heads and other components.Source: §8.2 · p. 22
• The discussion proposes using Machine Psychology to build an anthropomorphic account of LLM mechanisms, understand gaps with human cognition, and guide targeted improvements.Source: §8.2 · p. 22

• Current attention-head interpretability research is relatively scattered and primarily studies individual heads.Source: §9 · p. 22
• The paper's categorization of head functions through human cognitive behavior may not be perfectly orthogonal because a rigorous overall framework is lacking.Source: §9 · p. 22
• The proposed stages may overlap rather than cleanly separate attention-head functions.Source: §9 · p. 22

Materials availability

• No new unique reagents were generated.Source: Materials availability · p. 23

Data and code availability

• The reference list is available on GitHub.Source: Data and code availability · p. 23