PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Contextualization by Reinterpreting Attention Head Channels

Hakaze Cho, Haolin Yang, Zhun Sun, Naoya Inoue, Benjamin Heinzerling, Kentaro Inui

arXiv:2609.33851v1cs.LGcs.CL

TL;DR

Prior work has focused on specific words and named attention heads, leaving general patterns of contextualization less systematically described. This paper studies information differences among words and interprets attention heads as gated information channels. It reports that less-informative words tend to absorb context, transmission is selective, and singular vectors characterize head operations.

  • Problem

    Prior studies focus on specific word interactions and attention heads, often using heuristic analyses and discrete labels rather than a general account of information exchange.

    Source: §1 · p. 1

  • Method

    The paper measures word information and contextualization in language models, then models attention heads as information channels gated by singular vectors.

    Source: §1 · p. 2

  • Results

    Word information varies, low-information words show greater contextual variation, matched pairs transmit more strongly, and singular vectors characterize head operations.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Singular vectors offer an alternative to heuristic head discovery and support continuous representations of attention-head functionality.

    Source: §1 · p. 2, §7 · p. 9

    Singular-vector features need not be monosemantic, and existing automatic interpretation methods can produce spurious explanations and have limited generalization.

    Source: §7 · pp. 9–10

Abstract

from arXiv · show

Contextualization, the core operation of language modeling, transmits information across words to build sentence-specific word representations. Prior works mainly study contextualization, focusing on individual words and attention heads as a growing discrete dictionary, lacking a global view of their general behavior. Therefore, we propose a general principle: Globally, we find and estimate that different words carry different amounts of information, and less-informative words tend to absorb more contextual information. Specifically, these low-information words do not absorb contextual words uniformly, and finer-grained selectivity enables more precise routing to promote information transmission between matched words. Moreover, to find what mechanism causes such processing, we reinterpret attention heads as channels gated by their singular vectors and find that: (1) these singular vectors point to the hidden states of more informative words, allowing such words to write their information to others more strongly to act as information sources, and vice versa; and (2) these singular vectors can be viewed equally as hidden state features, enabling automated interpretation of attention heads beyond prior heuristic head discovery, also embedding heads into a continuous space rather than treating them as discrete, independent dictionary entries.

1 INTRODUCTION

The paper frames contextualization as information flowing from informative words to less-informative words, with semantic selectivity, and examines both word behavior and attention-head mechanisms.

1 INTRODUCTION

  • Prior work examined specific word interactions and attention heads, often through heuristic case studies and discrete head labels, leaving general principles difficult to state.The paper positions contextualization as a core operation in language modeling and human language understanding.Source: §1 · p. 1
  • The proposed principle is that informative words tend to source information absorbed by less-informative words, while semantic compatibility makes transmission selective.For example, “Newton” may transmit more information to “he” than to “she.”Source: §1 · p. 1, Figure 1 · p. 2
  • Across LM outputs and hidden states, word information varies, low-information words show greater contextual variation, and matched pairs transmit more strongly than unmatched pairs.The examples contrast “Newton” with “he” and “she.”Source: §1 · p. 2
  • Attention heads are modeled as channels whose singular directions jointly gate transmission, and high-information words project more strongly onto writing directions.The authors also use these directions as interpretable features and continuous representations of head functionality.Source: §1 · p. 2

2 BACKGROUND

The background describes prior word-wise studies of particular token types and head-wise studies that identify task-related attention heads through heuristic procedures.

2 BACKGROUND

  • Prior contextualization studies examined specific word types, including separator words, structural prompt tokens, last-position tokens, and tokens receiving attention at [CLS] or [MASK].The cited investigations include Chen et al. (2024a), Wang et al. (2023), Hendel et al. (2023), Abnar & Zuidema (2020), and Clark et al. (2019).Source: §2 · p. 2
  • Typical head-wise studies specify tasks or input distributions, then use ablation, circuit discovery, or heuristic inspiration to identify and name important heads.Examples include induction heads and repeated token heads.Source: §2 · p. 2
  • The resulting discrete head labels may be crude, can obscure relations among heads, and are limited by attention-head polysemy.Prior work also raises concerns about robustness of input-dependent attention scores.Source: §2 · p. 3

3 GENERAL PRINCIPLES OF CONTEXTUALIZATION

The section proposes a macroscopic account of contextualization: words differ in information, information tends to move from richer to poorer words, and semantic compatibility further shapes pairwise transmission.

3 GENERAL PRINCIPLES OF CONTEXTUALIZATION

  • Rather than enumerate word pairs and heads, the paper seeks shared regularities in information exchange as a macroscopic principle.The analogy is to statistical mechanics, which describes collective behavior through macroscopic quantities.Source: §3 · p. 3
  • Before contextualization, words carry different amounts of information and therefore exert different magnitudes of influence on model outputs.The paper contrasts the specific semantics of “Newton” with the less specific pronoun “he.”Source: §3 · p. 3
  • During contextualization, information-rich words tend to remain stable sources, while less-informative words tend to absorb context and have their representations rewritten.The section notes that abstract words such as “he” may need more specific contextual semantics before contributing meaningfully to prediction.Source: §3 · p. 3
  • Semantic compatibility further modulates individual pairs: “Newton” is more likely to transmit information to “he” than to “she.”The authors present this selectivity as linking broad regularities with case-specific interactions.Source: §3 · p. 3

4 EXPERIMENTAL SETTINGS

Experiments use a range of encoder-only and decoder-only language models, sampled words and queries or templates, and separate measurements for static information, contextualization, and selectivity.

4 EXPERIMENTAL SETTINGS

  • The experiments cover 10 encoder-only and decoder-only language models, with Llama 3.2-1B results shown by default and other model results in the appendix.The listed models include BERT-Base, XLM-RoBERTa-Large, Llama, Qwen, Granite, and OLMo variants.Source: §4 · p. 3
  • The data comprise 800 sampled words from 16 categories, 150 queries for static-information experiments, and 128 templates for contextualization experiments.For multi-token words, results may be averaged over their subword tokens.Source: §4 · p. 3
  • Figure 2 measures static information by output divergence against a blank background, contextualization through hidden-state similarities, and selective transmission by comparing background and activation-patched outputs.The three panels correspond to the paper’s measurements of word information, contextualization magnitude, and selective transmission.Source: Figure 2 · p. 4

5 PHENOMENOLOGY: OBSERVING THE CONTEXTUALIZATION

The experiments support a global pattern: words carry different amounts of static information, and less-informative words tend to absorb more context. That pattern is qualified by fine-grained selectivity: low-information words do not absorb from all words uniformly.

5.1 STATIC INFORMATION: MEASURING THE INFORMATION GAIN OF TOKENS

  • Words differ in static information, with more semantically specific or lower-frequency words generally carrying more.The paper estimates a word’s information by its effect on model outputs across queries, compared with a blank-control distribution.Source: §5.1 · pp. 4–5

5.2 INFORMATION TRANSMISSION ORIENTED BY STATIC INFORMATION GRADIENT

  • Words with higher static information show weaker contextual information absorption, while lower-information words absorb more information from other tokens.The study estimates static information from output-distribution changes across queries and measures absorption through variation in a word’s hidden states across sentence contexts.Source: §5.1 · p. 4, §5.2 · pp. 5–6

5.3 SELECTIVITY OVERRIDING INFORMATION GRADIENT

  • The selectivity experiment measures modification from A to B by comparing B’s normal output distributions with distributions after patching in B’s contextualized hidden state.The contextualization magnitude is the average JS divergence between those distributions across queries.Source: §5.3 · p. 6
  • Contextualization is selective: low-information words do not absorb uniformly, but focus on words with identified semantics from coreference or semantic and syntactic rules.The experiments report, for example, that animate nouns tend to modify “who” more than “which,” and vice versa.Source: §5.3 · p. 6

6 MECHANISM: ANALYZING THE CHANNEL OF ATTENTION HEAD

The paper models attention heads as singular-vector-gated information channels, linking word information to routing and writing. This perspective also supports automatic interpretation of head operations and a continuous representation of heads.

6.1 MULTI-HEAD ATTENTION AS GATED INFORMATION CHANNEL

  • Attention transmits information when tokens jointly activate matching q/k directions and a right singular direction of W_O W_V with sufficiently large singular values.The q/k pair gates the attention score; the o direction sets the write direction, while the projected value controls write magnitude.Source: §6 · p. 7, §6.1 · p. 7

6.2 WORD INFORMATION CAN BE SELF-SNIFFED BY ATTENTION HEADS

  • Higher-information words align more strongly with attention-head singular directions, especially k and v, suggesting stronger source-side routing and larger write magnitudes.The authors relate this alignment to their static-information measure.Source: §6.2 · p. 8, Figure 8 · p. 8

6.3 AUTOMANTIC EXPLAINATION FOR ATTENTION HEADS

  • AutoInterp applies to singular vectors as hidden-state features, allowing distinct vectors within one head to receive separate interpretations.The approach is presented as non-heuristic and able to capture multiple operations associated with a head.Source: §6.3 · p. 8
  • In Layer 11 Head 25, q2 and k2 interpretations connect holiday-related queries with seasonal-event keys; setting λ2 to zero damages the cited pumpkin attention operation.The same head also contains position-localization singular vectors.Source: §6.3 · p. 9

6.4 EMBED ATTENTION HEAD SETS INTO CONTINUOUS VECTOR SPACE

  • The authors embed each head using [q1; k1; −q1; −k1], and report a slight but statistically significant negative correlation between head distance and attention-score similarity.They suggest unaligned input spaces may explain the slight correlation and leave layer alignment for future work.Source: Figure 10 · p. 8, §6.4 · p. 9

7 CONCLUSION AND DISCUSSION

The conclusion presents contextualization as information flowing from informative words to less-informative sinks, and connects this principle to a singular-vector account of attention heads. It also identifies open research directions and limitations of automatic feature interpretation.

CONCLUSIONS

  • The paper concludes that informative words act as sources and less-informative words as sinks, with singular vectors controlling and identifying attention-head operations.The framework is presented as advancing heuristic head discovery toward automatic interpretation and discrete taxonomies toward continuous embedding.Source: §7 · p. 9

FUTURE DIRECTIONS

  • Future work includes testing whether absorbed information makes low-information words new sources, clarifying optimization pressure, and developing higher-order statistics tied to language-modeling quality.The authors also propose geometric analysis of meaningful basis directions as a route to studying primitive language-model operations.Source: §7 · p. 9

LIMITATIONS

  • Automatic interpretation relies on existing AutoInterp methods, which may produce spurious explanations and have limited generalization, while the framework is compatible with improved interpreters.The stated limitations concern the interpretation method rather than the framework.Source: §7 · pp. 9–10

A DATASET DETAILS

The appendix describes the sampled words, query set, and sentence templates used in the experiments.

A DATASET DETAILS

  • The study samples around 50 words in each of 16 categories and lists their frequencies, estimated from the first 500,000 Pile samples (Gao et al., 2020).The word list appears in Table 2 and in a structured JSON file attached to the supplementary materials.Source: A DATASET DETAILS · p. 15
  • For the information-content experiments, the authors sample 150 queries with output candidates about the selected words.The queries and candidates are listed in Table 3 and supplementary JSON.Source: A DATASET DETAILS · p. 15
  • For contextualization-magnitude experiments, the authors use 128 templates with a blank for the tested word.The templates are listed in Table 4 and supplementary JSON.Source: A DATASET DETAILS · p. 15

B.1 EXPERIMENTAL DETAILS IN §5.1

The appendix specifies procedures for estimating word information, measuring hidden-state projections, interpreting singular-vector features, and comparing attention heads.

B.1 EXPERIMENTAL DETAILS IN §5.1

  • The authors motivate their information measure as output influence relative to a blank control, rather than relying only on word frequency.They argue a never-seen token could have zero frequency yet negligible model influence, conflicting with frequency-based information estimates.Source: §B.1 · p. 15

B.2 EXPERIMENTAL DETAILS IN §6.2

  • For projection analysis, the authors sum the top three singular-vector projection magnitudes weighted by singular values, using absolute values because singular vectors are unoriented.The resulting score is used in the correlation analysis in Figure 8.Source: §B.2 · p. 15

B.3 EXPERIMENTAL DETAILS IN §6.3

  • The AutoInterp setup treats both signs of each singular vector as separate features, injects them into ReLU SAE encoder weights, and selects high-activation examples from five million Pile tokens.Explanations use 16 highest-activating examples; scoring uses 28 examples, including 8 activating and 20 randomly sampled non-activating examples.Source: §B.3 · pp. 15–16

B.4 ATTENTION HEAD DISTANCE CALCULATION IN §6.4

  • The head-distance analysis compares top-one q/k singular vectors and averages attention-score cosine similarity across 256 Pile sentences.The distance calculation removes singular-vector sign effects.Source: §B.4 · p. 16

C EXPERIMENT IN §5.2 ON GRAMMATICALLY-CORRECT INPUTS

The authors reran the §5.2 experiments after removing grammatically incorrect sentences produced by crossing word categories with sentence templates; the new results align with the original ones.

C EXPERIMENT IN §5.2 ON GRAMMATICALLY-CORRECT INPUTS

  • Removing ungrammatical sentences from the category-by-template combinations preserved the §5.2 results.The authors reran the experiments, and the results aligned with the normal ones.Source: C EXPERIMENT IN §5.2 ON… · p. 16
  • The original Cartesian product of word categories and sentence templates controlled variables by keeping filled words in the same position and context.It could also generate grammatically incorrect sentences.Source: C EXPERIMENT IN §5.2 ON… · p. 16
  • The authors removed ungrammatical sentences before rerunning the experiments.Source: C EXPERIMENT IN §5.2 ON… · p. 16

D TOKEN LENGTH CONTROLLING ANALYSIS

To test whether tokenization length confounds the §5.1 results, the authors repeated experiments using only two-token words; results across Figs. 3–5 aligned with the uncontrolled variations.

D TOKEN LENGTH CONTROLLING ANALYSIS

  • Restricting the vocabulary to two-token words was intended to rule out tokenization length as a confound in §5.1.The authors chose this restriction because it retained a substantial number of words in every category.Source: D TOKEN LENGTH CONTROLLING ANALYSIS · p. 16
  • Under the two-token-word control, the augmented results for Fig. 3 aligned with the uncontrolled variations.The results are shown in Figs. 58–65.Source: D TOKEN LENGTH CONTROLLING ANALYSIS · p. 16
  • Under the same control, the augmented results for Fig. 4 and full-layer-correlation Fig. 5 aligned with the uncontrolled variations.The results are shown in Figs. 66–73 for Fig. 4 and Figs. 74–81 for Fig. 5.Source: D TOKEN LENGTH CONTROLLING ANALYSIS · p. 16

E AUGMENTATED RESULTS

This section presents augmented results and supporting materials, including experiments across additional models, extended attention visualizations and selected head-feature interpretations. It also documents sampled questions, answer candidates, and masked-sentence templates for probing word meaning and connective behavior.

E AUGMENTATED RESULTS

  • Across most models and layers, word information gain shows a clear positive correlation with the negative absorption measurement.The reported full-layer correlations cover all 10 models.Source: E AUGMENTATED RESULTS · p. 16, Figure 48 · p. 24, Figure 49 · p. 24
  • Results for the remaining nine models align with the Fig. 3 and Fig. 4 experiments.The additional results are shown in Figures 30–38 and 39–47, respectively.Source: E AUGMENTATED RESULTS · p. 17
  • The supplement releases full attention-head feature interpretations for Llama 3.2-1B and interpretations for 20% random-sampled head features for Qwen 3-8B and Llama 3-8B.The paper directs readers to its OpenReview page for these materials.Source: E AUGMENTATED RESULTS · p. 17
  • The paper visualizes additional cases like Fig. 7 in Figs. 14 to 19, providing further examples in the cited results section without specifying model names in this passage.Source: E AUGMENTATED RESULTS · p. 17
  • For Llama 3-8B, the Fig. 8 and Fig. 11 experiments use only 20% of randomly sampled attention heads due to computational limitations.The results are presented in Figures 12 and 13.Source: Figure 12 · p. 16, E AUGMENTATED RESULTS · p. 17, Figure 13 · p. 17
  • The word-focused prompts cover semantic categories and features including temporal meaning, frequency, intensity, stance, scope, register, and evidentiality.The passages show questions paired with candidate labels, such as temporal relations and register levels.Source: E AUGMENTATED RESULTS · p. 38
  • The connective-focused prompts probe semantic relations and discourse functions, including symmetry, coordination type, temporal ordering, implicatures, and cross-category linking.They also ask about presupposition, polarity, contrastive focus, register, and direction of reasoning.Source: E AUGMENTATED RESULTS · p. 40
  • The examples are sentence contexts with a [MASK] slot, covering everyday subjects such as medication instructions, repeated experiment measurements, travel, and time management planning.Source: E AUGMENTATED RESULTS · pp. 43–44

Related papers

Loading 2609.33851v1…