PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying the Slash Pattern in Attention: The Role of RoPE

Yuan Cheng, Fengzhuo Zhang, Yunlong Hou, Cunxiao Du, Chao Du, Tianyu Pang, Aixin Sun, Zhuoran Yang

arXiv:2601.08297v2cs.LGcs.AIcs.CL

TL;DR

The paper investigates why LLMs develop slash attention patterns that pass information across tokens. Through empirical studies of open-source models and theoretical analysis of a shallow RoPE Transformer, it finds that near-rank-one queries and keys plus medium- and high-frequency RoPE components suffice for SDHs, including on out-of-distribution inputs.

  • Problem

    The paper asks how pretrained LLMs implement Slash-Dominant Heads (SDHs), which pass information across tokens along offset sub-diagonals.

  • Method

    The paper combines empirical studies of open-source RoPE-based LLMs with training-dynamics analysis of a shallow RoPE Transformer under cone-shaped embeddings and a slash-dominance frequency condition.

  • Results

    SDHs are intrinsic to Transformer models: almost rank-one queries and keys suppress token-wise semantic variation, while medium- and high-frequency RoPE interactions produce slash peaks and generalize to out-of-distribution inputs.

  • Takeaways & Limitations

    The results identify token-embedding geometry and RoPE frequency structure as sufficient conditions for SDH emergence.

  • Takeaways & Limitations

    The analysis focuses on pretrained LLMs with RoPE; SDHs under other positional embeddings remain future work, and large-Δ analysis is limited to two models.

Abstract

from arXiv · show

Large Language Models (LLMs) often exhibit slash attention patterns, where attention scores concentrate along the $Δ$-th sub-diagonal for some offset $Δ$. These patterns play a key role in passing information across tokens. But why do they emerge? In this paper, we demystify the emergence of these Slash-Dominant Heads (SDHs) from both empirical and theoretical perspectives. First, by analyzing open-source LLMs, we find that SDHs are intrinsic to models and generalize to out-of-distribution prompts. To explain the intrinsic emergence, we analyze the queries, keys, and Rotary Position Embedding (RoPE), which jointly determine attention scores. Our empirical analysis reveals two characteristic conditions of SDHs: (1) Queries and keys are almost rank-one, and (2) RoPE is dominated by medium- and high-frequency components. Under these conditions, queries and keys are nearly identical across tokens, and interactions between medium- and high-frequency components of RoPE give rise to SDHs. Beyond empirical evidence, we theoretically show that these conditions are sufficient to ensure the emergence of SDHs by formalizing them as our modeling assumptions. Particularly, we analyze the training dynamics of a shallow Transformer equipped with RoPE under these conditions, and prove that models trained via gradient descent exhibit SDHs. The SDHs generalize to out-of-distribution prompts.

1 Introduction

The paper asks how Slash-Dominant Heads (SDHs) arise and argues that they are intrinsic to Transformer architectures, driven mainly by low-rank queries and keys together with RoPE. Empirical studies and theoretical analysis connect these mechanisms to slash patterns and their out-of-distribution generalization.

  • Motivation: SDHs let tokens attend directly to earlier tokens at offset Δ, supporting information transfer and mechanisms such as in-context learning.The induction-head circuit is a special case with Δ = 1, while other slash patterns support long-context inference.
  • Empirical Studies: The paper studies whether SDHs depend on prompts, finding that slash dominance persists for randomly generated out-of-distribution prompts.This supports the conclusion that SDHs arise mainly from an intrinsic algorithmic mechanism of the Transformer model.
  • Empirical Studies: Queries and keys in SDHs are almost rank-one, making them nearly identical across tokens and leaving RoPE as the main source of attention-score variation.The rank-one structure is associated with token embeddings lying on a cone and WQ, WK projecting them toward its principal axis.
  • Empirical Studies: Medium- and high-frequency RoPE components interact constructively at a specific offset Δ, producing peaks in attention logits and the resulting slash pattern.Low-frequency components contribute little to forming these patterns.
  • Theoretical Analysis: Under cone-shaped token embeddings and a slash-dominance frequency condition, gradient-based training of a shallow RoPE Transformer is proven to produce SDHs.The theoretical model analyzes training on in-context-learning regression tasks and establishes generalization to out-of-distribution inputs.

2 Related Works

Related work identifies slash patterns as one of several recurring attention structures and connects them to information transfer, in-context learning, and efficient long-context inference. The paper also situates its RoPE analysis among positional-embedding and RoPE-mechanism studies.

  • RoPE: RoPE research has modified frequencies, interpolated positions, and controlled feature gaps to extend pretrained models’ context windows.Related work also investigates how RoPE mechanisms produce robust attention behavior.
  • Algorithmic Roles: The induction-head circuit uses a forwarding head and a feature-matching head to support in-context learning through slash patterns.The forwarding head attends to the immediate predecessor, while the feature-matching head identifies semantically matching prefixes.

3 Preliminaries

The preliminaries define decoder-only Transformers and causal self-attention with RoPE, from hidden states through queries, keys, values, positional rotations, masked attention scores, and weighted outputs. RoPE represents positions through rotations at a sequence of frequencies.

  • Transformer Architecture: A decoder-only Transformer maps token embeddings through stacked blocks and an output layer that produces results such as next-token predictions.Each block preserves the vector dimension while transforming the sequence representations.
  • RoPE: RoPE uses a decreasing frequency sequence in which high frequencies capture local syntactic structure and low frequencies capture long-range dependencies.The operator rotates each pair of vector components by the position multiplied by its corresponding frequency.
  • Causal Self-Attention: Causal self-attention maps hidden states H to queries, keys, and values using WQ, WK, and WV.The resulting matrices are Q = HWQ, K = HWK, and V = HWV.
  • RoPE: RoPE rotates each query and key according to its token position before attention scores are computed.At position i, the rotation uses angles i·θℓ across paired vector components.
  • Causal Self-Attention: The attention mechanism computes scores from rotated queries and keys, applies a causal mask and row-wise softmax, then uses the scores to weight values.The causal mask allows attention only from position i to positions j ≤ i.

4 Empirical Study of Slash-Dominance of Attention

Empirical studies find that slash-dominant heads (SDHs) persist across prompts and are associated with nearly rank-one pre-PE queries and keys. Their slash patterns arise from RoPE, especially the constructive interaction of medium- and high-frequency components.

  • 4.1 OOD Generalization of SDHs: SDHs identified in three open-source LLMs generalize to out-of-distribution prompts, indicating that slash dominance is intrinsic to model architecture rather than prompt semantics.The same patterns persist for Qwen2.5-7B-Instruct, Llama3-8B-Instruct, and Gemma-7B under randomly generated prompts.
  • 4.2 Approximate Low-Rankness of pre-PE Queries and Keys: Approximate rank-one structure makes query and key directions nearly invariant across tokens, so their semantic contents contribute little to differentiating attention scores.The empirical analysis attributes this structure either to large bias parameters or to token embeddings lying approximately on a cone, depending on the model.
  • 4.2 Approximate Low-Rankness of pre-PE Queries and Keys: On SDHs, pre-PE queries and keys have substantially lower effective ranks than other heads, with at least one matrix nearly rank-one.For each model, at least one of r1(Q) and r1(K) exceeds 0.9 on average over SDHs.
  • 4.4 Collaboration of Frequencies in RoPE Determines Slash Pattern: With nearly rank-one queries and keys, attention logits admit a Fourier-like decomposition whose peaks at offset ∆ are determined by RoPE frequencies.RoPE rotates two-dimensional query and key subvectors by frequency-dependent angles, while the associated amplitudes and phases are nearly token-invariant.
  • 4.4 Collaboration of Frequencies in RoPE Determines Slash Pattern: Medium- and high-frequency RoPE components are more important than low-frequency components for forming SDHs’ slash patterns.Selective frequency-removal experiments and the paper’s takeaway identify medium and high frequencies as the dominant contributors.

5 Theoretical Study of Shallow Transformers

The theoretical study formalizes two empirically motivated conditions—cone-structured token embeddings and slash-dominant RoPE frequencies—and proves that gradient descent trains shallow Transformers to exhibit SDHs and generalize OOD.

  • Theoretical Setup: The analysis studies a shallow attention-only Transformer trained by gradient descent on in-context regression tasks with RoPE.The theoretical framework includes a cone assumption for token embeddings and a quantitative slash-dominance condition on RoPE frequencies.
  • Main Result: Under the cone and frequency conditions, gradient descent provably produces SDHs and OOD-generalizable behavior.The theorem establishes these conditions as sufficient for SDH emergence in the shallow model.
  • RoPE Frequency Condition: Assumption 5.1 requires sufficiently small low frequencies and characterizes frequency interactions that produce slash dominance.Low frequencies correspond to semantically dependent content, while the condition ensures constructive concentration at a specific offset.
  • Main Result: At the end of Stage II, the loss converges with L(bθ) ≤ ϵ2.The second-layer error is larger because feature matching depends on accurate prefix recognition by the first-layer SDH.
  • Main Result: The training dynamics proceed in two stages: the first layer learns a Δ=1 slash pattern, while the second layer performs feature matching.The first-layer attention concentrates on immediately preceding tokens; the second layer attends to inputs sharing the question feature.
  • General Offsets: The proof extends from Δ=1 to arbitrary offsets when the prompt pairs labels and desired prefixes at distance Δ.The offset is determined by the data construction rather than being intrinsic to the proof.
  • OOD Generalization: For a question feature xq=vk, the learned model predicts approximately ⟨w,vk⟩ with high probability, including when the task lies outside the training distribution.This follows from second-layer attention concentration on inputs featuring the queried feature.
  • Convergence: The convergence rate is asymptotically independent of α and dc under the theorem’s dimensional assumptions.Terms involving these parameters are higher-order and negligible in the final convergence bound.

6 Extensions to SDHs with Large ∆

For large offsets, slash-dominant heads remain intrinsic and robust: they generalize to out-of-distribution prompts, exhibit low-rank pre-PE queries and keys, and depend especially on medium- and high-frequency RoPE components.

  • OOD Generalization: Large-offset SDHs persist under out-of-distribution prompts, with OOD average slash scores generally higher or comparable to in-distribution scores.The OOD prompts use independently sampled tokens from a uniform alphabet distribution.
  • OOD Generalization: Large-offset slash dominance is intrinsic to the model architecture rather than relevant to prompt semantic meaning.The conclusion extends the intrinsic-pattern finding to large offsets.
  • Approximate Low-Rankness: At least one of the pre-PE query or key matrices is close to rank-one in large-offset SDHs, with average r1 exceeding 0.88 for each model.The effective ranks of Q and K are low specifically on SDHs compared with other heads.
  • Approximate Low-Rankness: Low-rankness arises from the interaction between hidden states and query/key weights, because H, WQ, and WK have much higher effective ranks than Q and K.This contrasts the ranks of the inputs and weights with those of the resulting pre-PE queries and keys.
  • Collaboration of Frequencies in RoPE: Medium- and high-frequency RoPE components are more important than low-frequency components for large-offset SDHs.Components ℓ∈[1,42] vary more than ℓ∈[43,64], and frequency-removal experiments quantify each band’s effect on slash scores.

7 Discussion

The discussion connects approximate low-rankness and RoPE frequency effects to possible efficiency and length-generalization applications, while noting that other positional encodings remain insufficiently studied.

  • Parameter Efficiency of Queries and Keys: Constraining some WQ and WK matrices to low rank could reduce parameters and computational resources during training and inference without significant performance loss.The paper reports an initial verification using compressed query and key matrices in pretrained Qwen2.5-7B-Instruct and Llama3-8B-Instruct.
  • Effective Length Generalization: Low-frequency RoPE components could be edited or reweighted to enhance length generalization, but systematic experiments are left for future work.The motivation is that medium- and high-frequency components predominantly contribute to slash-dominant behavior, while low frequencies play a negligible role.
  • Implications for Other Kinds of PEs: The analysis focuses on RoPE models, and SDHs under Alibi, NoPE, and sinusoidal positional embeddings are left for future study.These alternatives may exhibit behavior similar to or distinct from RoPE and may differ in context dependence or length extrapolation.

8 Conclusion

The paper concludes that slash dominance is an intrinsic algorithmic effect primarily driven by token-embedding geometry and medium- to high-frequency RoPE components, with supporting empirical and theoretical analysis.

  • 8 Conclusion: Slash dominance is primarily driven by token embeddings approximately lying on a cone and by medium- to high-frequency RoPE components.The conclusion also states that the proposed slash-dominance frequency condition is satisfied by real open-source LLMs.
  • 8 Conclusion: The paper studies SDHs alongside other specialized attention heads and transformer learning-dynamics work, distinguishing its focus on slash-pattern generalization.The supplied related-work passages position SDHs among successor, content-gathering, retrieval, and other specialized heads.

B Experimental Details

The experiments examine local and extreme long-range offsets in three open-source LLMs using standardized long-context prompts, regime-specific detection thresholds, and independently sampled OOD prompts.

  • Experimental Setup: The study identifies SDHs in Gemma-7B, Llama3-8B-Instruct, and Qwen2.5-7B-Instruct across local and extreme long-range offset regimes.Local offsets satisfy ∆<5, while extreme long-range offsets satisfy 500≤∆≤5000.
  • Experimental Setup: 500 diverse LongBenchV2 prompts are truncated to 6000 tokens to produce stable, consistent cross-model attention statistics.The prefill attention matrices deterministically reflect model attention geometry without sampling noise.
  • Slash-Dominance Detection: Detection uses κ=0.1 for local offsets and κ=10^-3 for extreme long-range offsets, matching the natural attention scale of each regime.The long-range threshold corresponds to a 5× enrichment over the uniform attention scale.
  • OOD Prompts: OOD evaluation samples 500 prompts of length 6000 by drawing every token independently and uniformly from the model vocabulary.For Qwen2.5-7B-Instruct, each token has probability 1/151,642.
  • Implementation: Experiments are conducted on NVIDIA A100 GPUs with 80G memory.

C Parameter Compression Experiment

The experiment evaluates low-rank compression of query and key projections on LongBench while targeting parameter savings and preserved performance.

  • Compression method: The method estimates effective singular-value power over LongBench prompts and derives an effective rank for each query and key matrix.Bias terms are handled separately, and singular values are retained until cumulative average power reaches the chosen threshold.
  • Compression method: Matrices below the compression-rank threshold receive low-rank approximations; others remain untouched to produce the compressed model.A retained rank R_thre uses R_thre × (d_head + d_model) effective parameters instead of d_head × d_model.
  • Evaluation: The evaluation spans 21 LongBench benchmarks across 6 task types and reports task-level means, the overall mean, and effective parameter reduction.The dataset contains 4750 prompts.
  • Results: 6.51% of the original WQ and WK weight parameters are saved while Qwen2.5-7B-Instruct performance remains effectively unchanged at 37.84 versus 37.80 baseline.The result uses power threshold thre = 0.92.
  • Results: Table 1 reports maintained performance with decent parameter reductions across Single-doc QA, Multi-doc QA, Summarization, Few-shot, Synthetic, and Code tasks.The table marks performance better than baseline in bold.

D.1 More Results of Slash-dominant Heads with Small ∆

The appendix reports additional small-offset slash-dominant heads across models, using LongBench prompts and thresholded average slash scores. Results list many heads at offsets 0–2 and fewer at offsets 3–4.

  • Reporting scope: The appendix reports at most three heads per offset, ranked by average slash score, so the displayed lists are not exhaustive.More small-offset slash-dominant heads exist but are omitted for space.
  • Identification: The reported SDHs are identified with κ = 0.1 using average slash scores over 500 random LongBenchV2 prompts.The lists cover offsets Δ ∈ {0, 1, 2, 3, 4}.
  • Additional model results: For the additional model results, the listed heads include six SDHs at Δ = 2, two at Δ = 3, and one at Δ = 4.The Δ = 4 head is L0H1.

D.2.1 Results of Llama3-8B-Instruct

For Llama3-8B-Instruct, the appendix compares average attention-score matrices under LongBench V2 and i.i.d. uniform-token prompts, alongside per-token hidden-state, query, key, and InP visualizations.

  • Attention matrices: Figure 27 averages attention score matrices for Llama3-8B-Instruct over prompts from LongBench V2.
  • Attention matrices: Figure 28 repeats the attention-matrix averaging for prompts whose tokens are i.i.d. sampled uniformly from the alphabet.
  • Per-token representations: The component panel includes PCA-reduced hidden-state representations together with query, key, and inner-product visualizations for L0H4.
  • Per-token representations: Figure 29 shows hidden states, queries, keys, and InP(100, j, l) across 100 token positions and 64 indexed components.Queries and keys both have dimension 128.

D.2.2 Results of Gemma-7B

The supplied appendix passages document attention visualizations, large-offset SDH reporting, and representation statistics across the named model analyses, with selected rather than exhaustive large-offset patterns.

  • Attention matrices: The appendix compares average attention-score matrices for LongBench V2 prompts and i.i.d. uniform-token prompts in Gemma-7B.
  • Per-token representations: Figure 32 visualizes Gemma-7B hidden states, queries, keys, and InP(100, j, l) for 100-token prompts with 256-dimensional queries and keys.Hidden states are reduced to dimension 100 with PCA.
  • Large-offset SDHs: Large-offset reporting sets κ = 10^-3 and selects a subset of patterns, listing each reported head with its corresponding Δ.A single head may exhibit multiple slash patterns, but only one Δ per head is reported in the per-head statistics.
  • Statistics: The appendix reports average attention scores under LongBench and out-of-distribution prompts together with rank information for Q, K, and H.It also reports frequency-band effects and average norms of projection-related quantities.

D.4.2 Results of Llama3-8B-Instruct

The section reports Llama3-8B-Instruct attention-score matrices and examines token-embedding, query/key, RoPE, and notation details associated with SDHs.

  • Figures 34 and 35 present average attention-score matrices for LongBench V2 prompts and uniformly sampled alphabet prompts.
  • Figure 36 shows hidden states, queries, keys, and InP(100, j, l) for a 100-token Llama3-8B-Instruct prompt.The query and key dimensions are both 128.
  • Table 11 reports relative variation of token embeddings projected onto SDH dominant subspaces in Gemma-7B and Llama3-8B-Instruct.
  • Table 12 lists average norms of weight matrices, while Table 14 summarizes frequently used proof notations.
  • The appendix introduces reduced-model expressions and specifies how submatrices of E and reduced parameters relate to CSA(E).

F.1 Roadmap of the Proof

The proof roadmap tracks attention-logit updates in Stage I, characterizes their dependencies, and analyzes three phases leading to slash dominance.

  • The analysis proves the update and characterization lemmas by induction across the three phases of Stage I dynamics.
  • The proof tracks attention-logit updates because the logits determine Layer 1 attention scores.The update is ΔA_l,r(t) = A_l,r(t + 1) − A_l,r(t).
  • Attention statistics with the same offset have the same order, linking offset structure to attention-score behavior.
  • Phase I: During phase I, the immediately preceding-token logit grows faster than other logits, while other attention scores remain relatively uniform.Theorem F.10 gives a lower bound on the growth-rate gap involving η1C1K^-1N^-1.
  • Phase II: During phase II, the preceding-token score reaches constant order, and its logit continues to dominate with a larger growth-rate gap.Theorem F.15 gives a gap of at least η1C1K^-2.
  • Phase III: During phase III, the preceding-token score keeps growing but remains bounded by 1 − ϵ1.

G.1 Roadmap of the Proof

The Stage II proof roadmap analyzes feature matching through growth and convergence of attention-related statistics under conditions established after Stage I.

  • After Stage I, Layer 1 attention nearly concentrates on the immediate prefix, enabling Stage II logits to depend strongly on question and input features.
  • The proof tracks B_m(t), whose dynamics determine Layer 2 feature attention scores.
  • Phase I: Growth: During Stage II phase I, the statistic B_k(t) grows at rate Ω(ηK^-2), while mismatched-feature statistics change at smaller rates.
  • Phase II: Convergence: During phase II, the matching statistic increases and mismatched statistics decrease, producing convergence of the loss.
  • The proof controls position dependence using sufficiently small frequencies and bounds the relevant Stage II updates by induction.
  • Under x_q = v_k and a high-probability prompt event, B_k(t) increases while each mismatched B_k′(t) remains smaller in magnitude.The stated bounds are B_k(t) ∈ [(1 − Ω(ϵ1 + N^(1−α))), log(K)] and |B_k′(t)| = O((log K)/K).

H Proof of Theorem 5.3

The theorem proof combines the established Stage I and Stage II results and concludes loss convergence.

  • The proof invokes the Stage I and Stage II theorems, then reduces the remaining task to showing convergence of the loss.
  • The loss expression is evaluated using conditional expectation and the Stage II convergence theorem.
Loading 2601.08297v2…