Demystifying the Slash Pattern in Attention: The Role of RoPE

Motivation

• Slash patterns support in-context learning through induction heads [Olsson et al., 2022] and have been used to accelerate long-context inference [Xu et al., 2025; Zhao et al., 2025; Lai et al., 2025; Li et al., 2025].An induction head’s forwarding component attends primarily to the immediate predecessor, forwarding prefix features to the current token.Source: §1 · p. 2

Role of Queries and Keys

• Almost rank-one pre-PE queries and keys point in the same direction across tokens, and arise because token embeddings lie on a cone that WQ and WK project onto its main axis.Consequently, semantic content contributes little to differentiating attention scores, leaving RoPE to drive most token-to-token variation.Source: §1 · p. 3

Role of RoPE

• Medium- and high-frequency RoPE components dominate slash formation, producing attention-logit peaks when i−j=Δ, while low-frequency components contribute little.With nearly token-invariant amplitudes and phases from rank-one queries and keys, the frequency components determine the slash pattern.Source: §1 · p. 4

Empirical Studies

• Empirical studies identify SDHs as an intrinsic architectural effect: almost rank-one pre-PE queries and keys suppress semantic variation, while medium- and high-frequency RoPE components interact constructively at offset Δ.The paper also reports that SDHs persist on out-of-distribution prompts, indicating they are largely independent of specific prompt content.Source: §1 · pp. 3–4

Theoretical Analysis

• A shallow RoPE-equipped attention-only Transformer trained on ICL regression tasks exhibits SDHs under cone-shaped token embeddings and a slash-dominance frequency condition on RoPE.The analysis characterizes the full training dynamics and proves that models trained under these conditions generalize effectively to out-of-distribution input.Source: §1 · p. 4

Rotary Position Embeddings (RoPE)

• RoPE applies multiplicative rotations to queries and keys so attention depends implicitly on relative positions, and it became the default positional embedding in many LLMs.RoPE was proposed by Su et al. (2024); the passage names Qwen Team (2025), Yang et al. (2024), Gemma Team (2024), and Grattafiori et al. (2024) as examples of its adoption.Source: §2 · p. 4
• Context-extension methods modify RoPE base frequencies, interpolate position indexes and frequencies, or control RoPE feature gaps.These approaches are associated with Xiong et al. (2024), Roziere et al. (2023), Xiong et al. (2025), Chen et al. (2023), Peng et al. (2024), Ding et al. (2024), Zhong et al. (2025), and Wang et al. (2024b).Source: §2 · p. 5
• Barbero et al. (2025) found that Gemma-7B uses RoPE high frequencies for robust positional attention and low frequencies for semantic information, and proposed p-RoPE.Xiong et al. (2025) proposed training-free DoPE, which suppresses noisy positional frequency components via truncated matrix entropy to improve long-context extrapolation.Source: §2 · p. 5
• Jin et al. (2025) found RoPE-induced massive query and key values across layers that support contextual knowledge understanding, with perturbations degrading context-dependent task performance.Source: §2 · p. 5

Transformer Architecture

• Decoder-only Transformers map token embeddings through identical blocks before a final output layer, with each block preserving the hidden-vector dimension.The first block receives token embeddings; subsequent blocks receive the preceding block’s outputs.Source: §3 · p. 6

Rotary Position Embedding (RoPE)

• RoPE encodes token positions by rotating component pairs at decreasing frequencies, with high frequencies associated with local syntax and low frequencies with long-range dependencies.The paper considers a general frequency sequence rather than only the classic choice θℓ = 10000^-2ℓ/d.Source: §3 · p. 6

Causal Self-Attention with RoPE

• CSA projects hidden states into queries, keys, and values, applies RoPE to queries and keys, computes causally masked softmax attention scores, and uses them to weight values.The causal mask excludes positions j greater than i when computing the score at position i.Source: §3 · p. 7

Prompt Distributions

• The later analysis considers prompts drawn from the pretraining distribution D or a specified distribution D′ that may include out-of-distribution inputs, and studies expected CSA statistics under either distribution.The paper identifies Sections 4 to 6 as analyzing CSA and attention-score properties.Source: §3 · p. 7

4 Empirical Study of Slash-Dominance of Attention

• All three models exhibit SDHs at offsets Δ < 5, reflecting short-range attention dependencies.An SDH is defined by average attention on a sub-diagonal at lag Δ exceeding threshold κ; the study identifies heads using κ = 0.1 and 6,000-token contexts.Source: §4 · p. 8

4.1 OOD Generalization of SDHs

• The same identified SDHs retain their slash pattern on random-token OOD prompts, showing that slash-dominance is intrinsic to the models rather than prompt semantics.For most identified heads, OOD average slash scores are at least half their in-distribution scores and remain at least 0.05, so they remain valid at κ = 0.05.Source: Figure 7 · p. 9, §4.1 · pp. 9–10, Figure 6 · p. 9

4.2 Approximate Low-Rankness of pre-PE Queries and Keys

• On SDHs, pre-RoPE queries and keys have low effective ranks, and at least one of the two is nearly rank-one.For every model, at least one of the query or key matrices has average r1 above 0.9 on SDHs; both matrices have higher effective ranks on other heads.Source: Figure 9 · p. 11, §4.2 · pp. 11–12

4.3 How to Get Approximately Rank-One Queries and Keys?

• Query/key low-rankness arises from interactions between hidden states and weights; in the zeroth layer, large Qwen2.5 biases or cone-aligned Gemma and Llama3 embeddings explain it.Hidden states and weight matrices have higher effective ranks than resulting queries and keys. For Gemma-7B and Llama3-8B-Instruct, token embeddings align around dominant weight subspaces; for Qwen2.5-7B-Instruct, bias parameters are implicated.Source: §4.3 · pp. 12–14

4.4 Collaboration of Frequencies in RoPE Determines Slash Pattern

• The RoPE analysis decomposes attention logits by frequency, expressing their contributions in a Fourier-like form.RoPE applies block-diagonal 2 × 2 rotations, rotating query sub-vectors by frequency-dependent angles.Source: §4.4 · p. 15
• Query and key directions and magnitudes are nearly invariant across tokens, so attention logits depend only on relative token distance, producing the slash pattern.The reported relative variation in query and key norms is at most 0.12.Source: §4.4 · p. 15, Figure 14 · p. 15
• Removing high-frequency RoPE components lowers average slash scores most, medium-frequency removal causes a milder decrease, and low-frequency removal causes the least drop across three models.The analysis removes six frequencies from each band and compares the modified average slash score with the original score.Source: §4.4 · p. 16, Figure 16 · p. 16
• The attention logits’ Fourier-like decomposition and frequency analysis indicate that high- and medium-frequency RoPE components matter more for SDHs than low-frequency components.The qualitative analysis reports greater variation for high- and medium-frequency components than for low-frequency components.Source: §4.4 · pp. 15–17, Figure 15 · p. 16

5.1 Data Model

• The ICL regression model samples linear tasks and prompts with demonstrations, using a feature-learning distribution over K orthogonal, normalized features sampled uniformly.Each prompt contains input-label demonstrations for one task followed by a question input.Source: §5.1 · pp. 17–18

5.2 Slash-Dominance Frequency Condition

• Assumption 5.1 requires low RoPE frequencies on semantic dimensions and sinusoidal components on the cone component that approximate a pulse.The low-frequency bound is θs ≤ O(N^-α) for α ≥ 2; the pulse approximation uses the Kronecker delta.Source: §5.2 · p. 18
• Pretrained-model active frequencies satisfy Assumption 5.1, while zeroing medium- or high-frequency components can break it and substantially reduce slash scores.Zeroing low frequencies does not affect the condition’s validity.Source: §5.2 · p. 19

5.3 Network Architecture

• The analysis uses a reduced two-layer, single-head disentangled transformer, equivalent to a standard decoder-based attention-only transformer, with first-layer queries nearly rank-one along the cone axis.The second layer focuses on semantically dependent input content.Source: §5.3 · p. 20

5.4 Training Settings

• The reduced transformer is trained on squared loss using two-stage gradient descent, motivated by the observation that different layers converge in distinct phases.Source: §5.4 · p. 21

5.5 Main Theoretical Result

• Under Assumption 5.1, balanced features, and the stated size conditions, gradient descent learns a Δ=1 first-layer SDH, feature-matches in layer two, and drives loss to ϵ2 in polynomial time.The first-layer attention concentrates on the immediately preceding token; the second layer concentrates on tokens matching the question feature.Source: §5.5 · p. 22
• The analysis extends to arbitrary offsets when each label is paired with its prefix at distance Δ, and predicts correctly on linear tasks that may lie outside the training distribution’s support.For an OOD task with question feature vk, the prediction approximates its true response ⟨w, vk⟩.Source: §5.5 · p. 22
• For both layers, sufficiently concentrated attention places parameters in sub-global-optimal regions, and softmax saturation lets gradient descent reach those regions with sufficiently large learning rates in essentially one step.The gradients naturally guide parameters toward these solutions without requiring a carefully tuned descent direction or precise learning-rate scheduling.Source: §5.5 · p. 23
• Although interactions across many layers make gradient-descent dynamics in deeper LLMs less predictable, the analysis still expects slash-dominant heads to emerge.Source: §5.5 · p. 23

6 Extensions to SDHs with Large ∆

• Large-∆ SDHs are identified using 500≤∆≤5000 and κ=10^-3 in Llama3-8B-Instruct and Qwen2.5-7B-Instruct.Lowering κ directly can spuriously classify offsets ∆<10 as slash-dominant because of nearby-token attention; Gemma-7B lacks sufficient context length, and a refined definition is left to future work.Source: §6 · pp. 23–24
• Average slash scores under OOD prompts are generally higher or at least comparable to in-distribution scores, and the same slash patterns persist, indicating that slash dominance is intrinsic rather than semantic.The OOD evaluation uses prompts whose tokens are independently sampled from a uniform alphabet.Source: Figure 18 · p. 23, Figure 19 · p. 24, §6 · p. 24

Approximate Low-Rankness of pre-PE Queries and Keys.

• For large-∆ SDHs, pre-RoPE Q and K have low effective ranks, and at least one of them is close to rank-one.For each model, at least one of r1(Q) and r1(K) exceeds 0.88 on average over SDHs; effective ranks are low on SDHs and higher on other heads.Source: Approximate Low-Rankness of pre-PE… · p. 25, Figure 21 · p. 25

How to Get Approximately Rank-One Queries and Keys?

• Higher-rank hidden states and WQ/WK yield low-rank Q/K through their interaction; Gemma and Llama token embeddings align with low-rank weight subspaces.The per-token rank measures closely match matrix-level measures in Llama3-8B-Instruct but differ in Qwen2.5-7B-Instruct.Source: How to Get Approximately Rank-One… · p. 25, Figure 22 · p. 26

Collaboration of Frequencies in RoPE Determines Slash Pattern.

• High- and medium-frequency RoPE components matter more than low frequencies for large-∆ SDHs; relative norm variation is at most 0.093, supporting nearly token-invariant queries, keys, amplitudes, and phases.Removing high frequencies decreases average slash scores in both models; removing medium frequencies causes a mild change or none, while removing low frequencies causes the least drop.Source: Collaboration of Frequencies in RoPE… · pp. 25–26

Parameter Efficiency of Queries and Keys

• Low-rank constraints on query and key matrices could reduce parameters and computation while preserving accuracy in pretrained models.Compression of Qwen2.5-7B-Instruct and Llama3-8B-Instruct yielded notable parameter reduction without significant accuracy loss; enforcing low-rank constraints during training remains future work.Source: §7 · p. 27

Effective Length Generalization

• Reweighting or editing low-frequency RoPE components may improve length generalization, since medium- and high-frequency components drive slash-dominant behavior while low frequencies contribute negligibly.The paper leaves systematic experimental exploration of this application to future work.Source: §7 · p. 27

Implications for Other Kinds of PEs

• Other positional encodings may produce different SDH behavior: Alibi may support out-of-distribution generalization, whereas NoPE and sinusoidal embeddings may make SDHs more context dependent.Alibi adds relative bias to query–key inner products and is designed for length extrapolation; studying SDHs under these alternatives remains future work.Source: §7 · p. 28

8 Conclusion

• Cone-like token-embedding geometry and medium- to high-frequency RoPE components drive intrinsic slash-dominance, with sufficient emergence conditions supported by theory and open-source LLM evidence.The analysis focuses on pretrained LLMs with RoPE and leaves other positional encodings for future work.Source: §8 · p. 28

A Additional Related Work

• Unlike other specialized heads, SDHs are shown to generalize to out-of-distribution cases.The passage contrasts SDHs with successor, content-gathering, and retrieval heads.Source: A Additional Related Work · p. 34
• Attention-pattern insights support efficiency methods: Xu et al. (2025) use antidiagonal sums to guide sparse inference, while Li et al. (2025) enable million-token-scale training.Li et al. leverage heads attending along fixed RoPE-induced relative offsets in distributed dynamic sparse attention.Source: A Additional Related Work · p. 34
• Training-dynamics work progressed from in-context convergence for softmax and linear attention to multi-head Transformer studies of multi-task and nonlinear tasks.The cited works include Huang et al. (2024), Zhang et al. (2024a), Chen et al. (2024a), and Yang et al. (2024).Source: A Additional Related Work · p. 34
• Further studies examine Transformer learning under causal structure and in chain-of-thought reasoning settings.The passage cites Nichani et al. (2024), Chen et al. (2024b), Edelman et al. (2024), Cheng et al. (2025), Wen et al. (2025), Kim and Suzuki (2025), Yang et al. (2025), and Huang et al. (2025a,b).Source: A Additional Related Work · pp. 34–35
• Vision Transformer studies analyze specialized positional attention and shallow-model training in supervised classification and in-context learning.Li et al. (2023) extend the shallow-ViT study under strict network-initialization assumptions.Source: A Additional Related Work · p. 35

B Experimental Details

• Experiments examine Qwen2.5-7B-Instruct, Llama3-8B-Instruct, and Gemma-7B at local offsets (Δ < 5) and extreme long-range offsets (500 ≤ Δ ≤ 5000), excluding confounded intermediate offsets.Intermediate offsets (5 ≤ Δ < 500) mix semantic dependencies with positional alignment, preventing clean isolation of the slash pattern.Source: B Experimental Details · p. 35
• 500 LongBenchV2 prompts per model, truncated to 6000 tokens, yield over 5 × 10^5 valid attention-pair samples after excluding positions 1–4.Prefill attention matrices are deterministic, and each prompt contributes at least 1000 valid pairs for the studied offset ranges.Source: B Experimental Details · p. 35
• Local-offset detection uses κ = 0.1, while long-range detection uses κ = 10^-3; no large-Δ SDHs are found in Gemma-7B.At context length 5000, the long-range threshold is 5× the uniform baseline of 2 × 10^-4; large-Δ results are reported only for Llama3 and Qwen2.5.Source: B Experimental Details · p. 35
• For out-of-distribution experiments, researchers generate 500 prompts of 6000 independently sampled tokens, drawing uniformly from each model’s vocabulary.For Qwen2.5-7B-Instruct, each token has sampling probability 1/151,642.Source: B Experimental Details · p. 35

C Parameter Compression Experiment

• The compression experiment is motivated by low-rank queries and keys and estimates singular-value power across heterogeneous LongBench v2 prompts.It extends the power computation to sequence hidden states and averages power across prompts to determine effective ranks.Source: C Parameter Compression Experiment · p. 36
• The method sorts singular values by average power, retains the bias term, and sets effective rank where cumulative power reaches the chosen threshold.For Qwen2.5-7B-Instruct, the bias is represented by σ0 = 1 and v0 = b; models without bias use v0 = 0.Source: C Parameter Compression Experiment · p. 36
• Only WQ and WK matrices below the compression rank limit receive low-rank approximations; matrices above it remain unchanged.The resulting compressed matrices retain the singular values selected by the effective-rank threshold.Source: C Parameter Compression Experiment · p. 36
• Evaluation covers 21 LongBench benchmarks across six task types and 4,750 prompts, reporting task-type and overall means plus effective-parameter reduction.For a truncated matrix, effective parameter count is retained rank multiplied by dhead + dmodel.Source: C Parameter Compression Experiment · p. 36
• Qwen2.5-7B-Instruct retains overall performance at 37.84 vs. 37.80 for the baseline while saving 6.51% of its WQ and WK parameters.The reported setting uses a power threshold of 0.92.Source: C Parameter Compression Experiment · p. 36, Table 1 · p. 37

D More results of Slash-Dominant Heads

• The additional SDH results focus on small Δ, with E[Si,i−∆] denoting average slash score in the tables.The section abbreviates Gemma-7B, Llama3-8B-Instruct, and Qwen2.5-7B-Instruct as Gemma, Llama3, and Qwen2.5.Source: D More results of Slash-Dominant Heads · p. 37

D.1 More Results of Slash-dominant Heads with Small ∆

• For each ∆, the appendix lists at most three heads with average slash scores above 0.1; additional heads with small ∆ are omitted.The selection is a space-limited subset.Source: §D.1 · p. 37

D.1.1 Full List of SDHs with Small ∆

• The full lists cover ∆=0–4 and identify SDHs using (6) with κ=0.1, averaging slash scores over 500 random LongBenchV2 prompts.The lists are reported separately for each model.Source: §D.1.1 · p. 37
• The reported SDHs decrease from six at ∆=2 (L0H1, L1H7, L2H2, L2H9, L3H9, L13H3), to two at ∆=3 (L0H1, L2H9), to one, L0H1, at ∆=4.Source: §D.1.1 · p. 38
• For Llama3, ∆=2 yielded seven found SDHs (L0H0, L0H2, L1H21, L7H2, L9H11, L14H8, L14H26); at ∆=3 and ∆=4, the found SDH was L0H0 in each case.Source: §D.1.1 · p. 38
• For Qwen2.5, the found SDHs decrease from eight at ∆=3 (L0H1, L0H5, L1H0, L1H3, L1H17, L3H9, L6H15, L13H10) to three at ∆=4 (L1H0, L1H17, L6H15).Source: §D.1.1 · p. 39

D.1.2 Attention Scores and Ranks of Q, K, and H

• Per ∆ and model, the appendix ranks SDHs by average slash score and reports at most the top three heads, with tables covering attention scores and Q, K, and H ranks.Table 2 specifies er1 and eR0.95 for SDHs in layer 0; Table 3 specifies LongBench and OOD prompt scores and Q, K, and H ranks.Source: §D.1.2 · p. 39, Table 2 · p. 39, Table 3 · p. 40
• Additional tables report WQ and WK projection-matrix ranks and quantify each RoPE frequency band’s effect by removing that band and reporting the average slash score.Source: Table 4 · p. 41, Table 5 · p. 42

• Figure 27 shows average attention-score matrices for Llama3-8B-Instruct using LongBench V2 prompts.Source: Figure 27 · p. 43
• Figure 28 shows average attention-score matrices for Llama3-8B-Instruct using prompts with tokens sampled i.i.d. uniformly from an alphabet.Source: Figure 28 · p. 43
• Figure 29 visualizes hidden states, queries, keys, and InP(100, j, l) for a 100-token prompt, with query and key dimensions of 128.Source: Figure 29 · p. 44

D.2.2 Results of Gemma-7B

• Gemma-7B figures show average attention matrices for LongBench V2 and uniformly sampled prompts, alongside PCA-reduced hidden states, queries, keys, and InP values.The hidden-state, query, key, and InP example uses a 100-token prompt; query and key dimensions are 256.Source: Figure 30 · p. 44, Figure 31 · p. 45, Figure 32 · p. 45

D.3 More Results of Slash-dominant Heads with Large ∆; D.3.1 Full List of SDHs with Large ∆

• With κ=10^-3, the appendix selects rather than exhaustively identifies large-∆ SDHs and lists Llama3 heads, including L0H29, L0H30, L1H16, and L1H18.The Llama3 lists include multiple ∆ values for some heads.Source: §D.3 · p. 45, §D.3.1 · p. 46

D.3.2 Attention Scores and Ranks of Q, K, and H

• Per-head tables cover average attention scores and Q, K, and H ranks, projection-matrix rank measures, and the effects of removing RoPE frequency bands.The section ranks each head’s ∆ values by average slash score and reports the largest values.Source: §D.3.2 · p. 48, Table 6 · p. 49, Table 7 · p. 50

D.4.1 Results of Qwen2.5

• The Qwen2.5 appendix includes a 5,000-token example displaying hidden states, queries, keys, and InP(5000, j, l).Query and key dimensions are 128.Source: Figure 33 · p. 52

D.4.2 Results of Llama3-8B-Instruct

• The Llama3 appendix shows average attention matrices for LongBench V2 and uniformly sampled prompts, plus hidden states, queries, keys, and InP for a 100-token example.Query and key dimensions are 128.Source: Figure 34 · p. 53, Figure 35 · p. 53, Figure 36 · p. 54

D.5 Tables related to token embeddings in the 0-th layer

• The token-embedding tables cover relative variation in dominant subspaces for Gemma-7B and Llama3, and projection and bias norms for Qwen2.5.The Qwen2.5 passage specifies norms over the alphabet for the 0-th-layer SDH.Source: Table 11 · p. 54, Table 12 · p. 54, §D.5 · p. 54

E Notations in Theory Sections and Expression of Reduced Model

• The theory appendix defines asymptotic notation and presents a reduced-model expression, with Layer 1’s key and value weights fixed during training.The reduction is visualized for Layers 1 and 2.Source: E Notations in Theory Sections and… · pp. 55–56, Figure 37 · p. 56

F.1 Roadmap of the Proof

• Across Stage I, the immediately preceding-token logit keeps growing while other logits typically oscillate more slowly; its growth first dominates, then accelerates as its attention score reaches constant order.In phase I, attention scores are relatively uniform and the growth-rate gap is ≳η1C1K^-1N^-1; in phase II, the preceding-token score is Ω(1) and the gap is ≳η1C1K^-2.Source: §F.1 · p. 57
• In phase III, the preceding-token logit continues growing at a smaller rate, while its attention score keeps growing but remains at most 1 − ϵ1.The roadmap states that the score differs in order across the three phases.Source: §F.1 · pp. 57–58

F.2 Stage I: Preliminary Development

• Stage I analysis tracks logit updates because logits determine attention scores, then simplifies the update terms ai,j by characterizing them in cases.The proof roadmap describes induction across three phases, while the preliminary development derives gradient and logit-update formulas under Assumption 5.1.Source: §F.1 · p. 56, §F.2 · pp. 59–60

F.3 Stage I: Auxiliary Lemmas

• Auxiliary lemmas characterize how low-frequency RoPE components influence token embeddings, with bounds depending on whether odd-position tokens share a feature or differ.These results help simplify I(i) and control its bounds.Source: §F.3 · p. 62, §3 · p. 63

3. If j is even, then Ex,y

• For even positions, the token-embedding subvector corresponding to x is zero, yielding zero inner product with the corresponding question subvector.The proof uses this case to conclude the even-position result of Theorem F.4.Source: §3 · p. 63
• Through Stage I, S(2)_k = Θ(1/K) for all i ∈ [N] and m ∈ [K].Corollary F.7 derives this order from the concentration results.Source: §3 · pp. 64–65

F.4 Stage I: Phase I

• During Phase I, the previous-token attention-logit gap increases monotonically and remains between 0 and log N for every position.Source: §F.4 · p. 65

F.4.1 Technical Lemmas

• Technical lemmas bound the previous-token increment advantage below by η1C1/(KN) and the spread among other increments above by η1C1 log(N)/(NK^2).They also give S(1)_i,i−1(t) = Ω(1/i) and characterize the orders of a_i,j(t) by position parity.Source: §F.4.1 · pp. 66–67

F.4.2 End of Phase I

• Lemma F.12 establishes an endpoint for Phase I at T(1)+1 and bounds the next-step quantity involving the previous-token attention weight.Source: §F.4.2 · pp. 69–70

F.5 Stage I: Phase II

• Phase II is defined as the rapid-growth phase for learning relationships between each position and its previous tokens, with an induction hypothesis asserted throughout.Source: §F.5 · p. 70

F.5.1 Technical Lemmas

• The Phase II lemmas bound the predecessor attention score below, other even-query scores negatively, and remaining odd-query scores near zero.They also establish a positive increment gap for the predecessor entry over competing entries.Source: §F.5.1 · p. 71

F.5.2 End of Phase II

• Phase II ends with a duration bound of O(1/K), completing the proof of Induction Hypothesis F.2.The proof also states a bound on attention-related quantities, but the supplied expression is incomplete.Source: §F.5 · p. 71, §F.5.2 · p. 72

F.6 Stage I: Phase III

• Phase III is defined as the convergence phase for ϵ1 = O(N^-2), with the predecessor-versus-competitor attention-score gap increasing monotonically.The induction hypothesis states that this gap is monotonically increasing for each token.Source: §F.6 · p. 73

F.6.1 Technical Lemmas

• For even i=2n, the predecessor score a_{i,i−1}(t) ≳ ε₁/(KN²), while competing odd j=2m−1 and even j=2m scores (m≤n) satisfy −1/(K²N²) ≲ a_{i,j}(t) ≲ −ε₁/(KN²).For odd i=2n−1, even j=2m (m≤n−1) and odd j=2m−1 (m≤n) satisfy |a_{i,j}(t)| ≲ 1/(K³N²). Also, ΔA_{l,l−1}(t)−max_{r≤l,r≠l−1} ΔA_{l,r}(t) ≳ K⁻¹η₁C₁ε₁.Source: §F.6.1 · pp. 74–75

F.6.2 End of Phase III

• Lemma F.22 proves the end-of-Phase-III claims using the Phase III induction hypothesis and increment bounds.The supplied proof fragments do not include the complete claims or their final quantitative bounds.Source: §F.6.2 · p. 75

F.7 Proof of Stage I of Theorem 5.3

• The Stage I proof uses Theorems F.12, F.17, and F.22.The supplied passage ends before stating the full resulting bound.Source: §F.7 · p. 76

G Proof of Stage II of Theorem 5.3

• In the Stage II proof, the passage fixes fWQ and Si,j and omits time indices and writes L(eθ) as L when unambiguous.The fixed quantities and notation simplification are the only Stage II details supplied here.Source: G Proof of Stage II of Theorem 5.3 · p. 76

G.1 Roadmap of the Proof

• The proof tracks feature-conditioned logits Bm(t) and their updates bm(t), which determine Layer 2 attention scores.After Stage I, Layer 1 attention nearly concentrates on the immediate prefix; for fixed query feature, logits and updates are nearly common across positions sharing a feature.Source: §G.1 · p. 76
• In Phase II, the query aligns with tokens sharing its feature, Bk(t) keeps increasing, and other-feature logits decrease, supporting loss convergence.The roadmap describes this as the convergence phase following rapid growth of the self-attention parameters in Phase I.Source: §G.1 · p. 76

G.2 Stage II: Preliminary Development

• Stage II gradient analysis identifies ui(t) as important for attention scores and characterizes feature-specific logit updates Bi,m(t).The position dependence in Bi,m(t) and its update is negligible enough to omit the position subscript when unambiguous.Source: §G.2 · pp. 77–78

2. If i is odd, then ui(t) is always relatively small and can be controlled

• With sufficiently small low frequencies, position dependence in the increments of ui(t) is controlled by a minor term; odd-position logits remain relatively small.The passage states that odd-position ui(t) can be controlled and concludes the result using the Stage I prefix-attention condition and the definition of Bm(t).Source: §2 · p. 80

G.3 Stage II: Auxiliary Lemmas

• Under pk = Θ(1/K) and K^3 ≪ N, the auxiliary high-probability event bounds each feature count |Vk| to Θ(N/K).The count bounds apply to prompts in E∗P.Source: §G.3 · p. 80

G.4 Stage II: Phase I

• Bk(t) is monotonically increasing and lies in [(1 − Ω(ϵ1 + N^(1−α))), log(K)]; for any k′ ≠ k, |Bk′(t)| = O(Bk(t)/K) = O((log K)/K).Source: §G.4 · p. 81

G.4.1 Technical Lemmas

• The Phase I technical lemmas are established by induction under Induction Hypothesis G.1 and condition on the query feature and a high-probability prompt event.The roadmap says the phase analyses formulate induction hypotheses and derive lemmas governing attention scores.Source: §G.4.1 · p. 81

1. For the attention scores related to feature k,

• The technical analysis of attention scores for the matching feature and the nonmatching features culminates in Theorem G.4.The following lemma controls the order of feature-specific updates, including |bk′(t)| ≤ O(bk(t)/K) for k′ ≠ k.Source: §1 · pp. 81–83

G.4.2 End of Phase I

• At the end of Phase I, the proof establishes Claim (2), using |V_k| = Θ(N/K).The supplied passage shows the scaling fact used in the proof, but does not expose the full claim statement.Source: §G.4.2 · p. 84

G.5 Stage II: Phase II

• During Phase II, B_k increases within [log(K), O(log(K/ϵ^2))], while each B_k′ for k′ ≠ k decreases and has magnitude O(B_k/K).These bounds hold under the stated Phase II induction hypothesis and conditioning.Source: §G.5 · p. 84

G.5.1 Technical Lemmas

• The technical lemmas bound attention scores for the target and non-target features, and show non-target updates b_k′ are nonpositive with magnitude O(b_k/K).The supplied passages state the attention-score bounds without preserving their full expressions.Source: §1 · p. 81, §G.5.1 · pp. 85–86

G.5.2 End of Phase II

• At the end of Phase II, the proof establishes that B_k exceeds the largest competing B_m by at least a logarithmic margin.The displayed threshold after log is corrupted in the supplied passage.Source: §G.5.2 · p. 87

H Proof of Theorem 5.3

• The Stage I and Stage II claims follow directly from Theorems F.22 and G.9, respectively.Source: H Proof of Theorem 5.3 · p. 88
• The remaining proof task is to show loss convergence by rewriting the loss.Source: H Proof of Theorem 5.3 · p. 88
• The loss derivation uses conditional expectation of w given P and direct calculation for step (i), and Theorem G.9 for step (ii), completing the proof.Source: H Proof of Theorem 5.3 · p. 88