PaperlayerView exact revision on arXiv

Source-linked AI summary

Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression

Xingyu Zhu, Pu, Yi, Ziheng Cheng, Ang Lv, Jing Liu, Lexing Ying, Yiyuan Ma, Xin Dong

arXiv:2609.36322v1cs.LGcs.AI

TL;DR

Chunked KV-cache compression reduces long-context memory and attention costs, but makes retrieval depend on source phase relative to compression windows. The paper tests this behavior in existing and pretrained models, using controlled retrieval evaluations, causal interventions, and idealized analysis. Retrieval varies periodically across phases, with gaps up to 40 percentage points, so average accuracy can conceal positional failures.

  • Problem

    Chunked KV-cache compression lowers long-context costs, but it is unclear how retrieval of the same information varies with its position relative to compression-window boundaries.

    Source: §1 · p. 1

  • Method

    The authors evaluate compressed-cache models and pretrained transformer variants, use causal interventions to examine attention contributions, and analyze an idealized compressed-attention retrieval task.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Retrieval performance varies periodically with source phase, with gaps up to 40 percentage points in DeepSeek-V4 and matching-stride periodicity across pretrained compressed models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Average accuracy can conceal phase-specific retrieval failures, so reliability should also be measured across compression phases.

    Source: §1 · p. 2

    The mechanistic conclusions are strongest in controlled settings and do not establish a universal cause of phase sensitivity in large, heterogeneous models.

    Source: §7 · p. 10

Abstract

from arXiv · show

Chunked KV-cache compression reduces the memory and attention costs of long-context inference by compressing windows of consecutive tokens into fewer cache entries at a fixed stride. Such compression also introduces a new positional coordinate: a token's phase, or its position relative to compression-window boundaries. We uncover a systematic asymmetry in models using such compression: the same information can be easy to retrieve at one phase and difficult at another. We call this periodic variation in retrieval performance phase sensitivity. In large open-weight models with such compression, long-context retrieval accuracy can differ by up to 40 percentage points across phases, revealing periodic weak spots that average benchmark scores can conceal. To investigate this behavior, we pretrain a family of transformers from scratch across multiple KV-compression designs, reproducing phase sensitivity across the variants. Mechanistic analysis using causal interventions in these models reveals phase specialization: different attention components contribute asymmetrically to retrieving information at different source phases. We further analyze idealized retrieval models, showing how gradient flow dynamics may favor sharp phase specialization. Evaluating models with chunked KV-cache compression thus requires measuring across compression phases: high average accuracy can coexist with systematic positional failures.

1 Introduction

Chunked KV-cache compression lowers long-context costs but groups tokens into fixed windows, making retrieval depend on a token’s phase relative to window boundaries. The paper documents this periodic sensitivity, investigates its mechanisms, and concludes that retrieval should be measured across phases.

1 Introduction

  • At 128K context, DeepSeek-V4 retrieval accuracy can differ by up to 40 percentage points across source phases, a periodic failure that average scores can conceal.The authors find that phase sensitivity also appears in DeepSeek-V4.1 and in all compressed models they pretrained.Source: §1 · p. 2
  • Pretrained compressed models show periodicity matching their compression stride, while full-attention baselines do not; window boundaries alone do not explain all phase variation.Accuracy also varies across phases where each key–value pair remains within one window.Source: §1 · p. 2
  • Causal interventions reveal phase specialization: different attention heads contribute differently across retrieval phases, with persistent gate preferences and strongest effects near favored phases.The pattern appears in the authors’ pretrained models and DeepSeek-V4.Source: §1 · p. 2
  • In an idealized compressed-attention induction task, the optimal compressor favors particular phases, and gradient flow drives compression gates toward persistent preferences.Mixing information within a compression window can weaken the match between a compressed key and a later query.Source: §1 · p. 2
  • Because average accuracy can hide phase-specific retrieval failures, the authors recommend measuring reliability across phases, for example by shifting input while keeping queried information fixed.Source: §1 · p. 2

2 Stride-Periodic Behavior in DeepSeek-V4 and DeepSeek-V4.1

DeepSeek-V4 and V4.1 show periodic changes in code completion and controlled retrieval as source positions shift relative to compression windows. Retrieval gaps can be large, and post-training does not remove all phase differences.

2.1 A Periodically Flipping Code-Completion Example

  • Changing only docstring length flips the preferred completion every four tokens, matching S = 4; the model predicts 8 when L mod 4 is 2 or 3, and 32 otherwise.The ground-truth completion is 8, and the figure reports averages over eight lengths in each residue group.Source: §2.1 · p. 3, Figure 1 · p. 3
  • The four-token pattern appears across tested filler families and in post-trained DeepSeek-V4 and V4.1 models, while DeepSeek-V3.1-Base rarely ranks 32 above 8 in the same inputs.The V4 and V4.1 reversals match their respective compression strides of 4 and 2.Source: §2.1 · p. 3

2.2 Measuring Phase Sensitivity in Retrieval

  • The controlled NIAH evaluation compares retrieval of the same key–value information at different positions relative to compression boundaries.Prompts have 128K tokens, and residue groups share prompt length, query position, and key–value bindings.Source: §2.2 · p. 3
  • Post-training raises accuracy and narrows gaps, but DeepSeek-V4 gaps remain large; V4.1-Flash has the smallest gap, with even-residue groups above odd ones.Source: §2.2 · pp. 3–4
  • Up to 40.2 percentage points: base-checkpoint retrieval gaps reach this maximum; all five models vary with period 2 or 4, a pattern average accuracy can conceal.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

3 Controlled Pretraining Studies

The controlled pretraining study isolates chunked KV compression in a Qwen3-derived model family and tests how compression design affects phase sensitivity. Every compressed variant shows stride-periodic retrieval variation, unlike full-attention baselines; changing the stride changes the period, and within-window variation remains beyond boundary effects.

3 Controlled Pretraining Studies

  • The study pretrains Qwen3-derived models with chunked compression as their only substantial architectural change, alongside full-attention baselines, to test whether compression produces phase sensitivity.DeepSeek-V4 has other architectural differences, and the authors cannot retrain it without compression.Source: §3 · p. 4

3.1 A Family of KV Compression Kernels

  • The kernel family varies window size, stride, gating, positional encoding, and compression design; the W8/S8 reference uses non-overlapping eight-token windows and scalar gates.The evaluation adapts needle-in-a-haystack retrieval to 64 adjacent single-token key–value pairs and uses Prefix Padding and In-sequence Padding protocols.Source: §3.1 · pp. 4–5

3.2 Phase Sensitivity across the Kernel Family

  • The period follows stride across tested geometries, and W8/S8 accuracy varies by more than 50 percentage points even among phases whose key–value pairs share a window.Sensitivity also persists without RoPE and with uniform averaging; In-sequence Padding shows it does not require all records to shift together.Source: §3.2 · pp. 5–6
  • Up to 78 percentage points for Prefix Padding gaps across compressed runs, versus at most 6.1 for full-attention baselines; compressed models vary periodically with source-key phase.The contrast appears in Figure 3 and holds across the compressed runs in Table 8.Source: Figure 3 · p. 5, §3.2 · p. 5
  • A window-boundary explanation is incomplete: in W8/S8, only phase seven separates adjacent keys and values across windows, yet phases zero through six still differ by more than 50 percentage points.The remaining variation motivates examining how the model writes to and reads from compressed memory.Source: §3.2 · p. 6

4 Phase Specialization

Phase specialization describes phase-dependent contributions of attention heads to retrieval, measured through KV-head knockouts. The analyses find phase-specific effects in compressed models and connect persistent gate preferences to those retrieval phases.

4 Phase Specialization

  • Phase specialization means that some heads’ retrieval contributions depend systematically on source phase, without presupposing a cause.Mean-replacement knockouts test whether a head’s prompt-specific output contributes causally to retrieval at a phase.Source: §4 · p. 6, §4.1 · p. 6

4.1 Probing Phase Specialization through KV-Head Knockouts

  • Compressed-model knockout effects vary across source phases, with DeepSeek-V4-Flash-Base showing a repeating four-position pattern matching its compression stride.Unlike the full-attention baseline’s uniform layer contributions, compressed models show heterogeneous phase patterns; effects also recur across most tested compressed models.Source: Figure 4 · p. 6, §4.1 · p. 7

4.2 A Case Study of Emerging Gate Preferences

  • In the W8/S8 case study, gates in layers 9, 10, and 14 develop static preferences for late, middle, and early window offsets, respectively.Key-gate peaks align with phases where the heads matter for retrieval; value gates peak one offset later, favoring a token and its successor.Source: §4.2 · pp. 7–8, Figure 5 · p. 7
  • Gate concentration distinguishes fixed offset preferences from input-dependent concentration, using effective offset counts based on entropy.Static concentration is measured as exp(H(E[α])); average per-window concentration is E[exp(H(α))].Source: §4.2 · p. 7
  • Cycling offset-specific gate parameters by one to three offsets largely shifts within-window accuracy patterns as predicted in five models, supporting a causal contribution of learned preferences to retrieval asymmetry.The study also reports more than 50 percentage points of variation among within-window phases in the reference model, including variation under uniform averaging.Source: §4.2 · p. 8

5 Theory: Emergence of Phase Specialization in Compression

The paper studies why query-agnostic compression may concentrate information at particular within-window offsets, and whether training makes those preferences static. Its idealized analyses distinguish concentration that can vary by chunk from concentration fixed for each head.

5.1 Concentration in Optimal Solutions

  • In the oracle analysis, every optimal compression is maximally concentrated on one effective offset per head and K/V branch in each chunk, even though the query is unknown.The selected offset may vary across contexts and chunks; when ε = 0, selected positions are distinct across heads within each context and chunk.Source: §5.1 · p. 9

5.2 Static Concentration under Gradient Flow

  • Under the stated assumptions, gradient flow makes each head’s key and value gates converge to one offset pair, fixed across contexts and chunks; different heads may choose the same offset.The value gate selects the position immediately after the key gate’s selected position.Source: §5.2 · p. 9
  • The gradient-flow result assumes exact geometry and sufficiently small random initialization, with sufficiently large vocabulary size relative to the initialization scale.The theorem gives persistent selection, but does not require different heads to select complementary offsets.Source: §5.2 · p. 9
  • Unlike the oracle optimum, gradient flow fixes each head’s selected position across inputs, providing an idealized account of the adjacent key and value preferences observed in some pretrained layers.The observed gate scores remain softer than the one-hot limit.Source: §5.2 · p. 9, §5.1 · p. 9

6 Related Works

The related work situates phase sensitivity alongside structured memory, positional retrieval failures, and retrieval-head specialization. The paper distinguishes its focus on source-token phase and query-agnostic compression from these neighboring lines of work.

6 Related Works

  • Prior compressed-memory and structured-attention methods impose block or token structure; Sparse Transformer’s strided head already has visibility that depends on position modulo a stride [8].Exact branches in hybrid systems such as NSA could mask failures of the compressed branch alone, whereas DeepSeek-V4 keeps exact tokens only in its local window [11].Source: §6 · p. 10
  • Unlike work on depth-dependent retrieval, easy needle tests, aggregate compression scores, or generation-segment boundaries, this paper studies source-token phase recurring every compression stride [19] [16] [5] [13].The paper distinguishes source phase from context depth and the query’s position within a segment.Source: §6 · p. 10
  • Earlier studies show retrieval heads differ in role, while this work finds that a head’s retrieval contribution also depends on phase and aligns with its gate-preferred offsets [38] [31] [39].The paper frames its theory around when query-agnostic compression is concentrated and when training makes that concentration static, rather than claiming the first theory of attention hardening or head specialization [32] [33] [42].Source: §6 · p. 10

7 Conclusion and Discussion

The paper concludes that retrieval in chunk-compressed KV caches can vary periodically with source phase, with evidence from DeepSeek models and pretrained compressed models. It also identifies limits to the mechanistic account and leaves possible robustness effects for future study.

7 Conclusion and Discussion

  • Phase sensitivity appears in DeepSeek-V4, V4.1, and all pretrained compressed models tested, with a period matching the compression stride.KV-head knockouts and attention-readout interventions provide causal evidence for phase specialization in the tested models.Source: §7 · p. 10
  • The mechanistic findings do not establish a universal cause of phase sensitivity in large, heterogeneous models, and gate cycling does not restore the boundary phase.A unified explanation of boundary and within-window asymmetries remains unresolved.Source: §7 · p. 10
  • The code-completion examples motivate, but do not systematically study, whether phase sensitivity affects robustness-related behavior in real applications.The authors leave potential effects on refusal or compliance under meaning-preserving request shifts for future work.Source: §7 · p. 10

A Deferred Proofs for Theorem 1 and Theorem 2

This section presents the deferred analysis of compression and phase specialization, defining oracle and linear compressor classes chosen before query location sampling and relating physical phase indices to window offsets and the adjacent key–value route. Its results connect balanced, sharpening-driven complementary selections in a two-phase toy model with general dynamics that guarantee selection but not distinct routes across heads.

A Deferred Proofs for Theorem 1 and Theorem 2

  • Theorem 8 says every oracle-class minimizer uses hard adjacent key–value phases, while Theorem 12 shows small-initialization gradient flow reaches one hard route per head, though heads may share routes.The selected route may vary across contexts and chunks.Source: A Deferred Proofs for Theorem 1 and… · p. 15

A.1 A Two-Phase Toy Example

  • In the two-phase toy model, gradient flow assigns the two heads different phases when output scale β is large enough, and the only global minima are the complementary endpoint selections.The toy model uses two heads, two phases, and an idealized value readout; its endpoint-minimizer result assumes D > 1.Source: §A.1 · p. 15, §A.1.1 · p. 17, If also · p. 17

A.1.1 Complementary Two-Phase Selection

  • Sharpening raises mean attention mass, while logistic-loss curvature favors equal phase coverage; both pressures are satisfied at opposite endpoints.With D > 1, the attention gain from increasing a target logit outweighs the loss from an equal decrease; balanced margins minimize loss at fixed mean.Source: §A.1.1 · p. 16, §A.1.2 · pp. 19–20

If also

  • From sufficiently small absolutely continuous initializations, gradient flow converges almost surely to a complementary minimizer; bounded initialization density gives a high-probability time bound to approach one.The theorem’s rate depends on L, β, and the initialization law; its stated high-probability bound is for δ, η ∈ (0, 1/2).Source: If also · pp. 17, 19

A.1.2 From the Toy Example to the General Model

  • In the general model, key–value pairing favors one-hot adjacent phases, but large vocabularies weaken cross-head balancing, so Theorem 12 guarantees persistent selection without complementary coverage.Under exact geometry, same-position selections retain the expected target logit and incur only a small normalizer penalty; the dynamics proof treats normalizer terms as perturbations.Source: §A.1.2 · p. 20

A.2 Model and Compressor Classes

  • The formal model studies adjacent-token retrieval from non-overlapping chunks of S positions, with L chunks and queries drawn uniformly from positions whose successor is the target.The context embeddings are sampled without replacement from a vocabulary of N distinct vectors, with N ≥ LS.Source: §A.2 · p. 20, §A.1.2 · p. 20
  • Oracle compressors may choose coefficients arbitrarily from the context before the query location is sampled.The information order is context X mapped to g(X), followed by a uniformly sampled query location (ℓ*, r*).Source: §A.2 · pp. 20–21
  • Physical phase indices map position r to window offset r − 1, and the adjacent route pairs key phase r with value phase r + 1.No phase is masked.Source: §A.2 · p. 21
  • Training-dynamics analysis restricts the oracle class to shared linear gate logits, a potentially strict subset of arbitrary context-dependent rules.The linear compressor uses vectors shared across contexts and chunks; the inclusion can be strict because oracle rules may depend on the whole context.Source: §A.2 · p. 21

A.3 Proof of Theorem 1

The proof shows that oracle minimizers use hard adjacent key–value phase selection. Under exact geometry, minimizing assignments also require distinct routes across heads; the oracle may choose routes separately for each context and chunk.

A.3 Proof of Theorem 1

  • Oracle minimizers use one-hot adjacent key–value phase selection for every context, chunk, and head.Replacing any mixed coefficients with a one-hot adjacent collection strictly lowers the loss; under the theorem’s assumptions, the perturbed-geometry gap remains positive.Source: §A.3 · pp. 22–24
  • When ε = 0, exactly the injective route assignments minimize the loss; any head collision strictly increases it.Here, injectivity means distinct heads select distinct key phases within each context and chunk.Source: §A.3 · pp. 22–24
  • The oracle’s route assignment may vary across contexts and chunks, so the theorem does not establish a fixed route for each named head.Source: §A.3 · p. 24

A.4 Proof of Theorem 2

This section examines phase-dependent retrieval behavior through a proof of how small random initialization under population gradient flow produces persistent head-specific preferences and uniform retrieval convergence for a selected phase. A code-completion example illustrates periodic reversals in next-token preference as filler length varies, with the pattern largely persisting across filler variants.

A.4 Proof of Theorem 2

  • With high probability, training produces aligned key and successor-value phase winners for each head.The theorem assumes the stated retrieval setting and small, uniform random initialization; the proof obtains aligned winners with a positive gap before showing their amplification and persistence.Source: §A.4 · pp. 25, 29
  • Linear gates can vary by input, but the flow’s unstable route-contrast directions align key and successor-value preferences, yielding static choices.The zero-start trajectory remains symmetric; differences between routes grow along an instability that acts separately on each head, and small random starts acquire routes with high probability.Source: §A.4 · pp. 25–26
  • On the mean-aligned subspace, adjacent winners persist, their gaps reach any fixed level in time bounded independently of vocabulary size, and then diverge.The terminal cone is forward invariant, with its displayed margins tending to positive infinity.Source: §A.4 · p. 30
  • The proof shows that every relevant margin diverges, yielding convergence uniformly over contexts and chunks for the selected phase.The terminal cone is forward invariant, and the proof uses divergence of the relevant margins to establish the theorem’s uniform convergence result.Source: §A.4 · p. 31

B Additional Details on the Code-Completion Examples

  • Appendix B provides exact token accounting, post-trained and non-chunked model comparisons, and two further code-completion examples.It also specifies that token phase is position t mod S for compression stride S.Source: B Additional Details on the… · p. 31

B.1 FP8 Example for DeepSeek-V4-Flash-Base

  • The test shifts the filler and source-code prefix together, so it does not isolate the position of an individual source token.It scores the next-token probabilities of reference continuation 8 and competing continuation 32 after a raw prefix, without an instruction, BOS token, or chat template.Source: §B.1 · p. 32
  • Families A, B, and C use content-matched position shifts, while Family D tests robustness to rephrasing with semantically similar comments.Across each family, the filler has 16 token counts; A/B/C add punctuation at successive lengths, while D uses lexically distinct comments without repeated-symbol padding.Source: §B.1 · p. 32
  • Across Family A’s 16 lengths, Δ follows an exact four-token pattern: 8 is top-1 when L mod 4 ∈{2, 3}, while 32 is top-1 otherwise.The mean combined probability P(8) + P(32) is 0.982, and Δ ranges from −0.786 to +0.948.Source: §B.1 · p. 33
  • Across four filler families, all 32 measurements favor 8 at L mod 4 ∈{2, 3}; at other lengths, 28 of 32 favor 32 and four favor 8.The four exceptions are Family B at L = 29, Family C at L = 36, and Family D at L = 24 and L = 36.Source: §B.1 · p. 33

B.2 FP8 Example for Other DeepSeek Models

Across FP8, shell, and SQL completion sweeps, changing filler length can alter model predictions, often periodically. The section also compares a model without chunked pooling and points to additional NIAH evaluation details.

B.2 FP8 Example for Other DeepSeek Models

  • Post-trained FP8 tests retain the source prefix and reference continuation under identical chat-formatted causal prefixes, with four filler families spanning lengths 23–38.The instruction and source code remain fixed throughout the filler variants.Source: §B.2 · pp. 33–34
  • 4/16 (25%) correct on A/C and 3/16 (18.75%) on B for DeepSeek-V4-Flash-0731 contrast with DeepSeek-V4.1-Flash’s 8/16 (50%) on A–C and 11/16 (68.75%) on D.DeepSeek-V4-Flash-0731 selects 32 at every tested D length; DeepSeek-V4.1-Flash alternates by odd/even filler length on A, B, and C.Source: Table 2 · p. 34, §B.2 · p. 34, Table 4 · p. 35
  • Shifting the A filler from 23 to 24 tokens reverses preference margins in opposite directions for the two post-trained models; these shifted variants are not a code-completion benchmark.The reported Δ changes from −0.910 to +0.615 for DeepSeek-V4-Flash-0731, and from +0.124 to −0.555 for DeepSeek-V4.1-Flash.Source: §B.2 · p. 35
  • DeepSeek-V3.1-Base shows no periodic reversal in the FP8 example, unlike DeepSeek-V4-Flash-Base; the comparison also differs in architecture, training, and inference backend.DeepSeek-V3.1-Base compresses each token’s keys and values into a low-rank latent vector but does not pool tokens into compressed entries.Source: §B.2 · p. 35

B.3.1 Omarchy Lid-Close Test

  • DeepSeek-V4-Flash-0731 follows an exact four-position cycle on the lid-close test, alternating two incorrect and two correct predictions.DeepSeek-V4-Flash-Base errs at inputs 8, 12, and 16; DeepSeek-V4.1-Flash alternates between incorrect and correct predictions over the final eight inputs.Source: Table 5 · p. 36, §B.3.1 · p. 37

B.3 Additional Code-Completion Examples

  • Across the shell and SQL examples, filler sweeps produce top-1 reversals that are periodic for most evaluated models.The examples use a lid-close shell test and a station-distance SQL task.Source: §B.3 · p. 36

B.3.2 SQL Station-Distance Query

  • Across 16 SQL-comment fillers, DeepSeek-V4.1-Flash alternates +/− exactly; DeepSeek-V4-Flash-Base and -0731 make 2 and 11 mismatches, while only + matches replay output.Both operators produce executable SQL, but the recorded replay output distinguishes the correct continuation.Source: §B.3.2 · pp. 37–38, Table 6 · p. 38

C Additional Details on the Needle-in-a-Haystack Evaluations

C.1 NIAH Evaluation for the DeepSeek-V4 Family

This section specifies the phase-controlled NIAH evaluations and model setups, covering DeepSeek-V4 and Qwen3-based models, Qwen3 compression kernels and attention, and positional-encoding variants and attention-value handling in the evaluation models.

C.1 NIAH Evaluation for the DeepSeek-V4 Family

  • DeepSeek-V4 residue groups use tK mod 8, spanning two stride-four phase cycles or four stride-two cycles for V4.1-Flash; shifting target phase preserves position statistics but not distractor phase.The comparison fixes query position and the first two moments of target position, but the target’s phase is not varied independently of distractors.Source: §C.1 · pp. 38–39
  • 40.23 percentage points is the best–worst residue gap for DeepSeek-V4-Flash-Base; DeepSeek-V4.1-Flash has 92.38% mean accuracy and a 6.09-percentage-point gap.Across the four other DeepSeek-V4 variants, mean answer-prefix accuracies are 52.6%, 91.4%, 72.1%, and 84.4%, with gaps of 40.23, 19.14, 34.77, and 14.84 percentage points, respectively. Their residue r versus r + 4 accuracy differences average 1.07–1.95 percentage points, consistent with approximate four-token recurrence.Source: Table 7 · p. 40, §C.1 · p. 40

C.2 NIAH Evaluation for Qwen3-based Models

  • Qwen3-based NIAH uses token-exact animal–name mappings with the target at pair index 24, comparing Prefix Padding and In-sequence Padding while holding query position and expected target depth and distance fixed.Scoring uses final-token logits for candidate-restricted accuracy over 64 names or full-vocabulary accuracy, and also records gold-token probability and negative log-likelihood.Source: §C.2 · p. 40

C.3 Padding Protocols for Qwen3-based Models

  • The Qwen3 padding protocols sample 24 source-key residues across 1,000 matched logical examples, changing the target phase while fixing total filler length and query and scoring positions.Prefix Padding moves the mapping body with complementary natural-language fillers; In-sequence Padding redistributes bounded spacers around the target. The latter uses 174 guard-and-spacer tokens versus 144 filler tokens, placing its query 30 tokens later, while both protocols reuse the same mappings and examples.Source: §C.3 · pp. 41–43

D Additional Details on the Architecture and Pretraining of Qwen3-based Models

D.1 Backbone and Compression Kernels

  • The reference Qwen3-0.6B compression kernel uses non-overlapping W = S = 8 windows and one KV head per layer; the DeepSeek-style variant uses W = 8, S = 4, vector gates, and tied K/V branches.The backbone has 28 layers, hidden width 1,024, 16 query heads, and head dimension 128. The DeepSeek-style kernel shares separate payload and gate maps within each half-window, while positional biases remain offset-specific.Source: §D.1 · p. 43

D.2 Windows, Attention Memory, and Positional Encoding

  • Queries jointly attend to local and compressed memories; fixed-width models use a 16-token raw window and admit every causally closed compressed entry, with RoPE applied to queries, local keys, and pooled compressed keys.In the reference setting, queries and local keys receive per-head RMSNorm before RoPE, while compressed keys are normalized and rotated after pooling.Source: §D.2 · p. 44
  • Rp rotates the final dr dimensions at angles pωℓ using Qwen3 split-half pairing, leaves the first dh−dr dimensions unchanged, and the no-Q/K-normalization variant replaces RMSNorm with identity.No additional frequency or context-length scaling is applied.Source: §D.2 · p. 44
  • The reference uses dr=dh=128; partial-RoPE uses dr=16 (a 1/8 rotary fraction), while NoPE uses dr=0 and Rp=I.Source: §D.2 · p. 44
  • The partial-RoPE model constructs a 16-dimensional rotary table for the last 16 of 128 head dimensions, rather than selecting the first dr/2 frequencies of the full-head table.The full-head table would use an exponent denominator of dh rather than dr.Source: §D.2 · p. 44
  • With separate K/V branches, local values are WV hu and compressed values are bVj; with tied branches, values equal normalized, rotated keys and the combined output is inverse-rotated at the query position.The inverse rotation applies to the same dimensions as RoPE and to the combined local and compressed readout; separate branches leave the output unrotated.Source: §D.2 · p. 44

D.3 Training Settings

The reference training run uses a 100B-token schedule with specified optimizer, precision, batch, and learning-rate settings. The appendix defines KV-head knockouts as mean-output replacements and explains how they are calibrated and applied to query-head outputs.

D.3 Training Settings

  • The reference run traverses the packed corpus once for 100B tokens, using sequence length 2,048, global batch 1,024, BF16, and seed 42.AdamW uses (β1, β2) = (0.9, 0.95), ϵ = 10−8, weight decay 0.1, and gradient clipping; the learning rate warms to 2 × 10−3 and cosine-decays to 2 × 10−5.Source: §D.3 · p. 45

E Additional Details on KV-Head Knockouts

  • A KV-head knockout replaces the selected component’s prompt-specific output with its mean over separate calibration prompts, then measures retrieval effects by source phase.The mean retains average output while removing information specific to the prompt; calibration and evaluation prompts are distinct.Source: E Additional Details on KV-Head… · p. 45

E.1 Replaced Attention Output

  • In GQA, the intervention replaces the separate readout vectors of every query head sharing the selected KV head, immediately before the output projection.With four KV heads, it replaces four query-head vectors; with one KV head, it replaces all 16 query-head outputs at that layer and position.Source: §E.1 · p. 45

E.2 Calibration and Mean Replacement

  • Calibration computes a separate mean for each query head from phase-balanced prompts, and the test-time patch changes only the selected layer and scoring position before downstream computation is rerun.The reference is independent of the evaluation prompt and its phase; model parameters and context cache remain unchanged.Source: §E.2 · pp. 45–46, Figure 7 · p. 46

E.3 Evaluation Prompts, Metrics, and Uncertainty

The appendix specifies phase-resolved evaluation and mean-replacement interventions for pretrained models and DeepSeek-V4-Flash-Base. In the latter, the largest phase-averaged target-token accuracy losses occur at layers 22 and 24, while the intervention does not isolate a particular retrieval component.

E.3 Evaluation Prompts, Metrics, and Uncertainty

  • Evaluation uses 24,000 prompts per padding protocol, split by logical example into calibration and evaluation sets, with source positions tracked modulo 24 and then folded by model stride.The 24 phase replicas of each logical example stay together across the split.Source: §E.3 · p. 47
  • Retrieval is scored by candidate-restricted or full-vocabulary accuracy alongside logit and probability metrics, and knockout effects are paired against clean runs on the same prompts.The relative accuracy-change measure scales loss by clean accuracy available to lose and is undefined when clean accuracy is zero.Source: §E.3 · p. 47

E.4 Knockout Sweeps for Qwen3-based Models

  • The 23-model knockout sweep covers 28 layers under both padding protocols, and its heatmaps are point estimates rather than independently confirmed selections.The sweep uses disjoint calibration and evaluation sets at checkpoint 47,518.Source: §E.4 · p. 47

E.5 Scope of the Causal Interpretation

  • Mean-replacement effects support the importance of a selected readout under that intervention, not exclusive storage of the target in that KV head.Other heads may contain redundant information or compensate, so a small knockout effect does not establish that a component is unused.Source: §E.5 · p. 47
  • In compressed layers, mean replacement acts on the combined readout over compressed and local entries rather than selectively ablating compressed memory or a source association.The intervention also does not change a compression gate.Source: §E.5 · pp. 47–48

E.6 Knockouts for DeepSeek-V4-Flash-Base

  • DeepSeek-V4-Flash-Base’s knockout replaces the post-projection layer output and can patch selected query-suffix positions, using a separate calibrated mean at each suffix position.The full-suffix condition patches all 16 positions, and earlier patched positions can affect later suffix computation.Source: §E.6 · p. 48
  • Layers 22 and 24 have the largest phase-averaged target-token accuracy losses in DeepSeek-V4-Flash-Base’s full-suffix knockout, with effects varying across source residues.These results indicate phase-dependent reliance on the patched layer readout, not a particular query head, compressed-only branch, or compression operation.Source: §E.6 · p. 48

F Complete Results for Qwen3-based Models

Across the tested pretrained configurations, no design change removes periodic weak spots, although some narrow them. Gate preferences and knockout effects also vary across configurations and seeds, with sharper preferences in some heads aligning with phase-specific retrieval damage.

F Complete Results for Qwen3-based Models

  • Under Prefix Padding, Table 8 summarizes 23 compressed runs and three full-attention baselines at step 47,518, reporting validation loss and mean, minimum, and gap in retrieval accuracy across 24 source-key residues.Source: F Complete Results for Qwen3-based… · p. 48, Table 8 · p. 50
  • Uniform averaging reduces the best-to-worst retrieval gap from 75.3 to 19.2 percentage points, but no tested change removes periodic weak spots.Seeds, positional encoding, and normalization reshape the phase pattern; the weak spots repeat with the compression stride.Source: F Complete Results for Qwen3-based… · p. 49

G Gate Preferences and Knockout Effects across Models

  • Across seeds and configurations, sharp gate preferences with knockout damage at favored phases recur, but not in every head; phase-folded preferences are weak when windows overlap.The comparison covers every head in four seed replications and every head in five other configurations.Source: G Gate Preferences and Knockout… · p. 54

G.1 Measuring Gate Preferences

  • Static concentration measures the effective number of offsets in the mean gate, while the per-window measure also reflects input-dependent concentration; smaller values indicate greater concentration.A gate can select different offsets across windows and have little stable preference in its mean.Source: §G.1 · p. 54
  • With non-overlapping windows, a few heads show concentrated, persistent preferences; with overlapping windows, many gates vary with input while favoring offsets across windows.For overlapping windows, folding offset preferences into phases can hide them; channel-averaged vector gates can also conceal channel-specific preferences.Source: §G.1 · p. 54

G.2 Variation across Random Seeds

  • Across four seed replications, each has four to six heads with concentrated mean key gates peaking at 2.0 to 4.0 times the uniform level, and knockout damage falls mainly at favored phases.The preferred phases vary across seeds and collectively cover all eight phases; layer 0 impairs retrieval at every phase despite its near-uniform gate.Source: §G.2 · p. 54

G.3 Other Configurations and Overlapping Windows

Cycling gate preferences in the W12/S12 model approximately shifts retrieval accuracy across phases, supporting a causal contribution of gate preferences to within-window retrieval there. Across other configurations, sharp phase preferences persist without rotary embeddings but are less evident after phase-folding overlapping-window gates.

G.3 Other Configurations and Overlapping Windows

  • Without rotary position embeddings, heads still favor particular phases with knockout damage at those phases; with four KV heads, preferences sharpen, while overlapping windows largely flatten phase-folded gate curves.The four-KV-head preferences span all eight phases; folding overlapping-window offsets into phases can hide offset preferences.Source: §G.3 · p. 55, Figure 15 · p. 57, Figure 17 · p. 59

H.1 Gate Cycling

  • Gate cycling shifts gate-map and positional-bias parameters together while keeping payload maps and hidden states at their original offsets.The accuracy prediction concerns associations within a window and excludes the boundary phase from R2 comparisons.Source: §H.1 · p. 61
  • The cycling test edits key and value gates in layers 3 to 23 at step 47,518 and measures full-vocabulary top-1 accuracy on 1,000 logical examples.Accuracy is folded across the 24 source-key residues modulo stride, and R2 measures linear agreement between predicted and observed changes.Source: §H.1 · p. 61

H.2 One-Offset Cycles in a W12/S12 Model

  • From the boundary phase, both cycles shift the accuracy pattern approximately as predicted, with R2 of 0.984 for δ = +1 and 0.995 for δ = −1, excluding comparisons where phase 11 is observed or source phase.Phase 11 departs from the prediction under both cycles, limiting a purely within-window account but not identifying a separate retrieval circuit.Source: §H.2 · pp. 61–62

H.3 Replication across Models and Cycle Sizes

Across five models, gate cycling largely shifted weak within-window phases as predicted but left the boundary phase weak and reduced overall accuracy. The related-work discussion distinguishes source position within compressed memory from other recurring cross-chunk effects and situates the paper’s phase-specific head and theory claims among prior work.

H.3 Replication across Models and Cycle Sizes

  • Across five models, cycling largely predicts non-boundary phase shifts (R2 0.66–1.00) but leaves boundary accuracy at or below 2.9% and lowers full-vocabulary accuracy by 3.3–18.4 percentage points.The figure compares accuracy by source-key phase and predicted versus observed changes across models and cycle offsets.Source: §H.3 · p. 62, Figure 20 · p. 63
  • Agreement weakens as W8/S8 KV-head counts rise: minimum cycle-fit R2 falls from 0.75 with 2 heads to 0.36 with 8, and joint cycling cannot isolate individual-head preferences.The cycled layer range was inherited from the one-offset experiment rather than selected for each model.Source: §H.3 · p. 62

I Extended Related Work

  • Prior learned-memory methods compress or route information in varied ways, but the reviewed works do not examine retrieval as a function of source phase within compressed memory.The discussion includes pooled or convolved memories, special tokens, reconstruction objectives, beacons, recurrent memories, and CAT.Source: I Extended Related Work · p. 64
  • Deng et al.’s [13] recurring degradation concerns query position within a 2K segment, whereas this paper’s phase is source position modulo the smaller compression stride in already-compressed memory.The passage identifies that sawtooth as a direct precedent for recurring cross-chunk degradation, but with a different phase variable.Source: I Extended Related Work · p. 64
  • An exact-token branch in hybrid methods such as NSA could mask compressed-branch failures, while DeepSeek-V4’s local exact window does not prevent phase sensitivity in the full model.The cited passage reports phase sensitivity in DeepSeek-V4 in Section 2.Source: I Extended Related Work · p. 64
  • Prior work establishes head and layer heterogeneity; this paper’s narrower claim is that examined compressed-model KV-head knockouts have phase-dependent effects matching their gates’ preferred phases.A five-model gate-cycling intervention with non-overlapping windows complements this association.Source: I Extended Related Work · p. 64
  • To distinguish source-phase effects from absolute-position bias, the evaluation matches target-position mean and variance across residue groups and reports retrieval by phase rather than mean accuracy alone.Source: I Extended Related Work · p. 64
  • The theory targets retrieval from compressed memory: optimal compressors select sharply and gradient flow can make selections persistent, but heads are not guaranteed to choose distinct positions.The discussion distinguishes this theory from prior analyses of attention hardening, symmetry breaking, and division of labor.Source: I Extended Related Work · p. 65
Loading 2609.36322v1…