GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs

• GDN heads differ greatly in memory duration, and which heads retain information longer is determined by weights and remains stable across documents.This observation concerns Qwen3.8-27B’s GDN memory dynamics.Source: §1 · p. 2
• GDN write gates, decay, and write magnitudes emitted during prefill predict future KV utility before any query exists, across document lengths and retrieval tasks.Source: §1 · p. 2
• The utility relation is distributed and nonlinear: no single signal, head, or layer captures it reliably, and nonlinear prediction recovers more of its high-utility tail than a linear map.Source: §1 · p. 2
• GateSight uses per-head MLPs to predict future KV utility from GDN signals and entry age, while the attention output gate sets each layer’s retention budget.It scores once after prefill without materializing attention scores, then physically compacts the cache for reuse across later questions.Source: §1 · p. 2
• With 10% of the attention KV cache, GateSight is most accurate of five query-agnostic methods on three of four model–benchmark pairs and leads KVzip on all four by 9.5 to 47.2 points.Evaluations use SCBench and RULER-4096 with Qwen3.8-27B and Qwen3.8-Flash-Next, up to 262K tokens.Source: §1 · p. 2

• The native GDN signals are the per-token write gate, log-decay, and write magnitude, obtained from the prefill without another forward pass.Source: §2 · p. 3
• GDN layers maintain recurrent matrix states, while full-attention layers retain a KV entry for every token, making the attention cache grow linearly with context.In Qwen3.8-27B, the attention cache stores 64 KiB per token in bfloat16.Source: §2 · p. 3
• Each KV entry is the eviction unit: removing it removes that position from the softmax of all six query heads in its group, while leaving GDN states untouched.Source: §2 · p. 3
• At a given kept fraction, eviction divides into choosing entries within each layer and allocating the total cache budget across layers.The model-wide budget excludes a protected instruction prefix kept by every method.Source: §2 · p. 3
• Future utility is the change in attention-layer output when a KV entry is removed across sampled model continuations, and serves as the analysis and training target rather than a serving-time calculation.The output gate is omitted from the leave-one-out influence calculation.Source: §2 · p. 3

3.1 OBSERVATION 1: GDN MEMORY DYNAMICS ARE STRUCTURED AND STABLE ACROSS DOCUMENTS

• Across 2,304 heads, document-pair half-life rankings agree at Spearman 0.99–0.997, while checkpoint-derived half-lives match measured ordering at 0.942.Median half-life is about 24 tokens, but each layer spans heads that forget within a handful of tokens and heads that retain information for hundreds to thousands of tokens.Source: §3.1 · p. 4

3.2 OBSERVATION 2: GDN WRITE AND FORGET SIGNALS ANTICIPATE FUTURE KV UTILITY

• Slow-head effective write correlates with utility at +0.23 versus +0.10 for all heads, and the highest write decile reaches the top utility decile with 18.2% probability.The lowest slow-head write decile reaches the bottom utility decile with 19.2% probability; forgetting is also positively correlated with utility (+0.06 to +0.19). The fixed slow set carries over to 2K–8K documents and RULER-8K without re-estimation.Source: §3.2 · pp. 4–5

3.3 OBSERVATION 3: THE RELATION IS DISTRIBUTED AND NONLINEAR

• Predictive signal is distributed: 40–70% of preceding (layer, head) cells correlate with utility at |ρ| > 0.1, but no cell exceeds 0.39.The best fixed heuristic recovers 19.2% of true top-10% entries, the slow-head effective write 18.2%, and the best single-cell in-sample ceiling 22.6%.Source: §3.3 · p. 5
• The per-head MLP recovers 41.0% of true top-10% entries versus 38.6% for ridge, despite slightly lower Spearman correlation (0.543 versus 0.548).It also covers 58.4% of utility mass versus 55.8% for ridge, and leads in all sixteen attention layers on both measures.Source: §3.3 · p. 5

• GATESIGHT predicts which entries to keep within each layer, allocates each layer’s cache budget from its own attention output gate, and deploys by compacting the cache for later questions.The observations determine the estimator, while the output gate supplies the allocation signal and deployment serves the compacted cache.Source: §4 · p. 5
• The estimator uses recurrent-memory observations to choose entries within a layer, while allocation answers how much each layer keeps.The allocation uses a signal from the attention layer itself: its output gate.Source: §4 · p. 5
• Deployment compacts the cache so it can serve later questions.The method’s three parts are the estimator, allocation, and deployment.Source: §4 · p. 5

1 Prefill once 2 Estimate utility 3 Budget & selection 4 Compact & serve

• After one prefill, a per-head MLP predicts entry utility, each head keeps its top-K entries under the gate-proportional layer budget, and discarded blocks return to the pool.The selection computes no attention score.Source: Figure 3 · p. 6, §1 · p. 6

4.1 ESTIMATOR

• Each estimator maps native signals from up to six preceding GDN layers and three normalized age features to predicted utility.The input has 867 dimensions; each estimator is a one-hidden-layer MLP with 64 units, and Qwen3.8-27B uses 64 estimators totaling 3.4M parameters.Source: §4.1 · p. 6

4.2 PER-LAYER ALLOCATION FROM THE OUTPUT GATE

• Each layer’s budget is proportional to its mean attention output-gate magnitude, with the same model-wide total as a flat allocation.The mean gate ranges from 0.022 to 0.459 across layers; a per-layer budget also keeps each layer’s retained cache as a dense prefix.Source: §4.2 · p. 6

4.3 DEPLOYMENT IN VLLM WITH PHYSICAL COMPACTION AND SESSION REUSE

• At prefill’s end, the estimator scores the context once, retained entries move to the front of each layer’s pages, and freed blocks return to the pool.The compacted cache is reused for later questions with the GDN states restored.Source: §4.3 · p. 7

5.1 SETUP

• The evaluation uses SCBench’s shared-context multi-query mode and RULER’s 13 synthetic tasks at 4,096 tokens, comparing five query-agnostic methods under a shared vLLM protocol.It tests Qwen3.8-27B and Qwen3.8-Flash-Next, with kept fractions from 0.9 to 0.1.Source: §5.1 · p. 7

5.2 MAIN RESULTS

• On Qwen3.8-27B, GateSight beats SnapKV, TriAttention, and KeyDiff at every kept fraction; on Flash-Next, its lead varies by benchmark and competitor.On Flash-Next, it beats SnapKV and KeyDiff throughout SCBench, while TriAttention is close at higher fractions; on RULER-4096, KeyDiff leads at p ≤ 0.5.Source: §5.2 · p. 8

5.3 SCORING COST

• At 256K tokens, GateSight scoring takes 0.246 s, or 0.5% of the 46.4 s prefill time; KVzip’s second pass takes 1.5 to 2.2 times prefill time.Across 4K–256K contexts, GateSight scoring takes 0.5% to 2.6% of prefill time.Source: Table 2 · p. 8, §5.3 · p. 8

5.4 PRUNING A SPARSE-ATTENTION CACHE

• For Flash-Next’s sparse cache, GateSight regroups kept tokens into QSA blocks, reducing each layer’s index from n/4 compressed keys to an average of pn/4.Each query attends to 2,048 context tokens, while the indexer scans one compressed key per four context tokens.Source: §5.4 · p. 8

5.5 COMPARISON WITH KVZIP

• At p = 0.1, GateSight leads KVzip by 9.5 and 47.2 points on Qwen3.8-27B, and by 15.4 and 25.6 points on Flash-Next (SCBench and RULER-4096).On Qwen3.8-27B, KVzip leads at intermediate fractions, while GateSight stays within 1.2 points at p ≥ 0.7.Source: Table 3 · p. 8, §5.5 · p. 8

5.6 ABLATIONS

• With equal total budgets, gate-proportional allocation is at least as accurate as flat and PyramidKV at every kept fraction; on RULER-4096 at p = 0.1, it scores 98.6% versus 95.5% and 52.1%.The comparison isolates where entries are kept by holding the total number of entries fixed.Source: Table 4 · p. 9, §5.6 · p. 9

6 RELATED WORK

• Attention-score eviction methods include heavy hitters, persistence, observation windows, per-head policies, redundancy terms, and layer- or head-varying budgets, but require queries or a context probe.The cited examples include heavy hitters [Zhang et al., 2023], persistence [Liu et al., 2023], observation windows [Li et al., 2024], and redundancy terms [Cai et al., 2025a].Source: §6 · p. 9
• Query-agnostic alternatives preserve attention sinks and recent tokens [Xiao et al., 2024], use static head partitions [Xiao et al., 2025], reconstruct context [Kim et al., 2025], or score cached keys [Mao et al., 2026; Park et al., 2025; Devoto et al., 2024].The paper notes that Observation 1 finds an analogous static structure in recurrent layers.Source: §6 · p. 9
• GateSight is query-agnostic like KVzip and probe-free like TriAttention and KeyDiff, but reads the recurrent state rather than the attention layer.The authors state that, to their knowledge, prior work had not used recurrent layers to manage attention-layer KV caches.Source: §6 · p. 9

7 CONCLUSION

• GDN write and forget decisions are structured, stable across texts, and predictive of which KV entries later change attention output; GateSight uses them to evict entries after prefill without a query or attention score.With a tenth of the attention KV cache, GateSight is the most accurate of five query-agnostic methods on three of four model–benchmark pairs and leads KVzip on all four by 9.5 to 47.2 points.Source: §7 · p. 9

AI USE STATEMENT

• The authors used language-model assistants for drafting, editing, and coding, but not for research ideas or method and experiment design.They report checking the generated numbers against raw-result scripts and testing AI-written code.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The release includes GateSight code, trained estimators, calibration artifacts, evaluation and training scripts, and a pinned software environment.The paper points to its repository and lists appendices covering hyperparameters, evaluation protocol, and observation definitions.Source: REPRODUCIBILITY STATEMENT · p. 10

A EVALUATION PROTOCOL

• Each method scores a prefilled parent session, then a physically compacted child is reused for every question; answers are token-identical to a fresh run.Contexts exceeding engine limits are middle-truncated before prefill, and comparisons require a real fork and physical compaction.Source: A EVALUATION PROTOCOL · p. 12
• SCBench evaluates each context with all its questions up to 262K tokens, while RULER covers retrieval, tracing, extraction, and question-answering tasks.RULER tasks use 100 items each, generated with the official scripts and seed 42.Source: A EVALUATION PROTOCOL · p. 12
• Two SCBench tasks use revised scorers: the corrections shift Qwen3.8-27B task means by +5.6 and −12.3 points, while the five-method ranking order remains unchanged under the official scorer.The revisions address answer parsing in qa chn and choice eng and apply to every method, budget, and model.Source: A EVALUATION PROTOCOL · pp. 12–13

B UTILITY LABEL

• The utility label is the maximum leave-one-out attention-output change over sampled continuation tokens, max-normalised per document, layer, and KV head.The procedure recomputes continuation attention in float32 and checks the closed form against brute-force key deletion to within 1.4 × 10−5.Source: B UTILITY LABEL · p. 13

Example document

• Figure 5 shows a median-correlation example, with traces scaled to their post-first-token maxima; Figure 6 reports per-cell correlations at four attention layers.The example document is selected by its median Spearman correlation between slow-head β∥v∥ and the KV-head mean of utility at attention layer 55.Source: C DETAILS BEHIND THE OBSERVATIONS · p. 13, Figure 6 · p. 14

Predictor evaluation

• Adding GDN signals raises top-10% overlap by 23.1 percentage points over an age-only model, which recovers 17.9%.The MLP's advantage over ridge in overlap@10% holds in all nine corpora and is widest on SFT code, 45.1 against 38.3%.Source: C DETAILS BEHIND THE OBSERVATIONS · p. 14
• The estimator and slow-head effective write recover 42.6% and 41.1% of top-10% entries on niah single 1, versus 40.7% and 30.3% on niah multikey 3.Across five RULER tasks, the estimator's top-10% recovery ranges from 31.2% to 52.3%; the largest gain from learning the combination occurs on multi-key retrieval.Source: C DETAILS BEHIND THE OBSERVATIONS · p. 15

Context length and task text

• Slow-head β∥v∥ correlates with utility at +0.34, +0.34, and +0.33 on the 2K, 4K, and 8K cohorts, and +0.30 on RULER-8K.The slow-head set is not re-estimated; the estimator was trained on 2,428 short documents and applied unchanged, making its 1K row in-sample and the other rows held out.Source: C DETAILS BEHIND THE OBSERVATIONS · pp. 14–15

Within a head the gate does not rank tokens

• Applying the output gate to values barely changes within-head token rankings: gated and ungated scores have Spearman 0.995, with indistinguishable end-to-end scores.The gate therefore enters GATESIGHT as a per-layer budget weight rather than as a per-token utility signal.Source: D GATE ABLATION · pp. 15–16

Per layer, not per head

• Layer means of the attention output gate span 0.022 to 0.459, and gate-proportional per-layer budgets outperform flat allocation at p = 0.2 and 0.5 on the RULER-8K five-task mean.At p = 0.2 and 0.5, per-layer scores are 80.0 and 95.0, versus 74.0 and 87.0 for flat allocation.Source: D GATE ABLATION · p. 16

Proportionality needs no exponent

• At p = 0.05, gate-proportional budgets score 61.5, versus 59.2 and 60.8 with gate powers 0.5 and 1.5, and 53.8 for flat allocation.The analysis includes tasks solvable at that budget, excluding multi-key retrieval; the method uses the natural exponent of one throughout.Source: D GATE ABLATION · p. 16

SCBench

• On Qwen3.8-27B SCBench, GATESIGHT leads KVzip by 16.0 points at p = 0.1 on seven tasks, retaining 98% of unpruned accuracy versus KVzip's 64%.KVzip leads on several verbatim-answer tasks; GATESIGHT is at least as accurate on ten of twelve tasks at p = 0.1.Source: E PER-TASK AND PER-BUDGET RESULTS · p. 16, Figure 9 · p. 17
• On Qwen3.8-Flash-Next SCBench, GATESIGHT leads KVzip by 23.1 points at p = 0.1 on seven tasks, retaining 98% of unpruned accuracy versus KVzip's 51%.It is at least as accurate as KVzip on eleven of twelve tasks at p = 0.1, while KVzip's gains again come from verbatim-answer tasks.Source: E PER-TASK AND PER-BUDGET RESULTS · p. 16, Figure 10 · p. 18

RULER

• On Qwen3.8-27B RULER, KVzip reaches at most 2% on five tasks at p = 0.1, while GATESIGHT reaches at least 99.8%; these tasks account for 81% of the 47.2-point mean gap.GATESIGHT is at least as accurate on ten of thirteen tasks at every kept fraction, though KVzip leads on three tasks at some fractions.Source: E PER-TASK AND PER-BUDGET RESULTS · pp. 16–17, Figure 11 · p. 19
• On Qwen3.8-Flash-Next RULER, GATESIGHT is at least as accurate as KVzip on twelve of thirteen tasks at p = 0.1, but KVzip leads on several tasks at intermediate fractions.KeyDiff's mean lead at p ≤0.5 comes from four needle tasks; on those tasks it scores at least 83.5% at every kept fraction.Source: E PER-TASK AND PER-BUDGET RESULTS · pp. 17–18, Figure 12 · p. 20

F ABLATIONS

• A six-layer window is retained: widening it to 9 or 12 layers raises Ov@10 by only 0.2 or 0.3 points, at 1.5 or 1.9 times the parameters.Halving the window to 3 lowers Ov@10 by 0.8 points; gains per added layer fall above six.Source: F ABLATIONS · p. 21, Table 7 · p. 21

G IMPLEMENTATION DETAILS

• After prefill, each attention layer gathers its retained entries to the front of its pages and returns discarded blocks to the pool for physical memory release.The plugin restores GDN states for later questions, which reuse the compacted cache.Source: G IMPLEMENTATION DETAILS · p. 22

H QWEN3.8-FLASH-NEXT

• On Qwen3.8-Flash-Next, GATESIGHT scoring takes 1.0% to 1.4% of prefill time, or 0.175 s against 14.2 s at 256K tokens.For sparse attention, retained tokens are regrouped into new four-token blocks, and layer budgets are proportional to mean output gate.Source: H QWEN3.8-FLASH-NEXT · pp. 22–23, Table 8 · p. 23

I EXTENDED RELATED WORK

• GATESIGHT differs from query-agnostic key-scoring methods by reading recurrent-state signals rather than only the attention layer's keys.The paper states that, to the authors' knowledge, prior work had not used hybrid-model recurrent layers to manage attention-layer KV caches.Source: I EXTENDED RELATED WORK · pp. 23–24