Source-linked AI summary
Thinking Outside the Box: Retention and Transmission of Information in Sliding-Window KV Inference
Timothy DeLise, Seth Cromelin
TL;DR
The paper asks whether information from tokens outside a fixed KV window can remain useful in retained states. It compares rolling-KV and fixed-window inference across retrieval experiments, finding that information can remain usable after source tokens leave the cache, especially for Mistral 7B and Muse Glimmer at measured post-eviction offsets.
Problem
The paper asks whether information originating outside the current KV window can persist through retained states and remain useful for retrieval.
Source: §1 · p. 2Method
Three experiments compare sliding-window KV inference with standard fixed-window inference to test information persistence and distance.
Source: §1 · p. 2Results
Mistral 7B and Muse Glimmer sustain high accuracy across measured post-eviction offsets, while Qwen2.5 models decline more sharply and Llama 3.1 accuracy declines at larger offsets.
Source: §5 · p. 12Takeaways & Limitations
Successful retrieval after the code's KV entries are evicted suggests a route toward recurrent-style memory, although the specific carrying states or computations are not identified.
Source: §5 · p. 12The evidence is behavioral retrieval on a synthetic four-choice task with a fixed W = 512 cache and few models; it does not establish preservation in natural dialogue or isolate why models differ.
Source: §5 · p. 13
Abstract
from arXiv · showhide
Sliding-window KV inference refers to processing a sequence incrementally while retaining only a fixed-size cache of recent key and value states. It can be applied to pretrained causal transformers at inference time without additional training, while its KV-cache memory remains fixed as more tokens are processed. Because cached states are computed in the context of earlier tokens, they may carry information from beyond the current window and transmit it to later states. This study presents a series of experiments using five open-weight models spanning Qwen, Llama, Mistral, and Muse Glimmer. We investigate whether information originating outside the immediate context window can persist through a rolling KV cache and remain useful for retrieval. Initial results show that retaining previously computed states improves retrieval across the models tested compared with recomputing the final fixed window from raw tokens. We then measure how far this effect extends and find that Muse Glimmer and Mistral 7B show the strongest \emph{latent information relay}: they can recover information even after the relevant source tokens have left the cache. Both models incorporate sliding-window attention in their published architectures, an association that motivates testing whether training with sliding windows promotes more reliable information retention.
1 Introduction
The section asks whether information from tokens outside a fixed KV window can persist in retained states and remain useful. It motivates and tests this possibility by comparing sliding-window KV inference with fixed-window inference.
1 Introduction
- The study tests whether information outside the current KV window propagates into retained states, how much survives, and over what distances.Three experiments compare sliding-window KV inference with standard fixed-window inference.Source: §1 · p. 2
- Causal attention can write earlier-token information into later residual-stream activations, and repeated computation through depth can carry that influence forward.This separates where information originates from the cached states in which it is later represented.Source: §1 · p. 2
- Under fixed-window inference, recomputing after a source token leaves the window loses dependence on it, whereas retained cached states may preserve its influence.The possibility follows because cached states were computed while earlier context was still available.Source: §1 · p. 2
- Sliding-window KV inference keeps memory bounded while potentially preserving information from tokens that have left the cache, making it an intermediate regime between growing-context attention and recurrent state.The section presents this as a possible route to useful recurrent memory in pretrained transformers.Source: §1 · p. 2
2 Related Work
Related work addresses bounded-cache inference, recurrent or compressed transformer memory, and hybrid architectures. These approaches differ in which cached states they retain, reuse, or replace with bounded-state computation.
KV Caches and Long-context Inference
- Figure 1 organizes causal computation by token position across columns and transformer depth across rows, with embeddings at the bottom and information flowing upward and rightward.The illustration marks how an earlier token's influence may remain in a later activation after the window shifts.Source: Figure 1 · p. 3, §2 · p. 3
- StreamingLLM retains initial sink-token KV states alongside recent tokens to stabilize attention, while H2O retains tokens with disproportionate cumulative attention.StreamingLLM reports that naive sliding eviction can destabilize inference and that sink-token retention prevents a large perplexity increase.Source: §2 · p. 4
Recurrent and Compressed Transformer Memory
- Transformer-XL reuses previous hidden states across segments, while Rae et al. compress older hidden states into lower-resolution memory rather than discarding them.The section distinguishes these explicit memory approaches from rolling KV, where historical information may remain embedded in retained states.Source: §2 · p. 4
- Hybrid architectures retain some attention alongside recurrent layers: Nemotron-H uses Mamba layers with periodic full attention, and Muse Glimmer alternates sliding-window and full-attention layers.The cited designs pursue bounded-state computation while retaining periodic access to context.Source: §2 · p. 4
3 Background and Inference Setup
The background formalizes causal transformer computation and how attention can transmit information across token positions. It then defines rolling-KV inference, where the cache advances without recomputing retained states.
3.1 Transformer Computation
- Each transformer layer applies causal attention over current and preceding positions, followed by an MLP, and maps the final state to vocabulary logits and a next-token distribution.Figure 1 depicts the computation across token position and depth, with causal information flow proceeding upward and rightward.Source: §3.1 · pp. 4–5
3.2 Information Flow in Causal Attention
- With window width W, a layer can transmit information across at most W−1 token-position steps, so successive layers can extend the possible receptive field into the past.This is a possible dependency path, not a guarantee that useful information is transmitted; retained states may carry earlier information forward as the window advances.Source: §3.2 · pp. 5–6
3.3 Rolling-KV Inference
- Rolling-KV inference ingests tokens sequentially, evicts the oldest position at capacity W, and retains the surviving KV states rather than recomputing them from tokens.Because surviving states were computed with earlier context available, they may contain information influenced by tokens no longer in the cache.Source: §3.3 · p. 6
4 Experiments
Across three secret-code experiments, Rolling-KV retrieval often outperforms recomputing the final window, but recovery after the code’s own KV state is evicted varies by model. Glimmer and Mistral show the strongest retrieval in the tested long-distance settings.
4.1 Experimental Setup
- With W = 512, the experiments compare Rolling-KV’s retained states with Last Window’s recomputation and Full Prompt; models are not trained, and answers are scored among four candidates.Experiment 3 uses 100 Rolling-KV trials and 20 control prompts per model and grid cell.Source: §4.1 · p. 7, §4.4 · p. 10
4.2 Experiment 1: Secret-Code Retrieval
- After eviction at +2, Rolling-KV scores 32%, 47%, 82%, 97%, and 100% for Qwen 0.5B, Qwen 3B, Llama 8B, Mistral 7B, and Glimmer 30B, respectively.Mistral and Glimmer remain at or above 95% through offsets +2 to +5.Source: §4.2 · p. 8, Figure 3 · p. 8
4.3 Experiment 2: Extended Secret-Code Retrieval
- At +10 after code-state eviction, Mistral and Glimmer retain 89% and 96% accuracy, respectively, while both Qwen models are near chance; at +1, Rolling-KV reaches 92.0% versus Last Window’s 27.0%.Experiment 2 lengthens the defining phrase and sweeps offsets from −10 to +10.Source: §4.3 · pp. 8–9, Figure 4 · p. 9
4.4 Experiment 3: Definition-to-Carrier Distance
- Experiment 3 independently varies the filler gap between definition and carrier and the code’s offset from the final window; positive offsets place the code outside both the raw window and retained cache.The carrier is a later marker–code occurrence, and the task tests whether the earlier definition remains useful across these distances.Source: §4.4 · p. 10
- With both the code evicted and the definition-to-carrier gap at least 512 tokens, Glimmer averages 62.9% and Mistral 39.9%, while the other models remain near chance; Last Window is generally near 25% once the code leaves its raw window.The regional means summarize displayed cells rather than a separately sampled test.Source: §4.4 · pp. 10–11, Figure 5 · p. 11
5 Discussion
The discussion finds that Rolling-KV can preserve task-usable information beyond the visible window, including after source tokens are evicted. It distinguishes these retrieval findings from claims of reliable or lossless memory, which the experiments do not establish.
Contextualized Representations as Carriers
- All five tested models benefit when retained code states were computed while the defining instruction was still available, unlike Last Window, which sees the code without that context.The authors call this contextualized representations as carriers.Source: §5 · p. 12
Latent Information Relay
- Mistral 7B and Muse Glimmer sustain high retrieval after both instruction and code leave the window, while Qwen2.5 declines more sharply and Llama 3.1 falls off at larger offsets.The authors call retrieval after code eviction latent information relay.Source: §5 · p. 12
Long-Range Transmission
- With a definition-to-carrier gap of at least 512 tokens, Mistral and Glimmer remain highly accurate while the code is retained; Glimmer stays above chance after code eviction.Performance varies substantially by model and distance.Source: §5 · p. 12
Limitations
- Retrieval is behavioral evidence that some earlier information remains usable, not proof of lossless memory, recoverability of evicted text, or where the information resides.Failure to retrieve also does not show that no information remains; direct interventions on retained states are needed to investigate the mechanism.Source: §5 · p. 12
- Evidence is limited to a synthetic four-choice task, fixed W = 512 cache, and small model set; model differences cannot isolate the cause of performance variation.The association between local sliding attention and stronger retrieval does not establish that training specifically to preserve information after eviction caused the result.Source: §5 · p. 13
Future Work
- The authors propose matched-model tests of rolling-cache training and interventions or circuit tracing to examine information flow; retrieval also serves as behavioral evidence in their safety framing.Successful retrieval does not itself reveal where or how information is represented.Source: §5 · p. 13
AI-Assisted Tools
The authors report using ChatGPT for manuscript editing, proofreading, and LaTeX formatting, and Codex for coding assistance; they reviewed the resulting manuscript and code and take responsibility for them.
AI-Assisted Tools
- ChatGPT assisted with editing, proofreading, and LaTeX formatting, while Codex assisted with coding; the authors reviewed the outputs and take responsibility for the manuscript and code.Source: AI-Assisted Tools · pp. 13–14