Separating Stream Stability from Long-Term Recall in Language Models

1 Introduction

• A growing KV cache makes ordinary dense attention a poor fit for continuous streams, while naively evicting old states from a fixed window can collapse generation quality.The setting includes dialogues, agents, and services that run beyond the original prompt.Source: §1 · p. 1
• Fluent generation can continue even after the model forgets old facts, so perplexity, needle retrieval, and task accuracy answer different questions.Streaming, long-context, and memory claims can therefore describe different capabilities.Source: §1 · p. 1
• The stability horizon measures how long prediction remains well behaved, the access horizon how far back information can affect output, and the utility horizon how far back useful task information can be placed.Source: §1 · p. 1
• THREEH separates unbounded stream stability from memory and proposes a budget-matched protocol for evaluating all three horizons.The framework also considers systems combining attention sinks with recurrent state or retrieval.Source: §1 · p. 1

2.1 Setup

• A streaming system retains state M_t under a budget B, updates it as tokens arrive, and predicts the next token from that state.Persistent state may include KV vectors, recurrent activations, compressed summaries, retrieved passages, or mixtures.Source: §2.1 · p. 2
• Stability is measured by whether excess predictive loss relative to a declared reference stays within a threshold as the stream continues.The paper allows other local generation criteria in place of loss.Source: §2.1 · p. 2
• Access is causal: changing an earlier token under a fixed counterfactual continuation must be able to change the output, and a pure sliding window loses non-initial content after W tokens.Retained initial sink tokens are fixed-position exceptions, not a general historical channel.Source: §2.1 · p. 2
• Utility measures task performance when decisive information is placed at lag d, and can fall below architectural access because accessible information may be ignored.Utility also depends on task, interference, prompting, and decision rule.Source: §2.1 · p. 2

2.2 Stability does not imply memory

• A bounded-state autoregressive model can have infinite stability while access and delayed-recall utility remain bounded by W.The construction uses a stationary order-W process and a delayed-recall answer unavailable in the retained state.Source: §2.2 · p. 2
• Token prediction dominated by local statistics can remain stable even when a memory task depends on remote variables.Thus an infinite stability horizon is compatible with a short semantic horizon.Source: §2.2 · p. 2

3 What Attention Sinks Do

• Retaining initial attention-sink states keeps attention geometry closer to the pretrained regime and can make long-running inference stationary with constant KV memory.The recent window need not be rebuilt, but initial states are computed before later events arrive.Source: §3 · p. 2
• Attention sinks do not encode arbitrary later events, so evicted tokens need another state-update or retrieval path to affect future outputs.Source: §3 · p. 2, Figure 1 · p. 3
• Methods that extend attention distance or reduce long-prompt processing cost do not guarantee reliable use of all available evidence.The paper distinguishes these methods from cache retention or loading, where semantic retention remains an empirical question.Source: §3 · p. 3

4.1 Experimental setup

• The evaluation streams 128K tokens, normalizes perplexity to each method’s 2K value, inserts four-way random bindings at lags 0.25W–8W, and delays evidence for three-action decisions over those lags.Targets never occupy fixed sink positions, and each lag contains 500 test examples.Source: §4.1 · p. 3

4.2 Attention sinks stabilize the stream

• At 128K tokens, StreamingLLM reaches 1.04× normalized perplexity versus 1.02× for recomputation, without recomputation’s 7.8× latency.Adding retrieval leaves stream stability nearly unchanged at 1.05×.Source: §4.2 · p. 3

4.3 Stability and retention separate

• At 2W, Window and StreamingLLM each reach 26% recall, while recurrent memory reaches 70% and retrieval 89%.StreamingLLM’s 1.04× perplexity pairs with 25% remote recall, showing that stable local modeling does not imply a long semantic access horizon.Source: §4.3 · p. 3, Figure 2 · p. 4

4.4 Task utility has its own horizon

• At 2W, StreamingLLM decision accuracy is 36%, near the 33.3% chance rate; retrieval reaches 73% at 8W, below its binding recall.Recurrent memory has a smoother but lower utility tail.Source: §4.4 · p. 4

4.5 Ablations and robustness checks

• More sink tokens improve stability only to a small saturation point, while cache size shifts the recall cliff in absolute tokens and retrieval depth improves remote recall at query cost.The experiments’ overall takeaway is that sinks support stable, efficient streaming, whereas recurrent and retrieval mechanisms extend access with utility still below raw access.Source: §4.5 · p. 4, Table 3 · p. 4

5 THREEH: A Three-Horizon Evaluation Contract

• THREEH asks systems described as streaming, infinite-context, or long-term memory to report three tracks under one declared resource budget.The protocol is compatible with sink-window, recurrent, compressed, and retrieval-augmented systems.Source: §5 · p. 4

5.1 Track A: stream stability

• Stream stability testing uses long natural-text streams and reports position-bucket predictive loss, worst-bucket degradation, latency, accelerator memory, and persistent state.References should be explicit, and document-boundary effects reported separately.Source: §5.1 · pp. 4–5

5.2 Track B: causal retention

• Causal retention testing inserts facts and queries them after controlled lags, using counterfactual streams that differ only in the target value.This tests whether the output changes appropriately rather than rewarding fluent guessing or dataset priors; suggested checks include interference, updates, placement, query delay, and negatives.Source: §5.2 · p. 5

5.3 Track C: delayed task utility

• Delayed utility testing embeds old information in downstream decisions and reports both task score and the lag-dependent utility horizon.The query should be hidden when stream state is written; retrieval systems should report retrieval frequency, retrieved tokens, and latency.Source: §5.3 · p. 5

5.4 Budget and reporting rules

• Resource accounting should include device-resident state, external stored state, amortized compute per stream token, generation latency, and query-time retrieval latency.A small KV cache paired with a large vector database is a hierarchical memory system, not constant-memory in the same sense as a pure sink window; horizon curves are preferred to a single maximum length.Source: §5.4 · p. 5

Sink-window streaming

• Sink-window streaming offers effectively unbounded stability and constant device memory, with general access limited to the recent window plus fixed initial positions.It suits settings where relevant information is local, continuous fluency matters, or another subsystem handles older state.Source: §6 · p. 5

Shared evaluation accounting

• THREEH tracks should use the same accounting of persistent state and inference cost.Table 4 separates three questions often collapsed into a single context-length measure.Source: Table 4 · p. 6

Recurrent and compressed state

• Recurrent and compressed state can extend access beyond the visible window, but bounded state risks lossy interference because it must preserve content before future queries are known.THREEH targets this regime with update, interference, and delayed-query tests.Source: §6 · pp. 5–6

Retrieval-augmented memory

• Retrieval-augmented memory can access content for as long as the store retains it, but utility depends on indexing, query formation, selection, and downstream use.Storage and retrieval costs belong in the resource budget.Source: §6 · p. 6

Hybrid systems

• A practical hybrid can assign stable local generation to sinks, compact working memory to recurrent state, and sparse historical evidence to retrieval.The framework suggests optimizing stability, retention, and utility in separate modules because one end-to-end score cannot identify which module failed.Source: §6 · p. 6

Precise claims

• Reserve “infinite” for the measured property: bounded-KV infinite-duration generation does not by itself establish arbitrary historical access or delayed task utility.“Infinite context” should require retention evidence beyond the active window, while “long-term memory” should additionally require delayed task utility.Source: §7 · p. 6

Anchors and semantic memory

• A dedicated sink token remains an anchor unless its state is updated with later content; semantic memory needs a write mechanism, an external store, or both.A dedicated sink token may make streaming behavior easier to learn.Source: §7 · p. 6

Training objectives

• Memory training requires losses with delayed-variable targets, preferably under interference and updates, while utility training must expose decisions that use retrieved or compressed evidence.Local next-token loss can reward stream stability without rewarding remote retention.Source: §7 · p. 6

Failure decomposition

• When an old fact fails to affect a decision, failure decomposition distinguishes eviction, compression loss, retrieval failure, and failure to use available evidence.Counterfactual retention pairs test whether an information path remains; oracle retrieval separates access from execution, and full-context controls separate memory failure from task difficulty.Source: §7 · p. 6

Streaming and inference methods

• Position extrapolation, efficient attention, and KV-cache policies address attention behavior, processing cost, or state retention, while this paper argues that semantic access and task utility need separate evidence.The cited approaches include Press et al. (2022), Su et al. (2024), Peng et al. (2024), Ding et al. (2024), Dao et al. (2022), Zaheer et al. (2020), Jiang et al. (2024), Xiong et al., Zhang et al. (2023), Li et al. (2024), Liu et al. (2023), Ge et al. (2024), Tang et al. (2024), and Xiao et al. (2025).Source: §8 · p. 6

Long-context and memory evaluation

• Long-context benchmarks cover retrieval, aggregation, and reasoning over large finite inputs.The passage cites Bai et al. (2024) and Hsieh et al. (2024).Source: §8 · p. 6
• THREEH connects finite-input long-context benchmarks and cross-session conversational-memory benchmarks to streaming through distinct lag curves and shared resource accounting.The cited work includes Bai et al. (2024), Hsieh et al. (2024), Liu et al. (2024a), Maharana et al. (2024), and Wu et al. (2025).Source: §8 · pp. 6–7

Conclusion

• Attention sinks can stabilize bounded-cache generation far beyond a model’s training window but are not, by themselves, semantic memory.The conclusion distinguishes stability from historical access and task utility.Source: §9 · p. 7
• THREEH makes the distinction measurable with matched budgets, lag sweeps, counterfactual controls, and task-level decisions.It replaces a single context-length claim with stability, access, and utility horizons.Source: §9 · p. 7
• The framework’s vocabulary distinguishes what streaming, recurrent, retrieval, and hybrid systems achieve and makes missing capabilities visible.Source: §9 · p. 7

• Horizon measurements depend on the stream, task family, score threshold, and reference system.Source: Limitations · p. 7
• Counterfactual stream edits can become unnatural, perplexity depends on tokenization, and external-memory accounting requires policy choices.Source: Limitations · p. 7
• Experiments cannot recover every causal dependency in arbitrary neural states; they establish only bounded access claims under specified interventions.Source: Limitations · p. 7
• The three horizons leave out privacy, calibration, factuality, and robustness to malicious memory content.Source: Limitations · p. 7