PaperlayerView exact revision on arXiv

Source-linked AI summary

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Piękos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, Jürgen Schmidhuber

arXiv:2610.15715v1cs.AIcs.LG

TL;DR

Linear attention’s fixed-size recurrent state must handle both ongoing computation and preservation of evidence from a growing history. NoteBank adds a sparse archive addressed with native features and replays retrieved text through the backbone; at one million tokens it substantially improves retrieval across recurrent architectures with negligible added analytical computation.

  • Problem

    A fixed-size recurrent state must support ongoing computation while preserving evidence from an ever-growing history, even when future questions make that evidence relevant only later.

    Source: §1 · pp. 1–2

  • Method

    NoteBank archives selected text before questions are known, addresses it with backbone-native features, and reads retrieved evidence through native forward replay.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    37.8–68.6 percentage-point gains at 1M-token retrieval across three linear attention architectures, with benefits also reported on real-world tasks and a strong post-trained hybrid.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The long-context capability of pretrained recurrent models can substantially exceed what their conventional fixed-state memory interface exposes.

    Source: §1 · pp. 2–3

    Generative AI assisted code implementation and debugging and manuscript editing; authors reviewed and tested the code and manually reviewed AI-assisted manuscript content.

    Source: AI USE STATEMENT · p. 11

Abstract

from arXiv · show

Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: \emph{recurrent computation need not bear the full burden of long-term storage}. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This \emph{training-free, plug-and-play} interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds $\leqslant$0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.

1 INTRODUCTION

Linear attention compresses history into a fixed-size state, which must support both ongoing computation and long-term evidence retention. NoteBank separates archival storage from recurrent computation, recovering million-token retrieval performance with negligible added computation.

1 INTRODUCTION

  • A fixed recurrent state must support evolving computation while preserving evidence from an ever-growing history, risking irreversible loss of future-relevant information.Prior work identifies finite-state copying limits, recency bias, over-smoothing, and associative-recall gaps in efficient models.Source: §1 · pp. 1–2
  • At one million tokens, NoteBank raises Gated DeltaNet’s mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%.Figure 1 compares NoteBank with no NoteBank across six retrieval tasks for three pretrained 1.3B recurrent models.Source: Figure 1 · p. 2, §1 · p. 2
  • At one million tokens, NoteBank adds at most 0.025% layer-level analytical FLOPs, while GDN context storage is approximately 1/2,600 of a matched MHA KV cache.The Figure 1 storage comparison is at matched width, depth, and BF16 precision.Source: Figure 1 · p. 2, §1 · p. 2
  • NoteBank archives selected spans before questions are known, uses backbone-native features for addressing, and replays retrieved evidence through the backbone’s ordinary forward computation.It is training-free and requires no external embedding model, separately trained retriever, or separate neural reader.Source: §1 · pp. 2–3

2 PRELIMINARY

Linear attention carries processed history in a recurrent key-value state, which is updated by keys and values and read by queries. In a context-then-question setting, the answer can use only information preserved in that state.

2 PRELIMINARY

  • Linear-attention models compress processed history into recurrent key-value states rather than retaining a separate representation for every past token.The backbone’s existing transformations produce query, key, and value vectors at each token position.Source: §2 · p. 3
  • Keys and values update the recurrent memory, while queries read it; the state supports both ongoing computation and access to past information.Gated and delta-rule variants modify the update, while standard additive linear attention uses A_t = I and β_t = 1.Source: §2 · p. 3
  • When a question follows a context, access to that context is limited to what remains in the retained inference state.The model must preserve evidence before knowing which information the later question will require.Source: §2 · p. 3

3 NOTEBANK

NoteBank keeps a sparse archive alongside the backbone’s unchanged recurrent state updates. It writes and retrieves notes with native features, replays retrieved text through the backbone, and bounds replay length while archive storage and matching grow with history.

3.1 DECOUPLING RECURRENT COMPUTATION FROM ARCHIVAL MEMORY

  • The archive stores selected text spans with compact addresses derived from native query features, and archived text persists independently of subsequent state updates.The backbone’s recurrent state updates continue unchanged during context processing.Source: §3.1 · p. 3

3.2 A NATIVE MEMORY INTERFACE

  • The writer scores candidate windows by native-key distinctiveness, selects spans subject to a token budget, and stores original token IDs with one address per layer-head channel.Each channel chooses the span token with the highest key-distinctiveness score and stores a compact version of its native query; different channels may select different positions.Source: §3.2 · p. 4
  • Question queries retrieve up to R notes by channel votes, which are concatenated in document order and replayed from a fresh copy of the retained state before question answering.Replay uses the backbone’s native forward computation and requires no separate neural reader; optional LoRA adapters can be enabled for replay and question answering while pretrained weights remain frozen.Source: §3.2 · pp. 4–5

3.3 MEMORY AND COMPUTATIONAL OVERHEAD

  • Archive storage and exhaustive matching grow linearly with history, while replay adds at most R × B tokens and its computation depends on the backbone and attention policy.Each note stores at most B token IDs and addresses for indexed layer-head channels; the replay budget remains constant as history grows.Source: §3.3 · p. 5

4 RECOVERING LONG-CONTEXT CAPABILITY

The evaluation tests whether NoteBank can restore long-range retrieval without weight updates and improve real-world task performance across recurrent and hybrid models. It compares training-free and adapted variants across synthetic retrieval, LongBench, MemoryAgentBench, and LongBench-v2.

4.1 TRAINING-FREE LONG-RANGE RETRIEVAL ACROSS ARCHITECTURES AND LENGTHS

  • At 1M tokens, NB-TF exceeds Plain by 68.6, 66.6, and 37.8 percentage points on GDN, GDN-2, and Mamba-2, respectively.Across six RULER tasks, NB-TF maintains roughly 70% accuracy on GDN and GDN-2; Plain and JRT approach zero as context grows.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 FROM SYNTHETIC RETRIEVAL TO REAL-WORLD TASKS

  • NB-TF raises overall LongBench-v1 scores from 15.49 to 16.07 on GDN and from 6.79 to 13.43 on Mamba-2; NB-LoRA raises them to 23.03 and 17.10.GDN NB-TF reaches 16.07, comparable to the 16.05 Transformer reference; NB-LoRA underperforms NB-TF on summarisation.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 PLUG-AND-PLAY ON OFF-THE-SHELF LLMS

  • NB-TF improves Mamba-2 on MemoryAgentBench from 0.0 to 14.7 and Qwen from 55.7 to 56.8 there, while Qwen rises from 46.1 to 49.3 on LongBench-v2.The reported plug-and-play gains require no additional training.Source: Table 2 · p. 6, §4.3 · p. 7

5 ALMOST A FREE LUNCH?

The section measures NoteBank’s added computation, persistent storage, and runtime cost. Its analytical overhead is very small, storage remains compact, and measured latency increases modestly.

5.1 NEGLIGIBLE MARGINAL COMPUTATION

  • At 1M tokens, NoteBank adds 0.0171%-0.0250% to full-model analytical FLOPs, with routing accounting for 0.50-1.72 GFLOPs.Relative overhead decreases as fixed question-encoding and replay costs are amortised over longer contexts; writing and routing remain small linear terms.Source: §5.1 · p. 7, Figure 4 · p. 8

5.2 COMPACT PERSISTENT STORAGE

  • At 1M tokens, GDN’s total persistent context storage is 74.5 MiB, approximately 1/2600 of a matched BF16 full MHA KV cache.NoteBank adds 24.4 MiB of GPU addresses and helper tensors, plus approximately 1.0 MiB of CPU text and metadata; accounting excludes weights and transient allocations.Source: §5.2 · p. 8, Figure 4 · p. 8

5.3 PRACTICAL RUNTIME AND MEMORY

  • At 1M tokens, GDN latency rises from 125.11 to 140.64 s and peak GPU memory from 2.93 to 3.27 GiB with NoteBank.Across matched measurements from 4K to 1M, NoteBank increases latency by 10.6%-13.2%.Source: §5.3 · p. 8, Figure 4 · p. 8

6 UNDERSTANDING NOTEBANK: EVIDENCE ACCESS AND UTILISATION

The section separates evidence preservation, retrieval, and reader use to examine what limits NoteBank. The interventions show that stored evidence can remain inaccessible, routing headroom varies by backbone, and read adaptation contributes differently across tasks.

6.1 WHAT IS PRESERVED, AND WHAT IS RETRIEVED?

  • At 1M tokens, oracle routing raises GDN’s RULER score from 75.0 to 75.7 and Mamba-2’s from 40.4 to 61.6 with archive, reader, and replay budget fixed.For GDN, retrieval closely tracks archival coverage; Mamba-2 leaves more archived gold values unretrieved.Source: §6.1 · p. 8, Figure 5 · p. 9
  • On four LongBench QA tasks, replaying one answer-bearing note improves Mamba-2’s macro score by 3.01 points frozen and 17.34 with LoRA; GDN gains 0.54 and 3.47.Answer containment is a proxy for relevant evidence, not guaranteed supporting evidence or a performance upper bound.Source: §6.1 · p. 9, Figure 5 · p. 9

6.2 HOW MUCH DOES ROUTING MATTER IN PRACTICE?

  • At 1M tokens on RULER, BM25 raises Mamba-2 from 37.6 to 52.7; on LongBench, it raises Mamba-2 from 14.25 to 18.66 with a fixed LoRA reader.Native routing outperforms random selection, mean-pooled embeddings, and BGE on both backbones on RULER; its GDN query encoding and routing take 26 ms versus 87 ms for CPU BM25.Source: §6.2 · p. 9, Table 3 · p. 10

6.3 WHAT DOES READ ADAPTATION CONTRIBUTE?

  • At 1M tokens, NB-TF scores 59.8 versus 1.7 for the comparison condition across six RULER retrieval tasks; LoRA yields 58.4, showing long-range retrieval recovery does not require read adaptation.On LongBench, scores increase from 15.49 to 22.72 without NB and from 16.07 to 23.03 with NB; the adapted models differ by 0.31 points, indicating adaptation accounts for most observed gains in this setting.Source: §6.3 · p. 10, Table 4 · p. 10

7 DISCUSSION AND LIMITATIONS

The discussion argues that separating evidence preservation from recurrent computation can extend long-context performance at little additional cost. The appendix describes NoteBank’s archive construction, native addressing, replay readout, optional adaptation, and evaluated backbones.

7 DISCUSSION AND LIMITATIONS

  • NoteBank reports 37.8-68.6 percentage-point gains at 1M-token retrieval across three linear-attention architectures, with negligible marginal arithmetic, compact storage, and modest runtime overhead.The discussion identifies writing, routing, and reading as areas for improvement; gains vary across tasks and backbones.Source: §7 · p. 10

A RELATED WORKS

  • Unlike methods that retrieve retained states or cached values, NoteBank archives raw historical tokens and replays them through the backbone’s native input-processing path.Related work contrasts NoteBank’s replay with InfLLM’s cached neural representations and describes other retrieval and memory-caching approaches.Source: A RELATED WORKS · p. 17

B.1 ALGORITHMIC OVERVIEW

  • At query time, NoteBank processes the question on a temporary state copy to select notes, then replays retrieved text and processes the question again on the answer branch.The algorithm overview abstracts span selection, address construction, and matching into named operations.Source: §B.1 · p. 17, §2 · p. 18, §3 · p. 18

1. Write

  • Before the question is known, the writer selects text spans at completed chunk boundaries and stores their token IDs with addresses derived from native features.For selected notes, distinctiveness scores choose representative positions, while compacted native query features form the stored addresses.Source: §B.2 · p. 17, §1 · p. 18, §B.3 · p. 19

B.2 WRITE OPERATION DETAILS

  • NoteBank scores candidate windows by token distinctiveness, starts a span with the sentence around the highest-scoring window, then expands it within a token budget.For each span, the stored original token IDs are paired with an address for each layer-head channel.Source: §3 · p. 18

B.3 NATIVE ADDRESS CONSTRUCTION

  • Address compaction uses fixed coordinate selection with no trainable parameters; replay uses original token IDs and continues position indices from the context boundary.The same coordinate mapping is used for notes and questions; replayed text is processed through the backbone’s configured attention policy.Source: §B.3 · p. 19, §B.4 · p. 19

B.4 REPLAY READOUT AND OPTIONAL ADAPTATION

  • Optional low-rank adapters keep pretrained weights frozen and are enabled during replay, question processing, and answer generation, but not context processing or retrieval.Only answer tokens contribute to the adapter loss; the context state and discrete note selection are treated as constants.Source: §B.4 · p. 19

C.1 BACKBONES

  • Experiments include 1.3B GDN, GDN-2, and Mamba-2, Mamba-2-8B, and hybrid Qwen3.5-9B; all NoteBank experiments keep pretrained backbone weights frozen.For Qwen3.5-9B, NoteBank uses features from its Gated DeltaNet blocks for writing and addressing, while replay passes through the entire hybrid backbone.Source: §C.1 · p. 20

C.2 BENCHMARKS AND EVALUATION PROTOCOLS

The appendix specifies evaluation protocols across four benchmarks, NoteBank and baseline configurations, Mamba-2 state handling, and the adaptation setup.

C.2 BENCHMARKS AND EVALUATION PROTOCOLS

  • RULER-NIAH evaluation uses six retrieval tasks, 100 fixed examples per task, and an unweighted mean at each of five lengths from 4K to 1M tokens.The lengths are 4K, 16K, 64K, 256K, and 1M.Source: §C.2 · p. 20
  • LongBench-v1 evaluation covers 12 subsets with 30 shared examples each and reports the unweighted macro-average of task-specific metrics.The 12 subsets span question answering, summarisation, few-shot tasks, and code completion.Source: §C.2 · p. 20
  • MemoryAgentBench evaluation covers 11 Mamba-2-8B task configurations and 2,100 questions, comparing plain inference, clean-start, and NoteBank at retrieval budgets R=1 and R=4.Input lengths range from approximately 7K to 746K tokens.Source: §C.2 · p. 20
  • LongBench-v2 evaluation uses 375 Qwen3.5-9B examples from four specified subsets, retaining only complete prompts that fit the position budget.The evaluation uses the official zero-shot prompt and answer parser; it covers a subset, not the full benchmark.Source: §C.2 · p. 20

C.3 NOTEBANK AND BASELINE CONFIGURATIONS

  • Short-note experiments use up to two 64-token notes per chunk, with chunk sizes of 512 for GDN/GDN-2 and 256 for Mamba-2, and retrieval budgets R∈{1,2,4}.The plain baseline processes the original prompt; JRT-Prompt processes context and question twice before answering.Source: §C.3 · p. 21
  • Mamba-2 state magnitude can explode and degrade readout on very long inputs before replay; NoteBank resets the recurrent state before replay, and its advantage remains under matched rescaling controls.The reset retains convolution caches and position information; resetting alone also improves over unmodified Mamba-2.Source: §C.3 · p. 21

C.4 ADAPTATION SETUP

  • The adaptation setup fine-tunes a shared per-backbone adapter on 2,400 examples from a 5,855-example pool, using fewer than 9.3M tokens and no task-specific fine-tuning.The setup uses LoRA rank 8, scale 16, and 600 AdamW updates.Source: §C.4 · p. 21

C.5 IMPLEMENTATION AND EFFICIENCY MEASUREMENT

The appendix details the efficiency-measurement protocols and estimates NoteBank's added FLOPs, including the scope and exclusions of that accounting.

C.5 IMPLEMENTATION AND EFFICIENCY MEASUREMENT

  • Efficiency timing uses an H100 GPU at BF16 and batch size 1, measuring prompt processing plus exactly 32 generated tokens with archive construction, probing, routing, and replay included.The archive is rebuilt for every run; model loading, initial tokenization, warm-up, and result serialization are excluded.Source: §C.5 · pp. 21–22
  • Peak GPU memory is measured as an absolute allocation including resident weights and temporary allocations, while analytical persistent-storage and host-storage estimates are reported separately.The fixed-output cost measurements are independent of task-specific quality-evaluation protocols.Source: §C.5 · p. 22

C.6 ADDED FLOPS ACCOUNTING

  • The added-FLOPs estimate compares NoteBank with the same backbone from context ingestion through answer generation, using recorded input, archive, and replay lengths.The accounting uses the corresponding Plain answer length to isolate method overhead.Source: §C.6 · p. 22
  • 0.025% added FLOPs are reported for GDN with top-2 replay on 600 RULER examples at the 1M target length.Mean Plain and additional costs are approximately 2891.4 and 0.722 TFLOPs, respectively.Source: §C.6 · p. 23
  • The reported FLOPs estimates cover principal arithmetic but exclude most pointwise operations, data movement, and evaluator work; they use recurrent-equivalent costs and do not measure latency.LoRA is excluded, and hybrid backbones incur replay-induced attention costs.Source: §C.6 · p. 23

C.7 PERSISTENT MEMORY ACCOUNTING

The appendix accounts for NoteBank's persistent payload against a matched full MHA KV cache and states the scope of that comparison.

C.7 PERSISTENT MEMORY ACCOUNTING

  • The reference is a full BF16 MHA KV cache matched in hidden width and layer count, with keys and values counted at two bytes per element and no KV-head sharing.The comparison is made at the start of answer generation and includes the entire archive.Source: §C.7 · p. 23
  • At the 1M target length, the reported GDN payload averages 48 MiB of recurrent states, 1.125 MiB of convolution caches, and 24.959 MiB for the archive.The accounting includes FP32 recurrent states, BF16 convolution caches, and archive payload.Source: §C.7 · p. 23
  • The storage ratio applies to this GDN configuration at 1M and describes payload rather than measured CPU heap usage or peak GPU allocation.The accounting excludes model weights, transient workspace, and encoding/replay cache copies.Source: §C.7 · p. 23

D.1 COMPLEMENTARY RESULTS

Complementary evaluations show that NoteBank improves long-context performance across retrieval and language-model benchmarks, while its added arithmetic remains small. The results also expose task-, backbone-, and routing-dependent trade-offs.

Per-task retrieval results

  • At 1M tokens, NB-TF sustains substantial retrieval accuracy, but gains vary by task: multi-query and multi-value remain harder, and Mamba-2 is more variable.Single-needle retrieval is generally stronger; recovery does not uniformly transfer to multi-query or multi-value tasks under limited replay.Source: §D.1 · p. 24

Comparison with pretrained state-caching methods

  • 16.07 overall on LongBench for off-the-shelf GDN with NB-TF exceeds MARCH’s 14.87 and approaches MC’s 16.50; NB-LoRA reaches 23.03 with fewer than 9.3M adaptation tokens.The state-caching references use 16K training sequences, whereas the evaluated GDN checkpoint was trained on 2K; NB-TF adds no training.Source: Table 5 · p. 25, §D.1 · p. 25

Task-family results and replay budgets on MemoryAgentBench

  • On MemoryAgentBench, Qwen3.5-9B top-4 replay raises overall accuracy from 55.7 to 56.8, while larger replay budgets can help one task and hurt another.For Mamba-2-8B, R=4 raises FactConsolidation from 19.0 to 22.0 but lowers EventQA from 17.9 to 16.4.Source: Table 6 · p. 25, §D.1 · p. 25

State reset and evidence replay in Mamba-2

  • For Mamba-2-8B, clean start raises overall accuracy from 0.0 to 13.0; NB-TF further reaches 14.5 with top-1 and 14.7 with top-4 replay.Resetting accounts for much of the recovery, with retrieved evidence adding task-dependent benefits.Source: Table 6 · p. 25, §D.1 · p. 26

Efficiency breakdown

  • At 1M tokens, NB-TF adds 0.0171%–0.0250% to full-model analytical FLOPs; writing and indexing reuse native features, while replay computation is limited to questions and retrieved evidence.Routing costs are 1.09, 1.72, and 0.50 GFLOPs for GDN, GDN-2, and Mamba-2, respectively; these estimates do not imply equivalent latency overhead.Source: Figure 7 · p. 26, §D.1 · p. 26

Practical runtime and memory discussion

  • At 1M tokens, GDN with NoteBank scores 70.33 on RULER versus SW4K’s 1.17; the comparison spans systems, while paired GDN measurements isolate NoteBank’s incremental cost.The table reports measured peak GPU memory, including model weights, and compact NoteBank archive payload accounting.Source: Table 7 · p. 26, §D.1 · pp. 26–27

Retrieval: quality vs cost

  • Native routing takes 26 ms, while CPU BM25 takes 87 ms and achieves the highest RULER scores; these are implementation costs, not matched per-query timings.BGE timing includes full-bank encoding, so the reported scopes differ.Source: Table 8 · p. 27, §D.1 · p. 27

D.2 ABLATION STUDY

Ablations show that retrieval depends on how many notes are written and replayed, as well as note granularity. Optimal settings vary with backbone, context length, and whether read adaptation is used.

Write budget W

  • Writing W=2 notes consistently improves accuracy across both backbones and evaluated context lengths, with little change in analytical added FLOPs but more notes stored.Native feature extraction and distinctiveness scoring are shared across notes.Source: §D.2 · p. 27, Figure 8 · p. 28

Replay budget R

  • At 1M tokens, increasing replay from R=1 to R=4 raises six-task mean RULER accuracy from 59.5% to 75.2% for GDN and from 37.1% to 40.8% for Mamba-2.Gains vary by backbone and context length, and larger replay budgets increase computation.Source: §D.2 · p. 27, Figure 9 · p. 28

Chunk size C

  • At long contexts, C=64 outperforms Mamba-2’s default C=256, while GDN’s C=512 matches or exceeds C=64; C=2048 substantially reduces accuracy.Smaller chunks permit denser archives and increase storage requirements.Source: §D.2 · pp. 27–28, Figure 10 · p. 29

Span budget B

  • At 1M tokens, the best tested span/replay setting, (B,R)=(128,4), reaches 76.8% for GDN and 41.3% for Mamba-2.Longer spans do not consistently help; their effects depend on replay budget and backbone.Source: §D.2 · p. 28, Figure 11 · p. 29

Interaction: replay × span budget

  • With NB-LoRA, GDN averages 23.02 at R=1 versus 21.36 at R=4, yet peaks at 24.13 with (B,R)=(256,2); Mamba-2 peaks at 17.81 with (128,4).The best replay and span settings remain backbone-dependent after read adaptation.Source: §D.2 · pp. 28–29, Figure 12 · p. 30

Related papers

Loading 2610.15715v1…