PaperlayerView exact revision on arXiv

Source-linked AI summary

Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute

Sietse Schelpe

arXiv:2610.10845v1cs.CLcs.AIcs.DCcs.LG

TL;DR

A fixed context window excludes earlier text, and serving rebuilds KV state for previously processed prompts. The paper stores per-block KV on encrypted disk for later reuse, reporting zero-recompute restores across 50M tokens that are faster and less energy-intensive than recomputation.

  • Problem

    A fixed context window makes earlier tokens unavailable to attention, while rebuilding KV state for every prompt repeats prefill for text already processed.

    Source: §1 · p. 1

  • Method

    The paper stores each block’s KV state on encrypted disk and restores it when needed, reusing stored state rather than recomputing the block.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    Across both models, all 100 probed blocks restored without recompute; restores were 2.8× to 4.3× faster and used 8.8× to 12.3× less GPU energy than recomputing the same tokens.

    Source: §1 · p. 1, §6.1 · p. 3, §6.2 · p. 3

  • Takeaways & Limitations

    Stored KV can be reused at depths up to 50M tokens while VRAM remains constant, without widening single-pass attention.

    Source: §1 · p. 1

    Only one block is resident at a time, and the durable store requires local NVMe, reaching 1.86 TB for 12B and 6.25 TB for 31B at 50M tokens.

    Source: §1 · p. 1, §8 · p. 4

Abstract

from arXiv · show

A large language model can only use the text that fits in its context window, and it recomputes its internal key-value (KV) state for a prompt every time the prompt is sent. We test a memory layer, the public package galahad-kv, that saves the KV state of each block of about 16,000 tokens to encrypted local NVMe disk and loads it back later, byte-exact, without recomputing it. We ran it on 50,000,000 tokens of real public text, served through vLLM on one NVIDIA H100, with Gemma 4 12B and Gemma 4 31B. Every block we probed was loaded back from the encrypted store with no recompute (100 of 100, at depths from 0 to 50M tokens) on both models. Loading a block was 2.8x to 4.3x faster than recomputing it and used 8.8x to 12.3x less GPU energy, and GPU memory stayed flat over the whole 50M-token stream. Asked about facts planted millions of tokens earlier, the 12B model gave the right answer 82 times out of 100 and the 31B model 98 times out of 100. Neither model made up an answer. The limits are as follows. This is reuse of stored state, not a wider attention window: one block is loaded at a time, and how well a question is answered depends on the model. Writing the memory is a one-time cost, and the store takes terabytes of local NVMe disk. We describe the test protocol, which is built to resist common ways of gaming long-context benchmarks, and give a single-GPU reproduction that uses public software and a free licence for the package.

1 Introduction

A fixed context window makes earlier text unavailable, while serving rebuilds KV state for text already processed. This paper stores and restores per-block KV state as durable long-term memory, measuring 50-million-token reuse and its limits.

1 Introduction

  • A fixed 16k-token context window excludes earlier tokens from attention, and rebuilding KV state for every prompt repeats prefill for previously processed text.Source: §1 · p. 1
  • The proposed long-term memory stores each block’s KV state on disk and restores it when needed, retaining processed material beyond the model’s fixed window.Source: §1 · p. 1
  • 2.8× to 4.3× faster restores and 8.8× to 12.3× less GPU energy than recomputing the same tokens accompany byte-exact restore at every probed depth across 50M tokens.The paper also reports constant VRAM and a cheat-resistant protocol with a single-GPU reproduction harness.Source: §1 · p. 1
  • The claims cover KV reuse at any depth, not wider single-pass attention; only one block is resident at a time and deeper content is restored from disk on request.Source: §1 · p. 1

2 Background and related work

Prior systems cache or offload KV, retrieve text into the context window, or manage which text enters it. This paper distinguishes its durable state reuse from those approaches and measures it separately from Gemma 4’s sliding-window property.

2 Background and related work

  • Unlike ephemeral prefix caches, several prior offloading systems stage KV on CPU or disk to reduce recompute, while this paper describes durable, encrypted, byte-exact state reuse.The cited offloading systems store plaintext and restage it for each request; some are architecture-specific.Source: §2 · pp. 1–2
  • RAG rereads retrieved text in the context window and pays prefill again, while cache-augmented generation must keep its fixed document set within that window.MemGPT-style agent memory instead decides which text is placed in the window.Source: §2 · p. 2
  • Most Gemma 4 layers use approximately 1,024-token sliding-window attention, reducing per-token KV footprint; the memory system benefits from but does not introduce this property.Source: §2 · p. 2
  • Compared with the authors’ prior 6M-token report, this work scales the window to about 8×, serves through vLLM, encrypts KV at rest, and applies a cheat-resistant protocol.The paper deliberately excludes Blaise learned retrieval to measure the window in isolation.Source: §2 · p. 2

3 System under test

The system under test is galahad-kv integrated with vLLM through its GalahadConnector. It deposits encrypted per-block KV to disk and later grafts a stored block back into the forward pass without recomputing its tokens.

3 System under test

  • The engine under test is the public galahad-kv==1.31.4 package, loaded by vLLM 0.31.0 through its GalahadConnector KV-transfer plug-in.Source: §3 · p. 2
  • On deposit, the connector writes each request’s per-block KV to an encrypted on-disk store and frees it, leaving one block resident while others remain on disk.The store uses AES-256-GCM with one key per organisation, and this moving window keeps VRAM flat.Source: §3 · p. 2
  • On a later request, restore grafts a stored block’s KV back so the forward pass continues as if that block had just been computed, without recomputing its tokens.Source: §3 · p. 2

4 Anti-cheat protocol

The protocol targets five ways long-context benchmarks can be gamed, using randomized source facts, blind ingestion, planted needles, decoys, and physical-store audits. It separately measures engine restoration and model recall, with a negative control for fabrication.

4 Anti-cheat protocol

  • Five benchmark risks are countered with nonce-mutated facts, blind ingestion, a planted high-entropy record per block, same-format decoys, and audits of on-disk store bytes.The audit checks each block’s MRLNCRY1 magic and agreement between stored bytes and KV-footprint math.Source: §4 · p. 2
  • KV restore measures whether context tokens came from storage without recompute, while text recall measures whether the model emits the exact planted code.GPU energy is measured with the idle-subtracted NVML hardware counter, and the answer key is held out of the engine.Source: §4 · p. 2
  • A negative-control probe asks for a record absent from the block, where the correct behavior is refusal rather than fabrication.Source: §4 · pp. 2–3

5 Experimental setup

The experiment streams 50 million tokens of public text in 3,125 blocks through Gemma 4 12B and 31B on one NVIDIA H100. The measured KV footprints correspond to 1.86 TB and 6.25 TB of encrypted storage, with all deposited blocks retained.

5 Experimental setup

  • The setup uses one NVIDIA H100 with local NVMe to serve Gemma 4 12B (bf16) and 31B (FP8) through vLLM.The 50,000,000-token stream comprises 3,125 blocks of 16,000 tokens from four public-text sources, then is mutated with 100 needle records and same-format decoys.Source: §5 · p. 3

5.1 KV footprint

  • The measured KV footprint is 36.4 KiB/token for 12B and 134 KiB/token for 31B, corresponding to 1.86 TB and 6.25 TB for 50M tokens.The 12B footprint is compared with approximately 160 KiB/token for full attention, which would imply approximately 8 TB.Source: §5.1 · p. 3
  • All 3,125 blocks carried the encrypted-store header, matched the audited byte totals, and remained present without pruning or eviction.The audited store size reflects the measured footprints for both models.Source: §5.1 · p. 3

6 Results

Across 100 probes at depths up to 50M tokens, both models restored stored blocks without recompute; restores were faster and used less GPU energy than recomputing. Recall varied by model, while VRAM stayed flat during the one-time deposit.

6.1 Byte-exact restore at every depth

  • 100/100 probes on each model restored KV with zero recompute; exact-code recall was 82/100 for 12B and 98/100 for 31B, with 0/20 fabricated negative-control answers.The engine restore result is distinct from model text recall; the companion Taliesin work, not this paper, establishes byte-exact logit equality.Source: §6.1 · p. 3

6.2 Restore is faster and cheaper than recompute

  • Restores were 2.8× to 4.3× faster and used 8.8× to 12.3× less GPU energy than recomputing; the 31B speedup and energy reduction were 4.25× and 12.3×.Restore time did not grow with depth; the 31B model’s advantage exceeded the 12B model’s (2.8× speedup and 8.8× energy reduction).Source: §6.2 · pp. 3–4

6.3 Deposit (the one-time cost)

  • The deposit is paid once; during the 50M-token stream, VRAM remained flat, corresponding to an O(1) moving window.Each reuse after deposit was 2.8× to 4.3× cheaper in time and 8.8× to 12.3× cheaper in energy than recompute.Source: §6.3 · p. 4

6.4 What the numbers say

  • Recall quality depends on the reading model: misses selected a wrong line that was genuinely in the block, such as a same-format decoy or another in-block number.The authors report that neither model fabricated a code; they suggest a cheaper model can serve and a stronger one can be escalated to over the same stored memory.Source: §6.4 · p. 4

7 Why this is long-term memory and not a cache

The paper calls its durable, encrypted, byte-exact stored state long-term memory rather than a cache. It persists beyond the context window and lets the model reuse previously processed facts without changing its weights.

7 Why this is long-term memory and not a cache

  • The store is durable, encrypted at rest, byte-exact, and reaches 50M tokens beyond the model’s 16k native window.It survives restart without pruning or eviction; a restored block reproduces the computation rather than serving a lossy summary.Source: §7 · p. 4
  • Unlike a short-lived cache rebuilt from source after a miss, the stored state persists on disk and can be restored later.All 3,125 blocks remain available in the measured store.Source: §7 · p. 4
  • Reuse makes inference faster and less energy-intensive while allowing the model to act on facts processed millions of tokens earlier; the weights do not change.The paper uses “long-term memory for AI” for this change in the set of facts available to the model.Source: §7 · p. 4

8 Limitations and scope

The system reuses stored KV state but does not widen single-pass attention or provide learned retrieval. Its operation also depends on one-block-at-a-time residency, local NVMe storage, and a one-time deposit.

8 Limitations and scope

  • The system does not widen single-pass attention: only one block is resident, and deeper content is restored from disk when requested.Stored depth is restored rather than attended all at once.Source: §8 · p. 4
  • The method reuses KV rather than performing learned retrieval; combining it with semantic passage selection at 50M scale is left for future work.The passage names Blaise as the out-of-scope passage-selection system.Source: §8 · p. 4
  • A per-probe restore takes approximately 0.3 seconds, near the NVML counter’s resolution, so absolute per-probe joules carry a wider uncertainty band.The restore-versus-recompute ratio is described as reliable.Source: §8 · p. 4
  • The 50M-token store requires 1.86 TB at 12B or 6.25 TB at 31B on local NVMe; network storage voids the reported latency and energy figures.Durable memory trades disk space for reuse.Source: §8 · p. 4

9 Reproducibility (single GPU, public software)

The paper provides a public, single-GPU reproduction path and a four-stage harness for building, depositing, auditing, and probing the memory. The harness can be adapted to other datasets, models, and needle designs, and reported figures have raw artifacts and SHA-256 manifests.

9 Reproducibility (single GPU, public software)

  • The reproduction instructions install galahad-kv 1.31.4 with vLLM 0.31.0 and serve a Gemma 4 model using the GalahadConnector.The command sets a maximum model length of 16,640 tokens and enables prompt-token details.Source: §9 · p. 5
  • The public harness supports substitutions of datasets, models, and needle designs, with raw artifacts and SHA-256 manifests for every reported figure.It is a thin HTTP client using transformers, datasets, and urllib.Source: §9 · p. 5

A Harness

  • The four harness stages build a hardened corpus, deposit it blind, audit the encrypted store, and probe with two metrics.The named scripts build the corpus and needles, deposit while measuring VRAM and energy, audit storage, and run probes with a recompute baseline, negative control, and miss classification.Source: §9 · p. 5, A Harness · p. 6

B Data and manifest

The data and manifest materials identify per-depth probe results and storage artifacts, alongside the sizes and block counts of the local NVMe stores.

B Data and manifest

  • The per-depth probe results are provided separately for the 12B and 31B models, alongside deposit logs, an answer key, and a SHA-256 manifest.The named result files are RESULT_12b_probe.json, RESULT_31b_probe.json, and SHA256SUMS_31b.txt.Source: B Data and manifest · p. 6
  • The local NVMe stores are 1.86 TB for 12B and 6.25 TB for 31B.Source: B Data and manifest · p. 6
  • Each model's store contains 3,125 MRLNCRY1 blocks.The stores are on local NVMe.Source: B Data and manifest · p. 6
Loading 2610.10845v1…