PaperlayerView exact revision on arXiv

Source-linked AI summary

KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU

Di Chai, Leye Wang, Zeshen Su, Zhiguo Xia, Zhihang Yu

arXiv:2609.04852v1cs.LG

TL;DR

Long-running agents need to manage workspaces that can exceed both GPU KV capacity and the model’s native context window, while text-based overflow handling may lose details or repeat prefill. KVMEM preserves processed history as tiered KV blocks and retrieves a bounded, query-dependent execution view. Across long-context evaluations, it generally improves utility and efficiency, including 48.4% Pass@1 versus 43.8% with compaction-only management, while supporting 1M-token local workspaces.

  • Problem

    Agent workspaces can exceed GPU KV capacity and the model’s native context window, while compaction may lose fine-grained evidence and text retrieval repeats prefill.

  • Method

    KVMEM preserves processed workspace history as addressable KV blocks across GPU memory, host memory, and NVMe, then retrieves and materializes a bounded execution view at each agent step.

  • Results

    Across three long-context agent benchmarks, KVMEM generally matches or exceeds retrieval-augmented compaction while reducing post-compaction recovery latency by 11.4–53.8×; DeepSWE Pass@1 rises from 43.8% to 48.4%.

  • Takeaways & Limitations

    KVMEM decouples addressable workspace size from GPU-resident KV state, enabling a 1M-token virtual workspace with an 80K-token execution view at approximately 50 tokens/s on a 24 GB laptop GPU.

  • Takeaways & Limitations

    The implementation requires control over KV allocation, retrieval, positional restoration, and tier movement, so it cannot be transparently layered over black-box LLM APIs.

Abstract

from arXiv · show

Modern LLM agents operate in persistent workspaces whose accumulated history can exceed both GPU KV capacity and the model's native context window. Existing systems typically compact older context into summaries or retrieve it later as text, either losing fine-grained execution evidence or repeatedly prefilling content that the model has already processed. We present KVMem, a KV-context virtualization system that preserves overflowed workspace history as paged KV state across GPU memory, host memory, and NVMe. KVMem uses lightweight, model-native attention-space indexes to select relevant historical blocks and materializes a query-dependent execution view bounded by the model's native context window. Extensive evaluations on long-context agent benchmarks spanning histories up to one million tokens, including LongMemEval, MemoryAgentBench, and AgentLongBench, show that KVMem generally achieves higher task utility and greater inference efficiency than compaction-based approaches, the de facto standard for handling context overflow. In the DeepSWE long-context test with Qwen3.8-27B, KVMem improves task success from 43.8% with compaction-only context management to 48.4%. In our local-deployment evaluation, KVMem runs Qwen3.6/3.8-27B NVFP4 with MTP on an off-the-shelf laptop equipped with a 24\,GB RTX 5090 Laptop GPU, virtualizing agent workspaces of up to 1M tokens-four times the model's native 256K-token context window. In a single-session setting, KVMem generates $\sim$50 tokens/s, providing interactive responsiveness for local agent execution. More broadly, by decoupling addressable workspace size from the LLM's native context window, KVMem provides a practical path toward long-running agents whose workspaces can grow beyond that window.

1 Introduction

Long-running agents accumulate workspace context that can exceed GPU KV capacity and the model’s native context window, while text-centric compaction and retrieval can lose execution detail or repeat prefill. KVMEM virtualizes this history as recoverable KV state, selecting bounded execution views and demonstrating higher utility, lower recovery overhead, and million-token local workspaces.

  • Persistent agent workspaces accumulate model-relevant context through repeated observations, tool use, artifact edits, feedback, and intermediate results.
  • GPU capacity can fall below the advertised context window, while accumulated histories can also exceed the model’s native window.Qwen3.6-27B’s native context window is 256K tokens, while long-context trajectories span hundreds of thousands to several million tokens.
  • Compaction and text retrieval can omit later-critical execution details, and retrieved text requires prefill again despite prior processing.Text-centric handling also discards reusable KV computation associated with historical workspace state.
  • KVMEM preserves overflowed history as paged KV across GPU memory, host memory, and NVMe, then materializes a bounded, query-dependent execution view.Relevant blocks are selected with model-derived attention-space indexes and restored in chronological, position-consistent order.
  • KVMEM combines step-level scheduling, KV retrieval, hierarchical placement, and position-consistent rematerialization to manage overflowed agent context.The working set is updated at agent-step boundaries and remains fixed during decoding.
  • Across controlled benchmarks and DeepSWE, KVMEM generally improves utility and reduces recovery overhead while supporting a 1M-token workspace at approximately 50 tokens/s on a 24 GB laptop GPU.The reported DeepSWE evaluation uses Qwen3.8-27B, and the local deployment virtualizes four times the model’s native context window.

2 Background and Motivation

Long-running agents accumulate workspace history that can exceed practical context capacity, while compaction and text retrieval trade off fidelity against repeated computation. KVMEM addresses this by preserving historical KV state for selective reuse instead of reconstructing recalled text.

  • Agent Workspaces and Context Overflow: Agent workspaces accumulate artifacts, tool outputs, dialogue, and execution state that can eventually exceed the model’s active context window.This growth occurs during repeated file reads, tool traces, edits, and discussions.
  • Existing Context Management: Existing workspace-memory systems commonly combine compaction with external text retrieval when summaries do not contain sufficient historical detail.This text-centric recovery path appends selected historical snippets back into the model-visible context.
  • Compaction Can Lose Task-Critical Details: Compaction compresses earlier workspace history before future relevance is known, so later task-critical details may be omitted.Increasing the summary budget reduces information-loss risk but consumes bounded active-context capacity.
  • Text Retrieval Recovers Details but Repeats Prefill: Text retrieval can recover omitted evidence, but recalled tokens require fresh prefill even though the model processed them earlier.The recovery cost grows with the number of retrieved tokens and can offset the benefit of recovering more evidence.
  • KV-State Recovery: KVMEM preserves overflowed workspace history as reusable KV state, enabling later recall through loading and restoration rather than repeated historical-text prefill.The approach retains previously computed model state after it leaves the active execution view.

3 Problem Formulation and Challenges

KVMEM formulates workspace memory as selecting and restoring a bounded historical KV working set from a much larger addressable repository. The design balances fidelity against recovery cost while addressing when, what, and how to recall context under model, GPU, and backing-storage limits.

  • 3.1 Problem Formulation: A workspace accumulates model-visible instructions, artifacts, tool outputs, logs, edits, feedback, and constraints that form task-relevant execution history.
  • 3.1 Problem Formulation: The active execution view is bounded by both the model’s native context window and the GPU capacity available for KV state.
  • 3.1 Problem Formulation: KVMEM preserves previously processed history as KV blocks in host memory and NVMe, rather than reducing overflowed context exclusively to compact text.
  • 3.1 Problem Formulation: Logical KV blocks are the units of retrieval scoring, selection, and movement within the addressable workspace.
  • 3.1 Problem Formulation: The active-context budget combines the model context limit and GPU KV limit, while the backing workspace provides additional host-memory and NVMe capacity.
  • 3.1 Problem Formulation: When the backing workspace reaches Bm, the current prototype falls back to compacting older history into text; KV-native reclamation remains future work.
  • 3.1 Problem Formulation: The policy selects a historical working set at recall points while reusing it between recall points when no workspace-wide reconsideration is needed.
  • 3.1 Problem Formulation: Fidelity measures task quality and recovered-state correctness, whereas RecoveryCost measures online working-set update and restoration cost.

4 KVMEM: KV Workspace Virtualization

KVMEM virtualizes workspace history as paged KV blocks and constructs a bounded execution view that changes at agent-step recall points. Its mechanisms schedule recall, select blocks in the serving model’s attention space, and restore them across storage tiers.

  • KVMEM decouples addressable workspace state from GPU-resident KV state by preserving overflowed history as paged blocks across GPU memory, host memory, and NVMe.
  • Step-Level Memory Scheduling: Within-agent-step attention is stable, averaging 0.070 bits KL, while cross-boundary windows average 2.59 bits, or 37.3× higher.
  • Step-Level Memory Scheduling: Step-level scheduling updates the historical working set once per agent step after prefill and before decoding, then keeps it fixed during decoding.
  • Query-Conditioned KV Retrieval: Historical attention is sparse: top-8 blocks capture 66.5% of historical attention mass, and top-16 capture 77.0%.
  • Query-Conditioned KV Retrieval: KVMEM summarizes each historical block with position-independent Mean-K vectors and ranks blocks using the serving model’s own attention-space signals.
  • Tiered KV Management: Tiered KV management restores selected blocks under a fixed GPU budget while handling placement across GPU memory, host memory, and NVMe.
  • Tiered KV Management: The system combines changing query-dependent selection, physical page reuse, tier coordination, and position-consistent restoration into each agent-step execution view.

5 Implementation

KVMEM is implemented in QW3, a native Qwen inference engine that jointly controls execution, paged KV allocation, attention page tables, positional encoding, and transfers. Its block manager and tiled retrieval design support scalable, remapped KV execution views.

  • QW3 provides KVMEM with end-to-end control over model execution, KV page allocation, positional encoding, and storage transfers.
  • Unlike standard serving abstractions with monotonically growing request-prefix positions, KVMEM constructs a changing query-dependent execution view at every agent step.
  • QW3 is a C++ and CUDA Qwen engine supporting paged KV allocation, attention page tables, transfer streams, continuous batching, prefix reuse, and MTP.
  • Logical KV blocks align with physical KV-page boundaries, allowing retrieval, movement, eviction, and remapping without splitting physical pages.
  • The complete attention-space index resides in host memory, while fixed-size tiles are streamed through a bounded GPU buffer for scoring.
  • The block manager computes working-set differences and orders selected blocks chronologically before assigning contiguous positions and applying positional remapping.
  • Incremental indexing computes Mean-K representations while completed-block K vectors remain resident on the GPU, avoiding an additional historical-KV scan.

6 Evaluation

Across long-history benchmarks, KVMEM preserves or improves task utility while substantially reducing historical-recovery latency relative to compaction-based methods. Its paged KV design also supports million-token workspaces with bounded GPU execution views and interactive consumer-device generation.

  • Utility: KVMEM consistently outperforms Sliding Window and Compact-only, while generally matching or exceeding Compact+RAG utility without reconstructing recalled history from text.It preserves previously processed workspace state and selectively recalls relevant KV blocks.
  • Utility: 85.6% answer accuracy on LongMemEval-S is within 1.0 percentage point of Full Context at 86.6% and 0.6 points of Compact+RAG at 86.2%.Sliding Window and Compact-only achieve 26.8% and 45.6%, respectively.
  • Utility: 60.87% task success on AgentLongBench is highest among evaluated methods, exceeding Full Context at 59.54% and Compact+RAG at 47.49%.The comparison concerns trajectories within the native 256K-token context window.
  • Beyond the native window: 40.99% on MemoryAgentBench histories longer than 256K tokens exceeds Compact+RAG at 34.80%, Compact-only at 27.54%, and Sliding Window at 17.95%.On AgentLongBench, KVMEM maintains 53.0% task success at 512K tokens and 50.0% at 1M tokens.
  • Efficiency: 0.38–1.81 s pre-answer latency is 11.4–53.8× lower than Compact+RAG across five settings when compaction cost is included.Compact+RAG requires 26.63–416.38 s including compaction and 10.63–39.24 s excluding summary generation.
  • Consumer deployment: 1M-token workspaces run at around 50 generated tokens/s on a laptop with a 24 GB RTX 5090 Laptop GPU.KVMEM exposes an 80K-token execution view while virtualizing a workspace four times the model’s native 256K-token context window.
  • Scalability: GPU memory remains nearly constant as workspace size grows because the active execution view is fixed, while persistent KV state and indexes expand across host memory and NVMe.Across the evaluated range, NVMe storage grows from 8.5 GiB to 324 GiB and GPU memory remains approximately 34 GiB.
  • Scalability: KVMEM scales the addressable workspace from 256K to 10M tokens while keeping the GPU-resident execution view fixed at 64K.The 10M-token scale is the evaluated upper bound, with further growth depending on backing-storage capacity.

7 Discussion and Limitations

KVMEM’s benefits come with bounded execution views, retrieval-dependent fidelity, and substantial backing-storage costs. Its DeepSWE efficiency gains are measured against compaction-only context management, while deployment currently requires system-level control over KV handling.

  • Scope of Workspace Virtualization: KVMEM still limits each model invocation to a bounded execution view rather than jointly attending to the full virtual workspace.The 1M- and 10M-token figures describe addressable virtual workspaces whose query-relevant state is materialized per step.
  • Fidelity of KV Reuse: Reused KV state is not mathematically equivalent to recomputing historical text under the newly assembled context.KVMEM uses re-RoPE and re-prefills the current query, but historical KV was computed under its earlier causal context.
  • Fidelity of KV Reuse: Retrieval may miss task-relevant blocks, so larger active execution views can reduce reliance on retrieval when resources permit.This boundary follows from materializing only a retrieved subset of the workspace at each step.
  • Storage Overhead: KV state is substantially larger than text, increasing host-memory and NVMe usage, retrieval-index size, and retrieval overhead as workspaces grow.The 10M-token workspace is the largest scale evaluated, not a hard architectural limit.
  • Efficiency: 2.23× prefill speedup and 1.15× agent speedup are reported against the compaction-only Qwen3.8-27B baseline on the first 16 DeepSWE tasks.The table reports per-task averages; the corresponding request-time speedup is 1.18×.
  • Deployment Scope: KVMEM requires control over KV allocation, retrieval, positional restoration, and tier movement, so it cannot transparently layer over black-box LLM APIs.Concurrent multi-user serving and integration into cloud serving stacks remain future directions.

8 Conclusion

KVMEM virtualizes persistent agent workspaces by preserving overflowed history as paged KV and recalling bounded, query-dependent execution views. Across long-context benchmarks and DeepSWE, it improves utility and efficiency while supporting million-token workspaces on consumer hardware.

  • Approach: KVMEM decouples an agent’s addressable workspace from GPU-resident KV state by preserving history across GPU memory, host memory, and NVMe.It uses step-level scheduling, query-conditioned retrieval, and tiered KV management to avoid repeated historical-text prefill.
  • Benchmark Results: KVMEM generally outperforms sliding-window and compact-only management across three benchmarks and reduces post-compaction recovery latency by 11.4–53.8×.It generally matches or exceeds retrieval-augmented compaction.
  • DeepSWE Results: 48.4% Pass@1 and 93.8% Pass@4 are achieved by KVMEM versus 43.8% and 81.3% with compaction-only management in paired DeepSWE evaluation.The evaluation uses Qwen3.8-27B across 16 tasks.
  • Deployment Results: A 1M-token virtual workspace with an 80K-token execution view runs at approximately 50 tokens/s on a laptop with a 24 GB RTX 5090 Laptop GPU.On a server platform, KVMEM scales the Qwen3.6-27B workspace to 10M tokens while keeping the execution view bounded.
  • Conclusion: These results provide a practical path toward substantially larger persistent workspaces on commodity hardware.The conclusion states that workspaces can grow beyond both GPU KV capacity and the model’s native context window.
Loading 2609.04852v1…