PaperlayerView exact revision on arXiv

Source-linked AI summary

Cache Engineering for Agents

Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Hanling Zhang, Yu Wang

arXiv:2610.28577v1cs.CL

TL;DR

Long agent trajectories make KV-cache I/O and storage costly, while editing history can trigger re-prefilling and limit retained information. Cache engineering lets agents control active KV states, and AKV manages history-dependent cache reuse. Tool-response rolling drop improves average benchmark scores while reducing average cost versus conventional threshold-triggered summarization.

  • Problem

    Long agent trajectories make KV-cache storage and I/O costly, while editing history triggers re-prefilling and aggressive reductions risk losing useful intermediate information.

    Source: §1 · p. 1

  • Method

    Cache engineering lets agents drop selected KV from future attention and adjust retained-key positions, while AKV manages cache reuse, eviction, and restoration through a stateless interface.

    Source: §1 · p. 2

  • Results

    27.7% lower average cost and 1.7-point higher average benchmark scores were reported for tool-response rolling drop versus conventional threshold-triggered summarization.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across agent benchmarks, cache engineering improves average task performance while reducing inference costs, and AKV delivers higher serving throughput and lower latency.

    Source: §6 · p. 10

    Further exploration of cache-engineering strategies, training agents under this scheme, and extending serving support to linear-attention models remain future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.

1 INTRODUCTION

Long agent trajectories make KV-cache I/O and storage costly, while editing history can force re-prefilling and create a trade-off between cost and retained information. Cache engineering directly controls active KV states, and AKV supports their history-dependent reuse.

1 INTRODUCTION

  • Editing historical text invalidates subsequent KV states and triggers re-prefilling, while aggressive reductions can discard useful intermediate-round information.This creates a trade-off between re-prefill cost and task accuracy.Source: §1 · pp. 1–2
  • Cache engineering drops selected KV entries from future attention while keeping the text history and reusing retained states without recomputation.Agents can interleave drop and repos operations with append to control the active cache.Source: §1 · p. 2, Figure 1 · p. 2
  • Identical token prefixes and retained positions can yield different cache states under different cache-operation histories, making prefix matching alone insufficient for correct reuse.The paper illustrates this with different drop-and-append orders changing what a later token attends to.Source: §1 · p. 2
  • AKV combines a stateless API-compatible interface with an extended radix tree and segment-batch attention for cache reuse, eviction, and restoration.The system internally manages stateful cache operations so agents need not track which KV states remain resident.Source: §1 · p. 2
  • 27.7% lower average cost and 1.7-point higher average benchmark scores versus conventional threshold-triggered summarization were achieved by tool-response rolling drop.The reported comparison spans three LLMs and four agent benchmarks.Source: §1 · pp. 2–3

2 RELATED WORK

Related work addresses agent context reduction, KV-cache compression and management, and serving-system reuse. Cache engineering differs by letting agents make high-level eviction decisions while requiring systems to account for operation histories.

2 RELATED WORK

  • Context-engineering methods summarize or trim agent histories, but editing earlier context requires re-prefilling the retained suffix and creates a cost-versus-information-retention dilemma.Prior approaches include heuristic or agent-triggered summarization, summary-quality improvements, and direct trimming.Source: §2 · p. 3
  • Prior cache-level work compresses, evicts, or reuses KV states using techniques such as low-rank representations, quantization, attention-based importance, position adjustments, or partial recomputation.This paper instead focuses on agent- and harness-level eviction decisions and practical cross-request reuse and isolation.Source: §2 · p. 3
  • Serving systems use prefix caching and methods to batch context edits or overlap re-prefill, while cache engineering additionally requires tracking cache-operation histories.Identical token prefixes can produce different states after different operations.Source: §2 · p. 3

3 CACHE ENGINEERING

Cache engineering lets agents change the active KV cache without rewriting the trajectory, addressing the trade-off between re-prefill cost and retained context. It formalizes cache operations and their sequences, including position changes and history-dependent cache states.

3.2 DESIGN MOTIVATION

  • Directly dropping older tool-response cache improves accuracy while reducing cost, unlike trimming the latest 12 rounds (1.85× full-context cost) or 48k-threshold summaries (about 7% lower cost, 5-point accuracy loss).Figure 3 compares GPT-OSS-120B accuracy and cost on 100 BCP tasks.Source: Figure 3 · p. 4, §3.2 · p. 4

3.1 PRELIMINARIES

  • The active cache is the KV states available to attention, distinct from the trajectory’s logical token sequence and physical cache residency; position encodings can differ from logical indices.Cache engineering changes the active cache while leaving the logical token sequence unchanged.Source: §3.1 · p. 4

3.2 DESIGN MOTIVATION

  • Tool-response rolling drop retains all reasoning and recent tool responses while dropping older tool-response cache, reflecting lower attention to older tokens and tool responses.Figure 2 reports per-token attention to assistant and tool responses, averaged over 10 BCP requests.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Because dropping cache does not reset position indices, repos repositions retained keys and resets the next position index without re-prefilling.For RoPE, repos adjusts cached keys’ position encodings using rotations.Source: §3.2 · p. 4

3.3 CACHE OPERATIONS

  • Append adds a token conditioned on active cache; drop removes selected entries without changing retained positions or the logical sequence; repos reindexes retained entries and resets the next append index.These operations define the cache visible to attention, separately from serving-system storage and reuse.Source: §3.3 · p. 5

3.4 CACHE OPERATION SEQUENCE

  • For a fixed LLM, an operation sequence starting from an empty cache determines cache states; tokens and retained positions alone cannot ensure correct reuse because histories matter.The order of drop and append can change the states computed for later tokens, even when the same entries remain.Source: §3.4 · p. 5

4 AKV SERVING SYSTEM

AKV pairs a stateless serving interface with internal stateful cache management to support cache engineering. Its tree-based reuse and eviction and segmented restoration address cache states that depend on operation history and position encodings.

4.1 SERVING INTERFACE

  • AKV extends the serving API with token and cache-operation histories while keeping each request’s output independent of physical cache residency and other requests’ operations.Clients describe messages plus drop and repos operations; the server restores missing states after eviction.Source: §4.1 · p. 6

4.2.1 DESIGN OBJECTIVES AND CHALLENGES

  • Token prefixes alone cannot guarantee correct cache reuse, dropped entries may remain needed by other requests, and ordinary causal prefill cannot restore operation-dependent states in one pass.AKV addresses these challenges with an extended radix tree, drop-aware eviction, and segment-batch attention, respectively.Source: §4.2.1 · p. 6

4.2.2 DROP-AWARE CACHE MANAGEMENT

  • Drop nodes record where cache entries are removed, preserving prefix sharing while distinguishing later states produced under different drop histories.AKV can evict dropped entries once no computation or restoration needs them, while retaining tree placeholders for descendant reuse.Source: §4.2.2 · pp. 6–7
  • Segment-batch attention restores missing cache states in one prefill pass by splitting queries at drop points and attending to each segment’s active history.Only attention is segmented; the FFN processes the full sequence as one batch, avoiding repeated FFN weight reads.Source: §4.2.2 · p. 7

4.2.3 REPOSITION-AWARE CACHE MANAGEMENT

  • Repos creates positional cache versions by rotating copied keys and copying values, while the radix tree distinguishes versions for reuse and independent eviction.Restoration uses each segment’s original position encodings and temporary KV versions.Source: §4.2.3 · p. 7

5 EXPERIMENT

Across three models and four agent benchmarks, AKV has higher average success rates and lower inference costs than both context-engineering baselines. Its serving and cache-policy evaluations examine throughput, latency, repos, and policy choices.

5.1 SETUP

  • The evaluation compares Standard’s model-specific 96K- or 192K-token summarization thresholds with Text Engineering’s 48K threshold and a rolling-drop policy retaining the latest 12 tool responses.It covers three models and four long-horizon agent benchmarks, reporting success rate, per-task API cost, and serving metrics.Source: §5.1 · p. 8

5.2 PERFORMANCE

  • AKV’s accuracy advantage appears beyond baseline summarization thresholds: Text Engineering is lowest at 48–96K logical tokens, and Standard is lowest above 96K.Below 48K logical tokens, the three methods have comparable accuracy; the pattern is described as consistent with information loss from abrupt reductions.Source: §5.2 · pp. 8–9
  • AKV improves average success rates by 1.3–2.1% and 0.9–8.7% versus the two baselines, respectively, while reducing inference cost by 27.7% versus Standard and 21.9% versus Text Engineering.These averages cover three models and four benchmarks.Source: §5.2 · p. 8, Table 1 · p. 9

5.3 EFFICIENCY

  • At peak-throughput settings, AKV achieves 1.36× mean and 1.51× P95 request-latency speedups over Standard, with 1.53× input and 1.18× output throughput.The reported advantages widen at concurrency C ≥14 as Standard’s re-prefill work increases.Source: §5.3 · pp. 9–10, Table 2 · p. 9
  • On the first 100 SWE-bench Verified tasks, AKV reaches the highest final pass rate in the least GPU time and ultimately outperforms both baselines as logical input tokens accumulate.The online comparison uses GPT-OSS-120B at each method’s peak-throughput concurrency.Source: §5.3 · p. 10

5.4 ABLATIONS AND CACHE ENGINEERING POLICIES

  • Tool-response drop with repos achieves the highest pass rate on both ablation benchmarks; adding repos raises BCP pass rate from 55% to 62%.RSI policies reduce per-task cost by 15.6% and 17.8% versus Standard on held-out tasks; cache management as a tool yields the highest pass rates on a 100-task BCP subset for both models.Source: §5.4 · p. 10, Table 3 · p. 10, Table 4 · p. 10

6 CONCLUSION

The conclusion presents cache engineering and AKV as a way to manage active KV states without recomputing retained caches, with benchmark gains in task performance, inference cost, throughput, and latency. Further strategy exploration, agent training, and support for linear attention remain future work.

6 CONCLUSION

  • Cache engineering lets agents directly manage active KV states while reusing retained caches without recomputation, supported by AKV’s stateless interface and internal cache reuse, eviction, and restoration.Across agent benchmarks, the paper reports improved average task performance and reduced inference costs, alongside higher throughput and lower latency.Source: §6 · p. 10
  • Further exploration of cache-engineering strategies, training agents under this scheme, and extending serving support to linear attention models remain future work.Source: §6 · p. 10

A.1.1 RESTORE WITH SEGMENT-BATCH ATTENTION

AKV restores missing KV states by dividing queries into segments at drop or repos operations. It batches those segments into one model forward pass while maintaining the historical visibility and position mappings each segment requires.

A.1.1 RESTORE WITH SEGMENT-BATCH ATTENTION

  • Queries are partitioned whenever drop or repos changes visible history or position mapping, and each segment uses corresponding KV pages with standard causal attention.This represents historical visibility without constructing a dense custom mask.Source: §A.1.1 · p. 15
  • In the illustrated restoration, dropping and repositioning change which earlier tokens each segment can attend to and the positions used for later tokens.For example, Segment 3 excludes D after its drop, while a later repos moves E from position 3 to 2 before Segment 4.Source: §A.1.1 · p. 15, Figure 10 · p. 15
  • All four segments run in one model forward pass, with temporary position versions made by re-encoding keys and copying values.This avoids recomputing hidden states for each version or replaying separate forward passes at operation boundaries.Source: §A.1.1 · p. 15

A.1.2 REPOSITION SCHEME

Reposition-aware cache reuse matches exact cache versions first, then reuses compatible prefixes through positional transformation. Token positions and operation history both matter because repositioning can change later computations.

A.1.2 REPOSITION SCHEME

  • Retrigger matching reuses prefixes via positional transformation, while preserving operation records so reuse cannot cross divergent computation histories.Under fixed model and position-encoding configurations and exact arithmetic, identical projected prefixes preserve hidden states and unrotated KV content; finite-precision transformations may introduce rounding error.Source: §A.1.2 · pp. 15–16
  • Token nodes store each token ID and the position currently encoded in its cached key, while operation nodes preserve the history governing computation.The stored position need not be the position at which the token was originally computed.Source: §A.1.2 · p. 15
  • Position labels distinguish key encodings that need transformation before reuse, and reposition nodes distinguish states that share final positions but arose from different computations.Different positions can change relative distances during a later append and therefore yield different hidden states; a subsequent reposition cannot erase that difference.Source: §A.1.2 · pp. 15–16
  • The system starts with the longest valid exact prefix, searches compatible branches for a longer retrigger match, and transforms selected source caches before use.Exact matching compares token-node fields and requires identical operation nodes and parameters; retrigger matching ignores stored position indices but preserves operation records.Source: §A.1.2 · p. 16

A.1.3 SERVING CONFIGURATION

AKV uses a serving setup with separate prefill and decode workers and extends radix-tree cache tracking to distinguish operation histories as well as tokens.

A.1.3 SERVING CONFIGURATION

  • AKV runs on SGLang v0.5.20 with GPT-OSS-120B in BF16 on eight A800 GPUs, using separate prefill and decode workers.All methods enable radix caching; baselines use LRU eviction, while AKV enables drop-aware eviction.Source: §A.1.3 · p. 16, Table 5 · p. 16
  • The extended radix tree records cache operations alongside tokens, distinguishing cache versions by position and operation history even when retained tokens overlap.The figure depicts trajectories interleaving token appends, drops, and reposition operations.Source: Figure 11 · p. 17

A.1.4 WORKLOAD REPLAY AND EVALUATION PROTOCOL

The experiments replay long-context agent workloads across several benchmarks and compare AKV with Standard and text-engineering baselines. They report faster decoding, lower cache costs, and benchmark performance for additional settings.

A.1.4 WORKLOAD REPLAY AND EVALUATION PROTOCOL

  • The workload uses the 80 SWE-bench Verified tasks with the longest uncompressed trajectories, replayed in a fixed order; turns are sequential within each task and task concurrency is maintained during measurement.Evaluations adapt benchmark harnesses, retrieval and grading configurations, and step limits for BCP, SWE-bench Verified, GAIA-Text, and xbench-DeepSearch-2510.Source: §A.1.4 · p. 17, §A.2.1 · p. 18

A.2.2 CONTEXT ENGINEERING BASELINE DETAILS

  • Text Engineering uses Pi’s summarization prompts and settings with a 16,384-token reserve and 20,000 recent-token retention; API-equivalent cost estimates charge cache-miss, cache-hit, and output tokens separately.Compaction triggers at the specified exact prompt-token threshold; AKV repositioning is charged at the cache-hit rate as a conservative estimate.Source: §A.2.2 · p. 18, §A.3 · p. 19, Table 6 · p. 19

A.4.1 SPEEDUP BREAKDOWN

  • At the same batch size, AKV reduces full-attention time by 59.8% and total decode-step time by 16.0% versus Standard.Dropping older tool-response KV shortens active context, reducing KV loading and attention computation during subsequent decode steps.Source: §A.4.1 · p. 19, Figure 12 · p. 20

A.4.2 COST BREAKDOWN AND ACTIVE CONTEXT LENGTH

  • AKV reduces cache-read costs by approximately 17% versus Standard and cache-miss input costs by approximately 48% versus Text Engineering.Figure 13 compares average task cost on BCP for Standard, Text Engineering, and AKV.Source: §A.4.2 · p. 19, Figure 13 · p. 20
  • AKV continually drops older tool-response KV while retaining the most recent K rounds and reasoning KV, slowing active-context growth compared with Standard’s accumulation and threshold-triggered reduction.The active context still grows as reasoning accumulates.Source: §A.4.2 · p. 19, Figure 14 · p. 20

A.4.3 ATTENTION ACROSS MESSAGE HISTORY

  • Across 10 selected BrowseComp-Plus trajectories, recent history generally receives more attention; attention density per token is described across old reasoning, old tool responses, and recent history.On BCP, adding AKV to Pi improved accuracy by 3% over native Pi without compaction and 4% over Pi with compaction for both GPT-OSS-120B and GPT-OSS-20B.Source: Figure 15 · p. 20, §A.4.3 · p. 21

A.4.4 ADDITIONAL MODELS

  • 45.2% is AKV’s highest average pass rate on GPT-OSS-20B, with an average cost of 4.8 cents; it leads on BCP and SWE-Verified pass rate.AKV also has the lowest cost on GAIA-Text and xbench-DS; the authors note smaller gains on some metrics for the less capable smaller model.Source: Table 7 · p. 21, §A.4.4 · p. 21

A.5 ADDITIONAL RELATED WORKS

  • Related work includes model-directed cache eviction and editable-context approaches; AKV instead exposes policy-independent cache operations through a stateless interface and tracks operation histories in an extended radix tree.The paper contrasts AKV’s shared cache reuse and recovery with Context Language Models’ session-local reuse in private buffers.Source: §A.5 · p. 21
Loading 2610.28577v1…