Source-linked AI summary
TokenPilot: Cache-Efficient Context Management for LLM Agents
Buqiang Xu, Zirui Xue, Dianmou Chen, Chenyang Fu, Chiyu Wu, Caiying Huang, Chen Jiang, Jizhan Fang, Xinle Deng, Yijun Chen, Yunzhi Yao, Xuehai Wang, Jin Shang, Gong Yu, Ningyu Zhang
TL;DR
Long-horizon agents face rising inference costs as context accumulates, while sequence reduction can disrupt prompt-cache continuity. TokenPilot combines ingestion-aware compaction with lifecycle-aware eviction, reducing costs by 61% and 56% in isolated mode and 61% and 87% in continuous mode on PinchBench and Claw-Eval while maintaining competitive task performance.
Problem
Context growth raises per-turn inference costs, while existing sequence mutations can disrupt layouts and prompt-cache continuity.
Method
TokenPilot combines global ingestion-aware compaction with local lifecycle-aware eviction to stabilize layouts and offload context when task-level utility expires.
Results
61% and 56% lower costs in isolated mode, and 61% and 87% lower costs in continuous mode, were observed on PinchBench and Claw-Eval while maintaining competitive task performance.
Takeaways & Limitations
TokenPilot offers a cost-efficient approach for long-horizon agent systems without sacrificing task effectiveness under isolated and continuous modes.
Takeaways & Limitations
TokenPilot depends on backend prefix-caching support, requires environment- and task-specific tuning, and may be less effective on heterogeneous shuffled task streams.
Abstract
from arXiv · showhide
As LLM agents are deployed in long-horizon sessions, context accumulation drives up inference costs. Existing approaches utilize text pruning or dynamic memory eviction to minimize token footprints; however, their unconstrained sequence mutations alter layouts, introducing prefix mismatches and cache invalidation. This reveals a critical trade-off between text sparsity and prompt cache continuity. To address this, we present TokenPilot, a dual-granularity context management framework. Globally, Ingestion-Aware Compaction acts as a framework harness to stabilize prompt prefixes and eliminate open-world environmental noise at the ingestion gate. Locally, Lifecycle-Aware Eviction monitors the ongoing residual utility of context segments, enforcing a conservative batch-turn schedule to offload content segments only when task relevance expires. Experiments on PinchBench and Claw-Eval under both isolated and continuous modes demonstrate that TokenPilot reduces costs by 61% and 56% in isolated mode, and 61% and 87% in continuous mode, while maintaining competitive performance compared to prior systems. TokenPilot has been integrated into LightMem2 at https://github.com/zjunlp/LightMem2.
1 Introduction
TokenPilot addresses the trade-off between reducing context tokens and preserving prompt-cache continuity through dual-granularity context management. It stabilizes prefixes during ingestion, delays eviction until residual utility expires, and reduces inference expenditures while maintaining competitive task performance.
- Motivation: Context growth increases sequence lengths and per-turn inference costs, making context management essential for sustainable real-world deployment.
- Motivation: Existing text compression and dynamic trace-management methods reduce content but mutate layouts, breaking prompt-prefix continuity and causing cache inefficiency.
- Design Principle: An effective framework must reconcile text-level sparsity with hardware cache alignment by preserving physical prefix continuity during ingestion and deferring eviction until residual utility expires.
- TokenPilot: TokenPilot uses Ingestion-Aware Compaction globally and Lifecycle-Aware Eviction locally to optimize initial prompt layout, remove structural noise, and monitor trajectory utility online.Ingestion-Aware Compaction uses stable placeholders, shifts tool definitions downstream, and strips structural noise from incoming tool responses before ingestion.
- Results: 61% and 56% in isolated mode, and 61% and 87% in continuous mode, are the reported reductions in total inference monetary expenditures on PinchBench and Claw-Eval.TokenPilot maintained competitive task performance under commercial pricing structures.
2 Background
The paper models agents processing task sequences whose accumulated context is managed differently in isolated and continuous modes. Its framework transforms raw histories into optimized contexts by balancing context utility against serving cost under prompt-caching constraints.
- Task Settings: Agents process task sequences whose instructions, reasoning traces, tool calls, and responses accumulate into session context C.Isolated mode resets C at each task boundary, whereas continuous mode preserves histories across the sequence.
- Optimization Objective: A context management framework M transforms raw history C into an optimized runtime context C′ = M(C).The stated optimization objective is to maximize the ratio of context utility.
- Optimization Objective: Context utility measures the necessity of tokens for downstream reasoning and tool execution through the estimated marginal contribution of message m.The estimate is denoted ˆU(m | C′).
- Optimization Objective: Serving cost K(C′) is governed by backend KV prompt caching, with cached tokens receiving discounted cost and other tokens incurring full pre-fill cost.The formulation also imposes a length alignment constraint on C′.
3 TokenPilot
TokenPilot is a dual-granularity context-management framework that stabilizes sequence layout at ingestion and regulates local context retention by residual utility. It combines ingestion gating, prefix canonicalization, deterministic observation reduction, and conservative lifecycle-aware eviction to preserve cache continuity.
- Framework overview: TokenPilot combines Ingestion-Aware Compaction at the ingestion boundary with Lifecycle-Aware Eviction at the local context-sequence level.The architecture operates across two complementary operational levels.
- Ingestion-Aware Compaction: Internal intentional messages retain high utility density, while environmental feedback passes through an ingestion gate that restores full delivery when hash access exceeds threshold τ.The gate is defined as G(m) = 1[f(h(m)) > τ].
- Ingestion-Aware Compaction: Canonicalization replaces volatile runtime markers with static placeholders, securing a byte-identical prompt prefix and eliminating full-cost pre-fill penalties.Environmental messages with G(m) = 0 receive deterministic reduction; missing critical signals trigger recovery from the artifact registry and disable later truncation on that path.
- Lifecycle-Aware Eviction: Lifecycle-Aware Eviction tracks segments through active, completed, and evictable states, retaining completed segments while residual utility remains non-zero.Segments are not immediately truncated after execution concludes.
- Lifecycle-Aware Eviction: Batch-gated estimation updates lifecycle states conservatively every B turns, then performs a single-pass purge only after segments become evictable.A Qwen3.5-35B-A3B zeroshot validator incurs less than $0.03 across the continuous PinchBench stream.
4 Experiments
Experiments evaluate TokenPilot on PinchBench and Claw-Eval in isolated and continuous modes, measuring task accuracy and actual monetary expenditure. TokenPilot achieves low costs while preserving task effectiveness, and ablations attribute gains to ingestion-aware compaction, prefix stabilization, and lifecycle-aware eviction.
- Benchmarks and Metrics: TokenPilot is evaluated on PinchBench and Claw-Eval in isolated and continuous modes, tracking task accuracy and actual monetary expenditures from provider-reported cache metadata.All evaluated methods use GPT-5.4-mini as the agent backbone.
- Isolated Mode: $3.22 on PinchBench and $2.27 on Claw-Eval are TokenPilot’s lowest isolated-mode inference costs while maintaining competitive task accuracy.Text compression lowers expenditure but degrades performance, whereas dynamic frameworks preserve execution quality but incur cache-miss penalties.
- Continuous Mode: 81.3 and $2.79 are TokenPilot’s continuous-mode PinchBench score and expenditure, with cache misses restricted to 1.549M tokens.On Claw-Eval, TokenPilot reduces Vanilla’s $81.52 expenditure to $10.58 under continuous task streams.
- Ablation Study: $7.24 to $4.22 is the expenditure reduction from adding Ingestion-Aware Compaction, while cache misses fall from 5.943M to 1.589M tokens.Rule-based pruning filters environmental noise, and prefix stabilization converts expensive pre-fills into cache hits through reliable layout reuse.
- Lifecycle-Aware Eviction: 65.0% is the cache-read-token reduction from 26.716M to 8.551M after Lifecycle-Aware Eviction, lowering expenditure to $2.79 while maintaining the overall score.B = 3 balances accuracy, memory reduction, API-call times, and prefix continuity; disabling eviction causes both metrics to peak.
- Prefix Stabilization and Context Reduction: 38.7% to 79.2% and 67.2% to 83.1% are macro cache-hit-rate increases on PinchBench and Claw-Eval after stable placeholders standardize prompt layouts.Context reduction removes up to 115k characters of structural noise and 883k characters of terminal logs across heterogeneous tasks.
5 Related Work
Related work improves context utility through ingestion-stage compression, selective historical recall, and runtime memory-management abstractions. These approaches include externalized session storage, adaptive routing and isolation, and operating-system or virtual-memory designs for long-horizon planning.
- Static Content Compression and Abstraction: Ingestion-stage methods filter or restructure historical trajectories to maximize utility density within the context window.This preserves the premium context budget by reducing or abstracting historical content before runtime use.
- Static Content Compression and Abstraction: External-memory systems offload entire session histories to databases and selectively recall high-utility historical fragments.Remaining interactions are filtered out during recall.
- Runtime Context Management: Runtime frameworks support demand paging, adaptive parallel routing, and context isolation based on real-time trajectories and budget constraints.These mechanisms dynamically manage context during execution rather than only at ingestion.
- Runtime Context Management: For long-horizon planning, architectures manage memory as a self-organizing operating system or use virtual-memory abstractions to secure stateful data residency and tool durability.Both designs target dynamic retention of state and tool-related information.
6 Conclusion
TokenPilot reconciles text reduction with strict prompt-cache alignment through dual-granularity context management. Evaluations on PinchBench and Claw-Eval show reduced inference expenditures under isolated and continuous modes without sacrificing task effectiveness.
- 6 Conclusion: TokenPilot is a dual-granularity framework reconciling text reduction with strict prompt-cache alignment.It separates memory management into global ingestion-aware compaction and local lifecycle-aware eviction.
- 6 Conclusion: Global compaction stabilizes dynamic layouts, while local eviction conservatively offloads context according to task-level residual utility.
- 6 Conclusion: Evaluations on PinchBench and Claw-Eval show that TokenPilot cuts inference expenditures in isolated and continuous modes without sacrificing task effectiveness.The framework is presented as a scalable, cost-efficient foundation for long-horizon agent systems.
Limitations
TokenPilot may misclassify context segments in ambiguous or sparse interactions, require deployment-specific tuning, and depend on backend prefix-caching support. Its continuous-mode evaluation also has an additional limitation that is not specified in the supplied passage.
- The model-based estimator may misclassify context segments under highly ambiguous or sparse interaction patterns.
- The frequency threshold τ and batch size B may require tuning for different deployment environments and task distributions.
- Prefix stabilization provides no benefit to providers without backend support for prefix caching.
A Appendix · A.1 Dataset Configurations
TokenPilot is evaluated on PinchBench and Claw-Eval using configurations designed to reflect realistic continuous multitask agent execution. The evaluation covers frozen, multi-category, containerized, and uninterrupted single-session task settings.
- A.1 Dataset Configurations: TokenPilot is evaluated on two realistic agent benchmarks: PinchBench and Claw-Eval.These benchmarks provide the evaluation platforms for the framework.
- A.1 Dataset Configurations: PinchBench comprises 11 distinct task categories and 123 tasks in total.The benchmark uses a frozen snapshot to account for continuous rolling updates in its upstream repository.
- A.1 Dataset Configurations: PinchBench evaluation uses a frozen benchmark snapshot because its upstream repository receives continuous rolling updates.This configuration keeps the evaluated benchmark version fixed.
- A.1 Dataset Configurations: Claw-Eval is a containerized agent evaluation platform executed within isolated sandbox environments.Its evaluation environment isolates agent execution.
- A.1 Dataset Configurations: The Claw-Eval General task group encompasses 161 multi-step service orchestration and standalone analytical tasks.TokenPilot is evaluated on this General task group.
- A.1 Dataset Configurations: For both benchmarks, same-category tasks are grouped into contiguous, uninterrupted single sessions.This setup simulates realistic continuous multitask agent execution trajectories, with detailed structural statistics compiled in Table 6.
A.2 Evaluation Metrics and Cost Modeling
The evaluation uses benchmark-native task scoring, with Claw-Eval combining completion, safety, and robustness evidence and PinchBench aggregating task-specific verification checks. Continuous-mode trajectories are sliced by task boundaries for isolated-equivalent grading, while inference cost is modeled from cache-hit, cache-miss, and output-token pricing.
- Task Score: Claw-Eval evaluates Completion (scomp), Safety (ssafe), and Robustness (srob) as coupled parameters using auditable trajectory evidence.Evidence includes service audit logs, environment snapshots, and execution traces.
- Task Score: Safety acts as a strict multiplicative gate, while Completion measures goal-directed quality and Robustness measures error recovery under controlled perturbations.The rubric provides continuous partial credits rather than binary verdicts.
- Task Score: PinchBench aggregates task-specific verification checks on output deliverables to assess goal-directed capability.
- Task Score: Continuous-mode transcripts are sliced into task-specific segments at original task boundaries, then graded independently using logic mathematically comparable to Isolated Mode.This preserves identical evaluation logic across continuous task streams and isolated tasks.
- Inference Cost Modeling: Inference cost is modeled as a monetary metric over cache-hit tokens, cache-miss tokens, and generated response tokens using commercial deployment pricing.GPT-5.4-mini prices are phit = $0.075/M tokens, pmiss = $0.75/M tokens, and pout = $4.50/M tokens.
A.3 Baseline Configurations · A.4 Implementation Details
A.3 specifies reproducible configurations for Vanilla and comparison baselines, while A.4 details TokenPilot’s cache-stabilizing layout transformations, observation reduction rules, model assignments, and estimator prompts. The implementation uses fixed thresholds, shared GPT-5.4-mini execution, Qwen3.5-35B-A3B state estimation, and explicit lifecycle-based eviction signaling.
- A.3 Baseline Configurations: Vanilla uses OpenClaw without extra context management, with a 500k-token context window and a 0.5 compaction trigger ratio.
- A.3 Baseline Configurations: LLMLingua-2 applies token-level compression at ratio 0.6, while SelectiveContext applies self-information-based sentence compression at ratio 0.4.
- A.3 Baseline Configurations: LCM hierarchically summarizes losslessly at 75% context utilization, accumulating up to 80k tokens per leaf chunk and retaining 64 recent turns in full fidelity.
- A.3 Baseline Configurations: Pichay uses demand-paging thresholds of 60k, 100k, and 120k tokens, compressing tool results older than 4 user turns with a 500-byte minimum eviction size.
- A.4 Implementation Details: TokenPilot stabilizes cache prefixes by replacing volatile system-prompt fields with static placeholders and relocating tool definitions before inference.
- A.4 Implementation Details: Observation reduction deduplicates repeated tool results and truncates oversized parameters or outputs, retaining 600-character prefixes and 400-character suffixes recoverable through a recovery tool.
- A.4 Implementation Details: Reduction activates at triggerMinChars = 2200 for inputs and maxToolChars = 1200 for toollike fragments, with generic output truncation capped at 50k characters.
- A.4 Implementation Details: All agents use GPT-5.4-mini, TokenPilot’s estimator uses Qwen3.5-35B-A3B, and interval context processing tracks a 3-turn batch window.