Source-linked AI summary
Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks
Elias Lumer, Faheem Nizar, Akshaya Jangiti, Kevin Frank, Anmol Gulati, Mandar Phadate, Vamse Kumar Subbiah
TL;DR
Prompt caching for long-horizon agentic workloads lacks quantified evidence on cost savings and strategy trade-offs. This paper evaluates three caching strategies across three providers and reports substantial cost reductions and TTFT improvements, while finding that strategic cache boundaries are more consistent than naive full-context caching. The study also identifies provider- and threshold-dependent variability relevant to production deployment.
Problem
Prompt-caching benefits and strategy trade-offs for multi-turn agentic workloads remain underexplored, despite long contexts and extensive tool calling.
Method
The study compares three caching strategies across three providers and four models on DeepResearch Bench, measuring API cost and TTFT across over 500 sessions.
Results
41 to 80% API-cost reductions and 13 to 31% TTFT improvements are reported across providers, with strategic cache boundaries outperforming naive full-context caching for consistency.
Takeaways & Limitations
Caching stable system prompts while excluding dynamic tool results provides actionable guidance for more consistent cost and latency benefits in agentic systems.
Takeaways & Limitations
TTFT depends on provider-specific minimum thresholds and noisy production factors, so practitioners should test workloads representative of their usage patterns.
Abstract
from arXiv · showhide
Recent advancements in Large Language Model (LLM) agents have enabled complex multi-turn agentic tasks requiring extensive tool calling, where conversations can span dozens of API calls with increasingly large context windows. However, although major LLM providers offer prompt caching to reduce cost and latency, its benefits for agentic workloads remain underexplored in the research literature. To our knowledge, no prior work quantifies these cost savings or compares caching strategies for multi-turn agentic tasks. We present a comprehensive evaluation of prompt caching across three major LLM providers (OpenAI, Anthropic, and Google) and compare three caching strategies, including full context caching, system prompt only caching, and caching that excludes dynamic tool results. We evaluate on DeepResearch Bench, a multi-turn agentic benchmark where agents autonomously execute real-world web search tool calls to answer complex research questions, measuring both API cost and time to first token (TTFT) across over 500 agent sessions with 10,000-token system prompts. Our results demonstrate that prompt caching reduces API costs by 41-80% and improves time to first token by 13-31% across providers. We find that strategic prompt cache block control, such as placing dynamic content at the end of the system prompt, avoiding dynamic traditional function calling, and excluding dynamic tool results, provides more consistent benefits than naive full-context caching, which can paradoxically increase latency. An ablation study across prompt sizes (500-50,000 tokens) and tool call counts (3-50) demonstrates universal linear cost and TTFT benefits, after the provider caching token minimum, and reveal provider-specific strategy discrepancies across variants. We provide nuanced discussion and guidance for implementing prompt caching in production agentic systems.
1. Introduction
Prompt caching addresses an underexplored cost and latency problem in long-horizon agentic workloads. Across providers, caching reduces costs consistently, while latency benefits depend on strategy and cache-boundary control.
- Long-horizon agents accumulate large contexts across many tool-calling API calls, creating substantial cost and latency overhead.
- Prompt-caching benefits for real-world agentic workloads remain underexplored, with no prior work quantifying cost savings or comparing strategies.
- The study evaluates three caching strategies across three providers using DeepResearch Bench, 500 agent sessions, 10,000-token system prompts, API cost, and TTFT.
- 41% to 80% cost reductions occur across providers, with statistically significant savings for all four tested models and consistent benefits across strategies.
- 13% to 31% TTFT improvements occur across providers, but the best latency strategy can differ from the best cost strategy.
- Caching only stable system-prompt content or excluding dynamic tool results provides more consistent benefits than naive full-context caching.
2. Background
Prompt caching reuses provider-managed KV tensors for repeated prompt prefixes, but dynamic, tool-intensive agentic contexts complicate cache reuse. Provider thresholds, TTLs, and implementation differences further affect practical outcomes.
- Inference separates into prefill, which computes KV tensors for the input, and decode, which autoregressively generates output tokens.
- Prompt caching is a provider-managed feature that reuses KV tensors across requests sharing common prefixes, reducing repeated computation, latency, and cost.
- Provider implementations differ in automatic activation, explicit cache breakpoints, minimum token thresholds, TTL durations, and pricing structures.
- Agentic workloads accumulate dynamic, session-specific tool results that complicate reuse because content is interleaved with stable system prompts.
3. Methodology
The study evaluates prompt caching across flagship models and controlled cache-boundary conditions on a tool-intensive multi-turn benchmark. It measures per-session cost and TTFT against a no-cache baseline using statistical comparisons.
- The evaluation covers GPT-5.2, Claude Sonnet 4.5, Gemini 2.5 Pro, and GPT-4o across OpenAI, Anthropic, and Google APIs.
- DeepResearch Bench supplies multi-turn research tasks requiring autonomous web-search tool calls, including 100 PhD-level questions across 22 fields.
- Each model uses 40 independent sessions per cache condition and a 10,000-token system prompt, with fresh contexts per session.
- The four conditions are no cache, full-context caching, system-prompt-only caching, and excluding tool results through explicit UUID cache boundaries.
- API cost aggregates standard, cached, and cache-creation token charges across session calls, while TTFT measures time from request initiation to the first streamed response chunk.
- Independent-samples t-tests compare each condition with no caching at α = 0.05, using n = 40 per model and cache mode.
4. Results
Prompt caching consistently reduces cost, while latency effects vary more across providers and cache strategies. System-prompt-only and tool-result-excluding strategies generally outperform full-context caching for TTFT.
- 41% to 80% cost savings and 6% to 31% TTFT improvements are reported for the best-performing cache mode across all four models.
- All cache modes reduce cost relative to no caching, with model-dependent reductions of 79-81% for GPT-5.2, 78-79% for Claude Sonnet 4.5, 46-48% for GPT-4o, and 28-41% for Gemini 2.5 Pro.
- The system prompt drives most cost savings because it is large and stable, while caching conversation history and tool calls adds only marginal cost benefits in this setup.
- TTFT varies more than cost across providers; GPT-4o improves 28-31% with selective strategies but regresses 8.8% under full-context caching.
- Across models, caching strategies differ little for cost, typically within 2-4 percentage points, but differ more substantially for latency.
- System-prompt-only and tool-result-excluding caching generally outperform full-context caching for TTFT, which can incur cache-write overhead for dynamic content.
5. Discussion
The discussion recommends caching stable, reusable content while controlling cache boundaries around dynamic agentic content. It also highlights provider variability, operational caveats, and security considerations that require workload-specific evaluation.
- Strategic Cache Boundary Control: Strategic cache boundaries prevent write overhead from caching dynamic, session-specific content without corresponding read benefits.The paper identifies stable, reusable content as the preferred cache target.
- Strategic Cache Boundary Control: System prompts are the most stable cache target, while conversation history, tool calls, and tool results are dynamic and session-specific.Dynamic values embedded in system prompts can break the cache; necessary dynamic information should be placed at the prompt’s end.
- Tool Call Caching Considerations: Caching tool calls and results may reduce costs in 30-50+ tool-call sessions, but variable or unrepeated results can add overhead instead.Cache creation costs are amortized only when later requests benefit from cache reads.
- Tool Call Caching Considerations: Context summarization and pruning can break cached tool-call representations, making tool-call caching counterproductive.The paper therefore favors a stable system prompt while treating tool calls as dynamic content managed during the session.
- Provider Implementation Variability: Provider cache thresholds range from 1,024 to 4,096 tokens and TTL durations from 5 minutes to 24 hours.Automatic versus explicitly configured caching also differs across providers, so current documentation matters for deployment.
- Provider Implementation Variability: API response-time variance reflects server load, geographic distribution, and infrastructure differences, so practitioners should test representative workloads.Published benchmarks alone may not reflect a practitioner’s usage patterns.
- Provider Implementation Variability: Prompt caching can introduce timing side-channels that may leak information about cached content.The discussion identifies this as a security consideration for deployment.
6. Ablation Study
The ablation study finds that prompt size is the dominant workload factor for caching benefits: cost savings grow with context length, while tool-count effects are smaller and latency is more variable.
- Prompt-size ablation: Cost savings scale linearly with prompt size, reaching 89% for GPT-5.2, 88% for Claude Sonnet 4.5, and 54% for GPT-4o at 50,000 tokens.The corresponding costs are $0.253 to $0.029, $0.667 to $0.080, and $0.414 to $0.192, respectively.
- Prompt-size ablation: 60% TTFT improvement occurs for GPT-4o at 50,000 tokens, declining from 4,290ms to 1,699ms.
- Prompt-size ablation: At 500 tokens, below caching thresholds, cost savings are modest and TTFT regresses by 10-18%; above 2,000 tokens, TTFT improvements become positive.
- Tool-count ablation: GPT-5.2 maintains 77-81% cost savings across tool counts, while GPT-4o achieves 42-53%, showing that tool-call count does not substantially affect cost benefits.
- Tool-count ablation: GPT-4o shows 16-36% TTFT improvements across tool counts, whereas Claude Sonnet 4.5 declines from 19% at 3 tools to 5% at 50 tools.
- Discussion: Across models, prompt sizes, and tool counts, prompt caching consistently reduces API costs, but TTFT measurements are noisier in the ablation study.
- Discussion: Prompt size drives caching benefits more than tool count because cacheable prefix length is primarily determined by the system prompt.
7. Conclusion
The paper evaluates prompt caching for long-horizon agentic tasks across three major LLM providers and reports substantial cost and latency benefits. It concludes that controlling cache boundaries is more consistent than naive full-context caching.
- Prompt caching reduces API costs by 41-80% and improves TTFT by 13-31% across three major LLM providers.
- Caching only system prompts while excluding dynamic tool results provides more consistent benefits than naive full-context caching.Naive full-context caching can paradoxically increase latency.
- The evaluation provides actionable guidance for deploying prompt caching in production agentic systems.
A. Prompt caching pricing at time of evaluation
This appendix describes the pricing inputs, activation thresholds, and exact-prefix mechanism used to analyze provider-managed prompt caching. These implementation details determine when cached computation and pricing benefits can apply.
- A. Prompt caching pricing at time of evaluation: Token pricing was recorded in early January 2026 in USD per 1M tokens, with cached input and cache-write charges distinguished.
- A. Prompt caching pricing at time of evaluation: Google additionally charges $4.50 per million tokens per hour for context-cache storage, which the analysis accounts for separately.
- A. Prompt caching pricing at time of evaluation: Prompts shorter than provider-specific minimum lengths cannot benefit from prompt caching even when caching features are enabled.
- B. Prompt Caching Mechanism: Prompt caching reuses previously computed KV tensors only when the entire prompt prefix matches exactly.
- B. Prompt Caching Mechanism: A cache miss caused by any changed prefix token forces complete recomputation of the affected prompt tokens.
- B. Prompt Caching Mechanism: Cache hits reuse cached KV tensors, so only newly appended tokens require computation.
C. Cache Strategy Implementations
The cache strategies use exact prefix matching and UUID-controlled boundaries to separate stable content from request-varying content. Static content before a UUID remains cacheable, while content after it prevents later prefix matches.
- UUIDs control cache boundaries by marking where static, cacheable content ends and variable content begins.
- Content placed after the UUID varies between requests and prevents prefix matches beyond that point.
C.1. No Cache (Baseline)
The caching strategies differ by where cache-breaking UUIDs are placed: the baseline prevents prefix matches, while alternative placements preserve selected static prefixes and recompute dynamic content.
- A unique UUID at the system-prompt start prevents prefix matches and forces full token recomputation for every request.
- Full-context caching omits UUIDs so the provider can cache the entire prompt prefix, including potentially changing tool results.Variable content may trigger cache writes without corresponding cache hits.
- System-prompt-only caching appends a UUID after the system prompt, caching the static system prompt while recomputing dynamic conversation content.
C.4. Exclude Tool Results Caching
This strategy places cache breakers after the system prompt and each tool result, preserving static content while excluding session-specific tool results from caching.
- UUIDs appended after the system prompt and each tool result break the cacheable prefix at those boundaries.
- Excluding tool results prevents session-specific results from being cached, avoiding writes for content unlikely to yield future cache hits.
- The strategy mirrors cache-breaking context engineering that prunes or summarizes past tool calls.