PaperlayerView exact revision on arXiv

Source-linked AI summary

Silent and Costly: Characterizing and Detecting Prompt-Caching Mis-Integration in LLM Agents

Ying Zhang, Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Yepang Liu, Leo Zhang

arXiv:2610.34817v1cs.SEcs.AI

TL;DR

Repeated prompt prefixes lead to redundant LLM computation, while prompt-caching defects can remain unnoticed until operational metrics are examined. We study 120 defects and build CACHELINT, a static analyzer; it detects mis-integrations in labeled and real-world evaluations.

  • Problem

    Repeated prompt prefixes cause redundant computation, while many prompt-caching defects remain unnoticed until operational metrics such as cache-hit rates or API costs are examined.

    Source: §I · p. 1, §III · p. 3

  • Method

    We study 120 prompt-caching defects from 10 agent systems and build CACHELINT, which checks caching invocation sites against preconditions using a dated provider-capability oracle or an LLM guard check.

    Source: §I · p. 2, §X · p. 11

  • Results

    CACHELINT achieves 0.71 precision and 0.78 recall on 120 fixed defects, and 0.86 precision while uncovering 32 previously unknown mis-integrations across 571 repositories.

    Source: §I · p. 2

  • Takeaways & Limitations

    Because mis-integrations often preserve functional outputs and evade dynamic test oracles, static analysis is well-suited to detecting them.

    Source: §I · p. 2

    The taxonomy may not generalize beyond the ten studied GitHub applications, and multi-snapshot defects remain out of scope.

    Source: §VIII · p. 11

Abstract

from arXiv · show

Large language model (LLM) services process massive volumes of user requests, resulting in substantial inference costs. To reduce both cost and latency, providers have introduced prompt caching. However, correctly integrating prompt caching is non-trivial due to heterogeneous provider-specific APIs and continuously evolving caching semantics. To understand prompt-caching mis-integration in downstream applications, we conduct the first empirical study of 120 prompt-caching-related pull requests from 10 popular open-source agent systems. Our findings show that prompt-caching mis-integrations arise in both request-side misconfiguration and read-side usage miscount. More importantly, most mis-integrations fail silently without causing observable runtime failures. Motivated by these findings, we design CACHELINT, which models a prompt-caching mis-integration as an invocation site together with the preconditions required for correct cache integration. CACHELINT locates caching invocation sites with CodeQL and discharges each precondition either against a dated provider capability oracle or with an LLM guard check. CACHELINT achieves 0.71 precision and 0.78 recall in identifying the 120 labeled mis-integrations across the 10 studied agent systems. Our evaluation further demonstrates the effectiveness of CACHELINT in real-world settings, identifying 32 mis-integrations across 571 agent projects with 0.86 precision. These findings have received 21 developer confirmations and led to 18 upstream fixes, including one in Anthropic's official SDK.

I. INTRODUCTION

Prompt caching can reduce redundant LLM computation, but provider-specific and evolving requirements make correct application integration difficult. This paper studies 120 defects and introduces CACHELINT, reporting results on labeled defects and real-world repositories.

I. INTRODUCTION

  • Prompt caching addresses redundant processing of repeated prompt prefixes, but heterogeneous provider settings and changing semantics complicate integration.Prior work considered LLM-system defects and caching performance, but not application-level integration issues.Source: §I · pp. 1–2
  • The study analyzes 120 defects from prompt-caching-related pull requests in 10 popular open-source agent systems.The authors selected the 10 most popular applications from 581 repositories and analyzed 148,947 merged pull requests.Source: §I · p. 2
  • Of 120 defects, 97 corrupted outgoing requests and 23 misparsed cache-usage metrics; 108 cases failed silently, preserving outputs while increasing cost or latency.Request-side defects included provider-specific mechanisms and application-side prompt-prefix specifications.Source: §I · p. 2
  • CACHELINT models each caching invocation site with required preconditions, locating sites with CodeQL and checking conditions against a dated provider-capability oracle or an LLM guard.It reports failed checks with suggested fixes.Source: §I · p. 2
  • 0.71 precision and 0.78 recall on 120 labeled defects; on 571 repositories, CACHELINT found 32 defects at 0.86 precision, with 21 maintainer confirmations and 18 fixes.The labeled set covers 10 studied projects.Source: §I · pp. 2–3

II. BACKGROUND

Prompt caching reuses computation for matching prompt prefixes, which recur in agent calls. Correct integration depends on preserving prefix stability and meeting provider-specific configuration requirements.

A. Prompt Caching

  • Prompt caching reuses previously computed prompt-prefix work when a new prompt begins with an exact match; changes invalidate matching from that point onward.For agents, stable preambles and conversation histories create repeated prefixes, reducing cached-input costs and latency when reused.Source: A. Prompt Caching · p. 3

B. Prompt Caching Integration

  • Correct integration requires both an exactly stable prefix and suitable provider caching parameters, such as prefix length and cache lifetime.Insufficient prefix length or short cache windows can prevent reuse even when prompts are structurally identical.Source: B. Prompt Caching Integration · p. 3
  • A sub-second timestamp at the front of a system prompt breaks the cached prefix on every call; coarsening it to day granularity fixes the example.The figure illustrates how a changing early prefix invalidates reuse of later stable content.Source: Figure 2 · p. 3

III. MOTIVATING EXAMPLE

A changing timestamp in a system prompt illustrates how an application can defeat caching despite correct provider behavior. The paper then describes its repository and pull-request collection for studying such defects.

III. MOTIVATING EXAMPLE

  • A changing timestamp at the start of PrivateGPT’s system prompt invalidated subsequent prefix reuse, while coarsening it to date granularity kept the prefix stable through the day.The defect remained silent because outputs stayed correct; increased cost or reduced cache-hit rate could reveal it.Source: §III · p. 3

A. Dataset Collection

  • The methodology searches GitHub using cache markers, usage fields, and prompt-caching phrases, then filters repositories by application scope, maintenance activity, and verified caching integration.The search returned 581 candidate repositories.Source: A. Dataset Collection · p. 4
  • The study retained the ten most-starred qualifying agent applications, spanning TypeScript, Python, and Rust and major LLM providers.Across them, the authors searched 148,947 merged pull requests and identified 622 candidate PRs.Source: A. Dataset Collection · p. 4
  • Manual screening of 622 candidate PRs retained 120 prompt-caching defects and discarded 502 false matches.Three authors independently screened candidates; their Fleiss’ κ was 0.76, and disagreements were resolved through discussion.Source: A. Dataset Collection · p. 4

B. Data Analysis and Annotation

Three experienced agent developers independently coded each defect’s root cause, symptom, and fix strategy, resolving disagreements by consensus. The resulting distribution is weighted toward request-side defects.

B. Data Analysis and Annotation

  • For each defect, the authors collected its fixing code and report link, studied the discussion, and used open coding to classify causes, symptoms, and fixes.Three experienced agent-application developers coded all defects independently and resolved disagreements by consensus.Source: B. Data Analysis and Annotation · p. 4

A. RQ1: Defect Type and Root Causes

1) Request Side:

Request-side mis-integration arises when agents configure or construct caching requests incorrectly, preventing cache reuse or causing runtime failures. Provider compatibility and prefix stability are prominent sources of these defects.

1) Request Side:

  • Request-side mis-integration means constructing or configuring a request so the provider recomputes cacheable content or the request fails at runtime.Recomputation incurs full inference cost.Source: 1) Request Side · p. 4
  • Unsupported caching configurations can leave caching inactive, while incompatible provider-specific markers can cause requests to fail.Hardcoded model-name gates may silently disable caching after model renames; unsupported markers can trigger runtime errors.Source: 1) Request Side · pp. 4–5
  • Unstable prefixes prevent cache hits when per-request values, changed reassembly, nondeterministic serialization, or misplaced markers alter the cached span.Examples include timestamps in system prompts, reordered prompts, and random call identifiers.Source: 1) Request Side · p. 5
  • Cross-turn mutation can rewrite an already-cached conversation prefix, causing the next call to write the cache again instead of reading it.Pruning history images after three turns rewrote the cached prefix in one reported case.Source: 1) Request Side · p. 5

2) Read Side:

Read-side defects mishandle cache-usage data returned by providers, while most prompt-caching defects produce no visible runtime failure. Silent waste and incorrect usage telemetry are the main no-error symptoms.

2) Read Side:

  • Read-side mis-integration occurs when caching works as intended but the agent mishandles cache usage reported in the response.A usage miscount can hardcode cache-creation tokens to zero, making the readout report no caching while the cache keeps hitting.Source: 2) Read Side · p. 5
  • The study observed eight root causes, with provider compatibility and prefix stability accounting for the majority.Only 12 defects (10%) crash, and incompatible markers account for 8 of those crashes.Source: 2) Read Side · p. 5, B. RQ2: How Defects Manifest · p. 6

B. RQ2: How Defects Manifest

  • 108 of 120 defects (90%) remain silent, with no exception, failed assertion, or crash and correct model output despite degraded caching.Only 12 defects crash; 8 of those involve incompatible markers.Source: B. RQ2: How Defects Manifest · p. 6
  • 86 (72%) of defects are silent waste, while 22 (18%) produce wrong telemetry among the no-error defects.Silent waste means caching stops hitting while execution continues; wrong telemetry occurs when cache-token counts are parsed, summed, or displayed incorrectly despite successful caching.Source: B. RQ2: How Defects Manifest · p. 6

C. RQ3: Fixing Strategies

Fixes for prompt-caching defects are usually small and patterned, and their strategies are largely predictable from root causes. CACHELINT models defects as invocation sites paired with caching preconditions that must hold for correct integration.

C. RQ3: Fixing Strategies

  • Six recurring strategies address eight root causes, including capability lookup, marker guards, cache-boundary fixes, prefix canonicalization, lifecycle guards, and usage-counter fixes.Each strategy is a relatively small, localized repair; prefix canonicalization makes serialization deterministic, while cache-boundary fixes keep changing content outside the cached span.Source: C. RQ3: Fixing Strategies · p. 6, Figure 3 · p. 6
  • 95 of 120 fixes are predicted by assigning each root cause its most common strategy, with U(fix | rc) = 0.72.Five root causes map to a single strategy every time; three cross-cutting causes account for all 25 modal-fix errors.Source: C. RQ3: Fixing Strategies · p. 6
  • 82 of 120 fixes (68%) use four static, local, or report-only strategies; only cache-boundary fixes and lifecycle guards reach into runtime structure.The most common repair adds or corrects a single guard or arithmetic on already-correct token counts.Source: C. RQ3: Fixing Strategies · p. 6
  • Because defects are mostly silent, diagnosis—not repair—is difficult; most fixes are reusable templates of small, static, local edits.The reported 68% share refers to small, static, local edits.Source: C. RQ3: Fixing Strategies · p. 7

VI. METHODOLOGY

  • CACHELINT models each mis-integration as an invocation site paired with a caching-context precondition that must hold.An invocation site can send content to the cache or read an accounting field; a defect is an invocation reached without a required precondition being guaranteed.Source: A. Prompt-Caching Integration Modeling · p. 7

A. Prompt-Caching Integration Modeling

  • The model groups preconditions into prefix stability, provider capability, and accounting safety, but excludes cross-turn mutation and prefix reassembly from its grammar.Those two causes compare prefixes across calls or turns, which a single program state cannot check.Source: A. Prompt-Caching Integration Modeling · p. 7
  • Provider capability is time-varying: a provider change such as Bedrock’s one-hour TTL can make previously correct code defective without an implementation change.CACHELINT’s example pairs a cache invocation with a dated capability condition that detects when a model is cacheable but a hardcoded model-name predicate excludes it.Source: A. Prompt-Caching Integration Modeling · p. 7

B. Context-Guided Mis-Integration Identification

CACHELINT detects only recurring mis-integration patterns tied to a single statically inspectable invocation site. It checks each site’s preconditions using either a dated provider-capability oracle or an LLM guard check.

B. Context-Guided Mis-Integration Identification

  • CACHELINT implements only anti-patterns that are locatable at a single static invocation site and recur across projects.It excludes prefix reassembly and cross-turn mutation because a single program state cannot express their context.Source: B. Context-Guided Mis-Integration… · pp. 7–8
  • For each invocation, CACHELINT checks the relevant precondition against a dated capability oracle or asks an LLM whether a discharging barrier reaches the site.The guard check replaces an inter-procedural backward slice, which the paper describes as brittle under dynamic dispatch and reflection.Source: B. Context-Guided Mis-Integration… · p. 8
  • Capability assumptions are compared with a dated, sourced provider matrix, allowing the same one-hour-TTL code to be a BUG on Bedrock but OK on Anthropic.Provider behavior changes are handled by updating the matrix and rerunning CACHELINT, rather than changing its CodeQL queries or guard-check prompts.Source: B. Context-Guided Mis-Integration… · p. 8

C. Implementation

CACHELINT uses language-specific CodeQL queries to locate anti-pattern sites and a frontier LLM to check whether a protective barrier reaches each site. Its evaluation uses a labeled in-sample dataset and a broader held-out repository set.

C. Implementation

  • CodeQL queries locate anti-pattern sites across Python, JavaScript/TypeScript, Ruby, Go, and C#; a frontier LLM checks whether a barrier reaches each site.The LLM sees the invocation site and enclosing function and answers a targeted barrier question; the check is not sound and can miss or posit guards.Source: C. Implementation · p. 8

VII. EVALUATION

  • Dataset A contains 10 studied applications and 120 labeled defects linked to merged fixes, but its recall is an in-sample upper bound because those fixes informed the rules.Dataset B contains the other 571 repositories, is held out from rule derivation, and includes SDKs, frameworks, and single-purpose tools excluded from the empirical study.Source: §VII · p. 9
  • Precision is measured on both datasets, while recall is measured only on Dataset A because Dataset B has no denominator of known defects.Warnings are labeled true or false positives after manual inspection in source context; a missed labeled defect in Dataset A is a false negative.Source: §VII · p. 9

A. RQ4: Detection Effectiveness

CACHELINT identifies labeled mis-integrations with 0.71 precision and 0.78 recall, and finds previously unknown defects across a held-out set of 571 repositories with 0.86 precision. Most missed defects require reasoning across calls or turns.

A. RQ4: Detection Effectiveness

  • 0.71 precision and 0.78 recall on Dataset A; 0.86 precision across 571 repositories, where CACHELINT found 32 true positives.On Dataset A it reported 132 potential defects, including 94 true positives, and missed 26 of 120 labeled defects.Source: A. RQ4: Detection Effectiveness · p. 9
  • 24 of the 26 Dataset A false negatives required comparing prompts across calls or turns, beyond CACHELINT’s single-site scope.The other two were in-scope volatile-value defects introduced outside the enclosing function inspected by the guard check.Source: A. RQ4: Detection Effectiveness · p. 9
  • Most Dataset B findings involved model-name gating or volatile cached-prefix values; remaining cases involved marker-budget and usage-miscount defects.Examples include a hard-coded Claude model gate in dirac and a changing timestamp in n8n’s cached system prompt.Source: A. RQ4: Detection Effectiveness · p. 9
  • Five Dataset B false positives arose from already-satisfied caching requirements or conservative matching, including stable day-level timestamps and test-only model names.The authors reproduced all 32 true positives, triggering runtime errors for five crash defects and inspecting request/response pairs and cache-usage fields for 27 silent defects.Source: A. RQ4: Detection Effectiveness · p. 9

B. RQ5: Ablation Study

Removing either the LLM guard check or the dated capability oracle sharply reduces CACHELINT’s precision. In real-world use, maintainers confirmed 21 reports and fixed 18 defects.

B. RQ5: Ablation Study

  • Precision falls from 0.86 to 0.31 without the guard check, which otherwise filters benign stability and accounting sites.The ablated detector flags a day-level timestamp and safely guarded cache-token accounting as defects.Source: B. RQ5: Ablation Study · p. 10
  • Precision falls to 0.05 without the dated oracle, which must distinguish genuine capability mismatches from benign model gates.The ablation reported 349 candidate sites; the model answered only five of 33 dated one-hour-TTL capability facts confidently and correctly.Source: B. RQ5: Ablation Study · p. 10
  • Removing the guard check reduced true positives from 32 to 18, while removing the oracle reduced them to 17.Across five runs, full CACHELINT’s precision varied by 0.0 to +0.02, due to one intermittently reported false positive.Source: B. RQ5: Ablation Study · p. 10

C. RQ6: Real-World Application

  • Maintainers confirmed 21 of the 32 reported true-positive findings and fixed 18; none had been rejected at the time of writing.Confirmations included widely used projects such as AutoGPT, n8n, and Zed.Source: C. RQ6: Real-World Application · p. 10
  • Confirmed cases included an Anthropic SDK usage miscount, hard-coded model gating in altimate-code, and a volatile cached prefix in iTerm2.The reported fixes added a null check, replaced a model-list check with a capability check, and moved the volatile value outside the cached region, respectively.Source: C. RQ6: Real-World Application · p. 10

VIII. THREATS TO VALIDITY

The conclusion reports CACHELINT’s real-world detections and upstream fixes, while the preceding sections define evaluation boundaries and distinguish this work from related research. Its scope excludes implicit model-side caching.

VIII. THREATS TO VALIDITY

  • CACHELINT covers 96 (80%) of dataset defects, but multi-snapshot defects and implicit model-side caching remain out of scope; taxonomy generalizability beyond ten GitHub agents is uncertain.The taxonomy comes from the ten most-starred multi-provider agent applications on GitHub.Source: §VIII · p. 11

IX. RELATED WORK

  • Tian et al. [14] catalog caching as a prompt-defect category and Lumer et al. [15] evaluate caching strategies; CACHELINT instead targets application integration errors in code.It combines data-flow analysis with provider knowledge.Source: §IX · p. 11

X. CONCLUSION

  • 32 previously unknown mis-integrations were detected across 571 repositories at 0.86 precision, yielding 21 developer confirmations and 18 upstream fixes.The study also characterizes 120 defects from 10 open-source agent systems.Source: §X · p. 11
  • Across 120 defects, failures span request and read sides, mostly preserve model output, and are typically repaired with small, localized edits.The study pairs each caching invocation site with correct-use preconditions, checked against a dated provider-capability oracle or an LLM guard check.Source: §X · p. 11

Related papers

Loading 2610.34817v1…