PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

Aditi Patodiya

arXiv:2609.04748v1cs.SEcs.DCcs.LG

TL;DR

Prefix caching is widely treated as a transparent serving optimization, but its reproducibility cost under realistic agent workloads has not been established. This paper uses controlled repeated executions across engines, workloads, and weight formats to isolate cache-induced divergence. It finds that quantization sharply increases trajectory changes, while cache-enabled serving is deterministic only when cache state is restored and does not shift aggregate task accuracy.

  • Problem

    The paper addresses the missing evidence about whether prefix caching preserves reproducible outputs when serving identical requests, especially in multi-turn agents and under quantization.

  • Method

    The study compares paired cache-enabled and cache-disabled executions with fixed models, decoding, seeds, serial batch-one requests, repeated runs, and controlled cache-state experiments across two engines.

  • Results

    36.2 percent of agent episodes diverge at 16-bit versus 75.0 percent at four-bit, while cache-disabled execution differs in 0 of 800 episodes and cache-state restoration makes each path reproduce.

  • Takeaways & Limitations

    Cached serving is deterministic given cache state, but practical reproducibility fails when that state is invisible to requests and uncontrolled by default.

  • Takeaways & Limitations

    The primary reproducibility measure compares emitted token identifiers, so semantically equivalent answers expressed with different wording count as divergent.

Abstract

from arXiv · show

Prefix caching, in which a serving engine reuses the key and value tensors of a shared prompt prefix across requests, is enabled by default in the major open-source stacks and treated as a transparent optimization. We measure what it costs in reproducibility, and find that the cost rises sharply with weight quantization. Holding the model, decoding parameters, seed, and request order fixed, and issuing every request serially at batch size one, we ran an eighty-episode multi-turn agentic tool-use workload with caching enabled and disabled across two engines and four weight formats. Enabling the cache changed the agent's trajectory on 36.2 percent of episodes at 16-bit precision and on 75.0 percent at four-bit, a gradient that survives re-measurement under a controlled cache configuration. With caching disabled, repeated execution was bit-identical in every configuration, 0 of 800 episodes, which bounds other sources of nondeterminism at 0.5 percent. Repeated cache-enabled runs did diverge, and three experiments locate the cause: a single server-level prompt-cache setting moves run-to-run divergence by 37.5 percentage points, execution order acts only while that setting is active, and restoring cache state makes the cached and recompute paths each reproduce on 40 of 40 items while still differing from each other on 14. Cached serving is deterministic given cache state, and irreproducible in practice because that state is absent from the request and never reset by default. A single-turn bridge shows the divergence reaching task outcomes without shifting aggregate accuracy. We release the harness, logs, and analysis pipeline.

I. INTRODUCTION

The paper tests whether prefix caching remains a transparent optimization when reproducibility matters. Controlled experiments show that caching can change agent trajectories, with quantization amplifying divergence, while the effect is attributable to cache state rather than general serving noise.

  • Motivation: Prefix caching reuses previously computed key and value tensors, but can change floating-point accumulation order and token decisions.Those token changes can alter tool calls and therefore subsequent observations in multi-turn agent episodes.
  • Experimental framing: The study fixes model, decoding, seed, batch size, and request order while comparing cache-enabled and cache-disabled serving across two engines.It uses repeated runs of an agentic tool-use workload and a single-turn mathematics bridge.
  • Central finding: Cache-disabled serving is bit-identical across repeated runs, whereas cache-enabled serving changes outcomes in a large fraction of episodes.This separates cache-associated divergence from general nondeterminism in the serving system.
  • Mechanism: A single server-level prompt-cache setting shifts run-to-run divergence by 37.5 percentage points, while execution order is ruled out as the cause.Restoring cache state shows that cached serving is deterministic given that state, which deployments generally neither report nor control.
  • Central finding: Quantization substantially amplifies cache-induced divergence, establishing a first characterization of their interaction.The contribution concerns reproducibility rather than a general claim that quantization reduces accuracy.
  • Artifact: The released artifact includes a harness, raw per-request logs, and an analysis pipeline that regenerates the paper’s reported numbers.The release supports auditing cache exposure and reproducing the analysis.

II. BACKGROUND AND RELATED WORK

Prior serving research emphasizes the efficiency of shared attention-state reuse, but generally leaves behavioral reproducibility under warm versus cold caches unmeasured. This paper positions its contribution as a controlled extension from single-turn cached inference to multi-turn agents, quantization, and cross-engine testing.

  • Caching systems: Modern serving systems use shared prefix or prompt caches to improve throughput, latency, and memory efficiency.PagedAttention, RadixAttention, Prompt Cache, and related systems make reuse across requests practical.
  • Open gap: Existing evaluations commonly report cost and time-to-first-token benefits without testing whether warm-cache and cold-cache requests return the same answer.More than five hundred agent sessions were evaluated for efficiency, while the behavioral question remained open.
  • Related nondeterminism: Batch invariance research addresses variation caused by grouping requests, whereas this study issues requests serially at batch size one.The design therefore isolates a different potential source of divergence from batching.
  • Closest prior work: Prior work found that FP16 cached decoding can differ numerically from recomputation in single-turn arithmetic tasks.This paper extends that line of work to multi-turn agent episodes, quantization, and independent serving implementations.

C. QUANTIZATION AND BEHAVIORAL CHANGE

The section frames quantization as a factor that changes sensitivity to cache-induced numerical perturbations, not as a demonstrated cause of lower average accuracy. The study uses controlled paired experiments spanning multiple engines and serving configurations.

  • Quantization and behavior: Quantization can preserve aggregate accuracy while changing many individual predictions, making average accuracy insufficient to characterize behavioral churn.The paper distinguishes disagreement on individual items from degradation in aggregate competence.
  • Research questions: The study asks whether coarser weight quantization amplifies or dampens cache-induced divergence, alongside questions about baseline determinism, backend generality, and task correctness.These research questions separate reproducibility, cross-engine behavior, and outcome-level effects.
  • Experimental protocol: Each configuration holds model weights, quantization, engine build, sampling parameters, request order, and hardware fixed apart from the cache setting.Greedy decoding, a fixed seed, and serial batch-one requests reduce confounding from sampling and continuous batching.
  • Experimental protocol: The key-value cache remains 16-bit across weight formats, so cache precision does not vary with quantization.Cache exposure is verified from server responses where possible and from engine logs otherwise.
  • Serving implementations: The comparison uses llama.cpp and vLLM, with engine-specific cache controls and separately launched server processes where required.The implementation details establish the scope of backend coverage and cache control.

C. WORKLOADS

The workload design combines multi-turn agentic tool use with a single-turn mathematics bridge, then adds controlled experiments to separate cache-state dependence from execution order and residual randomness. The protocol repeatedly tests each path against itself before attributing differences to caching.

  • Workloads: The primary workload is multi-turn agentic tool use from the Berkeley Function Calling Leaderboard, with stateful API interactions and roughly ten requests per episode.Episodes contain several user turns and multiple agent steps, allowing early token changes to propagate through later behavior.
  • Workloads: The secondary GSM8K workload bridges prior single-turn cached-inference work and provides a clear outcome measure with room for correctness flips.Models solve roughly nine in ten items, leaving incorrect cases observable in both directions.
  • Workload protocol: Each mathematics item is run twice on the recompute path and twice on the cache-hit path, so path differences count only when each path reproduces itself.This per-path determinism check prevents residual randomness from being mistaken for cache-induced divergence.
  • Mechanism experiments: Mechanism experiments focus on Qwen2.5-7B at Q4_K_M under llama.cpp, the configuration with the highest measured divergence.They vary execution order, a separate host-memory prompt-cache flag, and direct restoration of cache state.

E. MEASURES

The study measures divergence by comparing repeated executions and cache-enabled versus cache-disabled arms under tightly controlled serving conditions. It establishes bit-identical cache-disabled behavior before attributing differences to caching.

  • Divergence measure: Divergence is counted from the first differing prompt, emitted token sequence, or number of requests in an episode.Token identifiers are used where available, while decoded tokens are used for the vLLM endpoint.
  • Outcome scoring: Task scoring uses the benchmark checker, comparing final environment state and invoked methods for agents and final numeric values for mathematics.Scoring is independent of the instrumentation.
  • Uncertainty: Wilson intervals are reported, with moving-block bootstrap intervals additionally computed because fixed-order episodes may be dependent.The bootstrap preserves dependence between neighboring episodes.
  • Baseline determinism: 0 of 800 episodes diverged when caching was disabled across ten configurations, bounding repeated-execution divergence at 0.5%.The runs used identical prompts, emitted tokens, and request counts under the reported conditions.
  • Baseline determinism: The cache-disabled control rules out sampling, scheduler, batch-composition, and unspecified engine noise as explanations for later divergence under these conditions.No episode raised an error in any cell.
  • Experimental design: The evaluation compares cache settings with greedy decoding, a fixed seed, batch size one, and serial requests, while repeating each configuration twice.The paired design supports both cross-arm and within-arm comparisons.

1) Cache-enabled and cache-disabled execution differ

Cache-enabled and cache-disabled executions frequently produce different agent trajectories, while repeated cache-enabled executions can also diverge. The effect appears early and is strongly influenced by the server-level prompt-cache layer.

  • Cache-enabled and cache-disabled execution differ: 36.2% to 91.2% of episodes changed trajectory between cache-enabled and cache-disabled arms across configurations.Controlled remeasurement reproduced the cross-arm rates within a few points, including 81.2% versus 75.0% at Q4_K_M.
  • Cache-enabled and cache-disabled execution differ: The first differing request had a median index between 1 and 4, while the first differing token had a median index between 3.5 and 17.Early token-level differences can compound into large episode-level trajectory changes.
  • Cache-enabled and cache-disabled execution differ: 8.8% to 77.5% of episodes changed when the cache-enabled arm was repeated across the ten configurations.These raw within-arm results motivated isolating the server-level prompt-cache setting.

C. THE MECHANISM: STATE CARRIED BETWEEN RUNS

Three controlled experiments identify carried cache state as the source of run-to-run divergence. The server-level prompt-cache layer, execution order, and state restoration together show that cached serving is reproducible when its state is controlled.

  • C. THE MECHANISM: STATE CARRIED BETWEEN RUNS: The server-level prompt-cache layer changed repeated-run divergence from 1.2% to 38.8%, a 37.5-percentage-point difference.The comparison used Qwen2.5-7B at Q4_K_M under llama.cpp with 80 episodes per condition.
  • C. THE MECHANISM: STATE CARRIED BETWEEN RUNS: With the prompt-cache layer active, inserting a cache-disabled pass raised divergence from 38.8% to 77.5%; with it disabled, divergence remained 1.2%.The two orderings produced byte-identical output in every cell when the layer was disabled.
  • C. THE MECHANISM: STATE CARRIED BETWEEN RUNS: Divergent episodes across execution-order quartiles numbered 16, 6, 6, and 3, with a permutation-test result of p = 0.0002.The corresponding grid configuration showed no trend, with p = 0.59.
  • C. THE MECHANISM: STATE CARRIED BETWEEN RUNS: After restoring cache state, both recompute and cache-hit paths reproduced on 40 of 40 items, while the paths differed on 14 of 40.The control verified cold and warm cache conditions using reported cached-token counts.
  • C. THE MECHANISM: STATE CARRIED BETWEEN RUNS: The reset control establishes reproducibility with the prompt-cache layer disabled and does not by itself characterize the default configuration.The default configuration is covered by the first two experiments.
  • C. THE MECHANISM: STATE CARRIED BETWEEN RUNS: Table 1 crosses execution order with the server-level prompt cache and reports the fraction of 80 episodes whose repeated cache-enabled trajectory changed.Its stated comparison is whether order matters while the prompt-cache layer is active.
  • C. THE MECHANISM: STATE CARRIED BETWEEN RUNS: The controlled results support treating cached serving as deterministic given cache state, while deployments do not include, report, or reset that state by default.The prompt-cache layer is distinct from the per-request prefix cache and persists whole conversation states.

D. RQ3: QUANTIZATION AMPLIFIES DIVERGENCE

Coarser quantization amplifies divergence between cache-enabled and cache-disabled execution, and the pattern persists under controlled cache settings. The effect appears across serving engines, although its magnitude varies.

  • Quantization gradient: 36.2% at 16-bit, 61.3% at Q8_0, 75.0% at Q4_K_M, and 77.5% at Q3_K_M cross-arm divergence occurred for Qwen2.5-7B under llama.cpp.Controlled remeasurement reproduced the pattern at 40.0%, 81.2%, and 77.5% for the tested 16-bit, Q4_K_M, and Q3_K_M cells.
  • Interpretation: The 16-bit divergence shows that quantization is a multiplier rather than the cause of cache-versus-recompute differences.The cache-versus-recompute pattern also survives remeasurement with the server-level prompt cache disabled.
  • Quantization gradient: z = 5.68 and p = 1.3 × 10^-8 support an ordered increase across the four Qwen2.5-7B weight formats.
  • Quantization gradient: Q4_K_M and Q3_K_M differ by only a few points, with overlapping intervals in both original and controlled measurements.The ordering of these two coarsest settings therefore carries little interpretive weight.
  • Cross-engine comparison: vLLM reproduces the split between bit-identical cache-disabled runs and divergent cache-enabled runs, but its magnitudes differ from llama.cpp.At 16-bit, repeated cache-enabled execution changed 8.8% of episodes on vLLM versus 20.0% on llama.cpp, while vLLM cross-arm divergence was 62.5%.

F. RQ5: INSTABILITY WITHOUT DIRECTIONAL BIAS

The study finds instability without evidence of directional accuracy bias. Agentic trajectories are difficult to use for outcome inference, so the single-turn bridge supplies the outcome analysis and reveals bidirectional correctness changes with unchanged aggregate accuracy.

  • Agentic outcome boundary: 1.2% to 18.8% agentic success rates leave the workload near the floor, preventing supported conclusions about task success.The paper therefore treats the agentic results as trajectory-level evidence rather than outcome evidence.
  • Single-turn bridge: 3.5% of single-turn items diverged at 16-bit, rising to 41.5–48.0% under quantization and 45.4% in the 500-item Q4_K_M run.By comparison, 36.2% of agent episodes diverged at 16-bit because changed tokens propagate through multi-turn episodes.
  • Directional bias: 13 cached-path wins versus 7 recompute-path wins produced exact McNemar p = 0.26, with no aggregate accuracy shift.No individual configuration reached significance, supporting instability rather than degradation at the tested resolution.
  • Measurement correction: 11% to 38% of responses were initially mis-scored for formatting, inflating the apparent flip count roughly fourfold before offline numeric rescoring.The corrected analysis uses stored response text and numeric comparison, and the released artifact preserves both score versions.
  • Relation to concurrent work: The single-turn result agrees with concurrent work on suffix divergence and bidirectional correctness flips, but differs on directional accuracy bias.The comparison is limited because the studies use different models, item sets, and determinism controls.

V. DISCUSSION

Caching preserves average accuracy in the tested settings but changes reproducibility and complicates replay. The paper argues for reporting cache configuration and controlling or recording cache state when repeatability matters.

  • Practical consequence: A cache-enabled endpoint does not guarantee repeatable outputs for audit or replay because the server’s cache state is absent from the request.A failure may not reproduce when replay encounters different cache state.
  • Mitigation: Disabling the server-level prompt cache reduced run-to-run divergence from 38.8% to 1.2% while leaving prefix caching enabled.Restarting the server produced the same reduction in the second engine.
  • Mitigation: Pinning cache configuration and resetting state at run boundaries can preserve most throughput benefits while restoring repeatability.Recording cache exposure per response provides a narrower way to detect irreproducibility after the fact.
  • Evaluation practice: Published evaluations generally omit cache configuration, despite cache-enabled scores sampling behavior influenced partly by serving history.The paper recommends reporting the serving engine, version, and prefix-cache status alongside decoding parameters.

C. RELATIONSHIP TO PRIOR SINGLE-TURN FINDINGS

The paper refines prior single-turn findings by separating numerical divergence from directional quality bias and by testing reproducibility controls directly. Its conclusions remain bounded by strict token-level measurement, configuration threats, incomplete runs, and limited external scope.

  • Prior findings: The single-turn bridge reproduces cached-versus-recomputed divergence but not the prior study’s systematic accuracy bias on these models and items.Both paths were individually deterministic, disagreed substantially, and produced correctness flips in both directions.
  • Construct validity: Token-identifier divergence is intentionally stricter than semantic equivalence, while benchmark checkers are used separately for quality outcomes.This distinction keeps reproducibility and task quality as separate measurements.
  • Internal validity: 0 of 800 cache-disabled episode repetitions were divergent, bounding other nondeterminism sources at 0.5% under the study conditions.The design also fixes batch size at one and uses serial requests, while the initial default prompt-cache setting required controlled remeasurement.
  • Cache-state control: 40 of 40 restored-cache items reproduced on each path, yet the cached and recomputed paths still differed on 14 of 40.This supports cache-state dependence rather than residual randomness within either path.
  • Run completeness: The controlled quantization cells at 16-bit and Q3_K_M were run once rather than twice, so they contribute cross-arm values but no within-arm repeat.One vLLM ordering cell lacked an end marker and another was not collected; reported figures use completed cells.
  • External validity: The study covers open-weight 7–14B models, one consumer-class accelerator, and English agentic and mathematics workloads, not frontier, hosted, or multi-tenant settings.The authors make no claim that behavior outside these conditions is the same.
  • Statistical conclusion validity: Agentic success rates near the floor prevent outcome conclusions there; outcome claims instead rely on mathematics accuracy of 89% to 93%.Trajectory-level measurements remain usable because they do not depend on task success.

VII. CONCLUSION

Prefix caching is not a transparent optimization: it makes outputs depend on cache state, with divergence increasing under coarser quantization and longer episodes. The paper recommends reporting cache configuration and releases the materials needed to reproduce its analyses.

  • Reproducibility: 0 of 800 episodes differed across repeated cache-disabled executions, whereas cache-enabled executions were not repeatable.This result held across all ten configurations spanning two independent engines.
  • Cause isolation: A single server-level prompt-cache setting moved run-to-run divergence by 37.5 percentage points, while restoring cache state made each execution path reproduce.The cached and recomputed paths still differed from each other after state restoration.
  • Quantization and trajectory divergence: 36.2 percent of agent episodes changed at 16-bit precision versus 75.0 percent at four-bit when caching was enabled.Longer episodes further amplify divergence because changed tokens propagate through subsequent turns.
  • Task outcomes: Individual answers changed correctness in both directions while aggregate accuracy remained unchanged, separating instability from degradation.The paper characterizes caching as a repeatability problem rather than a general accuracy loss.
  • Practical recommendation: Evaluations and deployment records should report the serving engine, version, prefix-cache status, and decoding parameters.Uniform per-request cache-exposure reporting would make irreproducibility detectable after the fact.
  • Reproducibility materials: The released harness, raw per-request logs, analysis pipeline, engine versions, model checksums, and launch flags support reproduction of the study.The repository scripts regenerate the paper's tables and figures from the recorded logs.
Loading 2609.04748v1…