PaperlayerView exact revision on arXiv

Source-linked AI summary

Close Enough is Nowhere Near: Fidelity-Consumability Decoupling in KV Caches

Adam Deng, Shicheng Liu, Yada Zhu

arXiv:2610.22016v1cs.LG

TL;DR

Text-based communication discards part of the sender’s reading, motivating cache sharing. The paper tests frozen-model substitution, explains its failure through the reader’s competitive attention, and reports augmentation as a different approach that improves transfer.

  • Problem

    Text communication loses information from the sender’s reading and makes the reader pay for that reading again, motivating internal-state sharing.

    Source: §1 · p. 1

  • Method

    The paper studies frozen sender-to-reader cache substitution without reader-aware fitting and contrasts it with retaining the reader’s cache while adding the sender’s as attendable context.

    Source: §1 · pp. 1–2

  • Results

    Sender-side substitution fails under the tested setting, while augmentation substantially improves transfer, can match native-context performance, and continues to work with pooled senders.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The conclusion redirects cache sharing from replacement toward adding another model’s state to the reader’s existing cache.

    Source: §6 · p. 10

    The reported measurements use one frozen reader and one split-context factual-recall probe, and the authors scope their claims to that setting.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multi-agent LLM systems communicate in text, which is lossy and wasteful: the sender throws away the reading it already did. A growing literature instead transmits the KV cache, the natural carrier of that reading. We study the most appealing form of the idea: both models stay frozen and the sender's cache is substituted for the reader's with nothing trained. It does not work, and the obstruction belongs to the reader. A frozen transformer reads only caches close to the one it would have written itself, a region we call its tolerable set. We show that geometric similarity to that cache does not predict whether the reader can use it, and the failure is not gradual, so a more faithful substitute is not the remedy it appears to be. We explain this phenomenon with competitive attention. Experimentally, we test eight diverse substitution methods against two measured anchors: an uninformative cache baseline and the reader's own accuracy from having read the passage. All sit at the baseline. That turns a long tail of separate negative results into one outcome with a stated condition for overturning it. A different design does work: if the reader keeps its own cache and attends over the sender's as extra context, adapting only its queries, it beats its own baseline, and also clears the baseline with eight senders pooled where undirected attention over the same union falls below it. We offer this as a characterization on one probe, not a leaderboard claim. Our central contribution is a redirection: pursuing faithful substitute caches is the wrong target; adding a cache to what the reader already has is the promising one.

1 Introduction and Related Work

Text-based communication discards the sender’s prior reading, motivating cache sharing. This paper tests frozen-model cache substitution and argues that its failure is a property of the reader, while augmentation offers a different path.

1 Introduction and Related Work

  • Text communication makes the reader repeat work and loses information that did not survive projection into words; KV caches offer an already-computed carrier of the sender’s reading.A KV cache is produced during prefill and resides in memory.Source: §1 · p. 1
  • With both models frozen and no reader-aware fitting, sender-side cache substitution fails because the reader accepts only caches in its narrow, reader-specific tolerable set.The paper reports that geometric similarity does not reliably predict transfer.Source: §1 · p. 1
  • The failure is not attributed to depth accumulation: corrupting one layer is reported as damaging as corrupting all twenty-eight.Source: §1 · p. 2
  • Augmentation instead retains the reader’s cache and adds the sender’s cache as attendable context, with adaptation on the reader’s query side.The paper reports improved transfer, performance that can match native-context performance, and continued operation with pooled senders.Source: §1 · p. 2
  • Separating substitution, augmentation, and reader-fitted mapping reconciles prior outcomes; the paper’s negative claim excludes bridges fitted against the reader’s caches.The paper states that such a fitted bridge is not tested in its information-asymmetry probe.Source: §1 · p. 2

2 The Object of Study

The object of study is what a particular frozen reader does with supplied key and value tensors. The paper distinguishes fidelity from consumability and frames substitution and augmentation as different claims.

2 The Object of Study

  • A cache is evaluated by supplying its key and value tensors to the frozen reader in place of recomputing them, then reading the output distribution.The mapping from cache to behavior belongs to the reader.Source: §2 · pp. 2–3
  • The reader’s cache on the true context, C*, anchors its tolerable set: the caches it can still read correctly.The set has positive radius and is strongly anisotropic, so it is not a singleton.Source: §2 · p. 3
  • Fidelity measures geometric closeness to C*, while consumability asks whether the reader can use the candidate; the two are distinct.Source: §2 · p. 3
  • Transfer is the fraction of probe items where the reader’s greedy continuation contains the answer string; on-manifold means a real forward pass of this reader could have produced the object.Source: §2 · p. 3
  • The measured floor is 0.180 and the native-cache reference is 0.665; neither is assumed to be a formal bound.The floor uses the reader’s own cache with the answer masked, while the reference uses its cache of the unmasked context.Source: §2 · p. 3
  • Sender-side substitution is claimed to stay near the floor unless it enters the tolerable set, whereas augmentation can approach the native-cache reference.The stated augmentation leaves C* in place, adds the sender’s cache as attendable context, and adapts only the query side.Source: §2 · p. 3

3 Three Measurements of the Tolerable Set

Three measurements test fidelity, depth of corruption, and selection from genuine keys. Across them, geometric closeness and manifold membership do not ensure that the reader can consume a cache.

3.2 A Single Corrupted Layer of Twenty-Eight

  • With one corrupted layer, accuracy falls from 0.665 to 0.030, and the off-manifold curve stays flat from one through twenty-eight corrupted layers.The remaining 27 layers are bit-identical to the reader’s own forward pass; the matched-cosine on-manifold control instead shows depth-wise accumulation.Source: §3.2 · p. 4
  • Across seven models spanning 6 to 36 layers, full substitution by nearest genuine keys transfers at or below 0.008 in every case.The shallowest model has little headroom above its native-cache reference, limiting its informativeness.Source: §3.2 · p. 4

3.3 Selection from Six Thousand Genuine Keys

  • Selecting genuine keys from a bank of 6,000 does not rescue transfer: the nearest key transfers 0.000, and sweeping rank to k=16 leaves the pattern unchanged.The experiment retrieves the k-th nearest key and interpolates toward it with weight α; the figure reports tolerance as flat in rank and collapsing in mixing weight.Source: §3.3 · p. 4, Figure 1 · p. 5
  • Even the best of nine selection variants transfers 0.000 at cosine fidelity 0.880 and attention-logit correlation 0.928, showing that genuine provenance and higher fidelity are insufficient.The candidate was the best cell across three codebook constructions and three snapping rules.Source: §3.3 · p. 5

3.1 Cosine Similarity Does Not Predict Consumability

  • At cosine 0.9949, the genuine cache transfers at 0.445, while isotropic noise at cosine 0.995 scores 0.005, reversing the similarity ordering.Source: §3.1 · pp. 3–4, Figure 1 · p. 5

4 Why Substitution Cannot Reach the Tolerable Set

The paper locates substitution failure in the reader’s consumption operation rather than in catastrophic amplification or accumulated depth effects. Competitive attention makes tolerance directional and sensitive to which keys win.

4.1 Not Amplification

  • Off-manifold readout gain is only an O(1) ratio, with no layer blowup and logit-stage attenuation, so displacement amplification cannot explain the collapse.Grouped-query attention introduces a real sensitivity factor because several query heads read each key slot, but the paper says it is too small to account for failure.Source: §4.1 · p. 5

4.2 The Softmax Consumption Point

  • Attention is competitive: keys are scored against one another, so small displacements can change which keys win even when geometric distance is negligible.Tolerance requires surviving numerous contests across queries, heads, and layers.Source: §4.2 · p. 6
  • Three measurements support margin structure, rather than distance, manifold membership, or depth accumulation, as the relevant constraint on consumability.Source: §4.2 · p. 6
  • Tolerance is anisotropic: it is wide along directions attention does not resolve and thin along directions it resolves, which a single norm averages together.Same-model compression perturbs a cache the reader wrote itself, while cross-model production need not respect that orientation.Source: §4.2 · p. 6
  • Keys set the competition while values are averaged under fixed weights, so matched-sparsity damage to keys costs more than damage to values.Destroying thirteen of 256 value coordinates is reported as free, while the same damage to keys is fatal.Source: §4.2 · p. 6

5 Consequences: Two Regimes Run and a Third Surveyed

The sweep tests whether cache placement, rather than how a sender-side cache is constructed, divides substitution from augmentation, using fixed readers, probes, and anchors. Its methods serve as instruments on one probe, not as a leaderboard.

5 Consequences: Two Regimes Run and a Third Surveyed

  • Fitted bridges, whose maps are regressed against the reader’s own caches, are outside the sweep; the paper states no claim that they are floor-bounded.Source: §5 · p. 6
  • Figure 2 reports that off-manifold damage is flat after ℓ=1, matched-cosine on-manifold substitution survives eight layers, and the off-/on-manifold readout-gain ratio is 1.47×.The logit-stage gain falls below one, so the measured effect is not constructed from amplification.Source: Figure 2 · p. 7

5.1 Substitution, Eight Ways

  • Across eight diverse methods, substitution stays at or below the floor; even the strongest, a full-width map, scores 0.200, while the genuine-key oracle transfers 0.000.The methods span different capacities, objectives, and construction strategies, but the results are measured on one probe against one pair of anchors.Source: §5.1 · pp. 7–8

5.2 Augmentation Clears the Floor and Reaches the Native-Cache Reference

  • Augmentation retains the reader’s cache and adds the sender’s as attendable context; the adapted arm reaches the native-cache reference, but the regime gap is not a single-variable contrast.The comparison also changes whether the reader keeps its own cache, so it does not isolate placement alone.Source: §5.2 · p. 8

5.3 Scaling the Sender Pool

  • With eight senders, undirected attention over the union falls below the floor, while the adapted bridge scores 0.395 and clears it with a control-subtracted lift of +0.165.The bridge is trained at N=4; N=8 is an extrapolation beyond that condition.Source: §5.3 · p. 8

5.4 What the Sweep Licenses

  • The sweep’s supported conclusion is that placement dominates the tested variations in capacity, objective, and construction, not that the two regimes differ in kind as a matter of principle.The proposed design rule is to supply a cache as context through the reader’s own query pathway, rather than reconstructing a substitute.Source: §5.4 · p. 9

5 Consequences: Two Regimes Run and a Third Surveyed

  • Every sender-side substitution method remains at or below the 0.180 floor, while augmentation clears it and the adapted variant reaches the 0.665 native-cache reference.Figure 3 summarizes the sweep; Table 1 states the floor and native-cache reference, with n=200 throughout.Source: Table 1 · p. 7, Figure 3 · p. 9

6 Limitations and Falsification

The claims are limited to frozen readers, sender-side cache construction, and short factual recall; anchors are measured baselines, not proven bounds. The proposed account is testable through changes to the carrier, readout, and task sensitivity.

Scope

  • The claims cover one reader and one split-context factual-recall probe, where short answers may be especially sensitive to attention-margin changes.Tasks whose answers depend on many tokens may be more forgiving; augmentation results are same-architecture, with no cross-family claim.Source: §6 · p. 9

Falsification

  • The account predicts a more permissive reader for pre-softmax carriers and tasks less sensitive to individual attention margins.Preliminary residual-stream runs are directionally consistent, but use different anchors and sharing protocols rather than holding displacement matched.Source: §6 · pp. 9–10
  • Separately varying readout sharpness and depth tests whether tolerance contracts with sharper attention rather than added layers.The text predicts contraction with increased sharpness, but not with added layers at fixed sharpness.Source: §6 · p. 9

Conclusion

  • Replacement is the excluded design; augmentation instead keeps the reader’s cache and adds the sender’s state as attendable context.The conclusion scopes the obstruction to replacement under sender-side construction, not internal communication between frozen models in principle.Source: §6 · p. 10

Use of AI Assistance

The paper reports AI assistance across implementation, experiments, analysis, writing, and literature work, with human review of the resulting code and claims. Its probe uses split-context factual recall with measured floor and native-cache anchors, while perturbations are calibrated to target cosine and checked empirically.

Use of AI Assistance

  • Generative AI assisted with coding, experiment design, hypothesis refinement, analysis, interpretation, drafting, literature searches, figure creation, and reference formatting.The authors report not using it to generate synthetic data, develop theoretical models, or formulate or prove mathematical claims.Source: Use of AI Assistance · p. 10
  • The authors reviewed AI-assisted work, traced quantitative claims to run records, checked experiment code and controls, and verified prose and cross-references.They state responsibility for the final paper and its artifacts.Source: Use of AI Assistance · p. 10

A Probe and Anchors

  • The SQuAD probe masks a short answer in the reader’s passage while a sender reads the full passage, then scores eight-token greedy generations over 200 items.The floor is 0.180 from the reader’s masked cache; the native-cache reference is 0.665 from its full-passage cache, and both are re-measured per run.Source: A Probe and Anchors · p. 12

B Cosine-Calibrated Perturbation

  • Noise is scaled to target a cache cosine and the achieved cosine is verified per item; genuine-cache interpolation is restricted to shared token positions.The genuine-cache blend leaves positions outside the matching-length overlap at the reader’s own values.Source: B Cosine-Calibrated Perturbation · p. 12

C Layer-Budget Sweep

The layer-budget sweep contrasts matched-cosine off-manifold corruption with genuine keys, then tests cross-reader transfer. The oracle interpolation and codebook grids find no graded recovery from increasingly faithful sender-side substitutes.

C Layer-Budget Sweep

  • The layer sweep corrupts a prefix of the reader’s cache, while its on-manifold arm substitutes nearest genuine keys and matches the off-manifold arm’s target cosine by construction.The on-manifold keys come from a bank of 6,000 genuine keys per layer; the noise arm’s cosine match is not independently re-logged.Source: C Layer-Budget Sweep · pp. 12–13
  • At target key cosine 0.822, off-manifold noise exhausts its budget after one corrupted layer, while genuine on-manifold keys are initially harmless and decay gradually.The layer sweep corrupts a prefix and leaves the remaining layers bit-exact; the table reports a floor of 0.180 and native-cache reference of 0.665.Source: Table 3 · p. 13
  • Nearest-genuine-key substitution transfers at 0.000 in six of seven readers and 0.008 in the seventh, whose 0.025 native reference leaves little headroom.The cross-architecture check uses each model’s own floor and native-cache reference.Source: C Layer-Budget Sweep · p. 13

D Oracle Interpolation Grid

  • For oracle interpolation, every cell with α ≥ 0.5 transfers at or below 0.02, and neighbor rank yields no graded tolerance.The α = 0 condition restores the true key and reproduces the floor by construction.Source: D Oracle Interpolation Grid · p. 13
  • Across nine codebook-and-snapping combinations, transfer is 0.000–0.010; the best-fidelity cell has cosine 0.880 and logit correlation 0.928 but transfers at 0.000.The three PCA cells have byte-identical metrics, so they are not independent measurements.Source: D Oracle Interpolation Grid · p. 13

E Conditioning Measurement

The conditioning measurements find order-one on/off-manifold gain ratios and no layerwise cliff. A pilot quantization comparison also finds greater attention-output error and query-subspace displacement than matched-norm Gaussian noise.

E Conditioning Measurement

  • At the attention-logit stage, off/on gains are 0.506/0.343 (1.475×); at the softmax-output stage, they are 2.192/1.482 (1.479×).Logit-stage gains are below one, indicating attenuation; an analytic Jacobian bound predicts measured per-layer gain at Pearson correlation 0.999.Source: E Conditioning Measurement · p. 14
  • Per-layer gains rise from about 0.11 at layer 1 to roughly 0.72 by layer 6, with no layer showing a cliff.The governing constant is √12 ≈ 3.46 at group size 6.Source: E Conditioning Measurement · p. 14
  • With displacement norm fixed at 0.05, quantization has attention-output error 0.153 versus Gaussian noise’s 0.129 and query-subspace share 0.447 versus 0.354.The six-probe pilot treats the 0.84× ratio as indicative rather than a tight estimate; Int8 displacement norm is 0.0048.Source: E Conditioning Measurement · p. 14

F Production-Method Sweep

The production sweep distinguishes cache substitution from augmentation, while pooled-sender results favor an adapted bridge over undirected attention as distractors increase.

F Production-Method Sweep

  • Substitution installs a produced cache in the reader’s slots, whereas augmentation retains the reader’s cache and cross-attends over the sender’s.The adapted augmentation arm tunes query-side projections with LoRA at rank 16, using four epochs and 200 training items.Source: F Production-Method Sweep · p. 14
  • The row-space target ranges from 0.01 to 0.085 across paired objectives, a spread larger than several neighboring-row gaps, while all three runs remain below floor.The by-construction row scores 0.01 in two runs and 0.015 in two others.Source: F Production-Method Sweep · p. 14
  • The zero-parameter cross-attention condition scores 0.555 at n=200, and runtime checks verify cache reconstruction and native greedy decoding.A separate multi-sender run reproduces the condition at 0.550.Source: F Production-Method Sweep · p. 14
  • The adapted cross-attention ladder scores 0.555 without adaptation, 0.600 with query-only adaptation, 0.640 with keys and values, and 0.695 with all three.After control subtraction, keys and values (+0.550) exceed the all-three variant (+0.495), so the ordering among adapted variants is not reliable.Source: F Production-Method Sweep · p. 14

G Scaling the Sender Pool

  • At N=8, the adapted bridge clears floor 0.180, while undirected attention falls below it by N=4; controls rise with N.Because controls rise with N, the control-subtracted lift is the appropriate figure of merit.Source: Table 5 · p. 15, G Scaling the Sender Pool · p. 15

H The Sparse-Damage Experiment

Sparse damage does not rescue substitution at matched fidelity, and key/value perturbations produce sharply different outcomes. Preliminary carrier results and the fitted-bridge comparison have distinct scope limits.

H The Sparse-Damage Experiment

  • At matched cosine, sparse and dense damage track closely: at p=0.01 scores are 0.197 and 0.177, while at p=0.05 they are 0.003 and 0.008.The comparison pairs each sparse condition with a dense perturbation calibrated to the same achieved cosine.Source: H The Sparse-Damage Experiment · p. 15
  • Accuracy drops from 0.180 at p=0 to 0.003 at p=0.05, but the answer-masked self-cache makes the floor also the ceiling, so 0.197 at p=0.01 is not a gain.The zero-damage control scores 0.192, bracketing the p=0.01 result.Source: H The Sparse-Damage Experiment · p. 15
  • At matched sparsity, destroying 13 of 256 value coordinates has no measurable cost (0.175), while the same key damage scores 0.018.The fidelity ordering reverses the outcome: keys at cosine 0.9949 score 0.128, while values at the higher cosine 0.9982 score 0.028.Source: H The Sparse-Damage Experiment · pp. 15–16
  • Destroying single heads scores 0.000–0.015 and the first two layers score 0.000, but the apparent late-layer tolerance is not established.This is a single-seed result covering four of 28 layers; the layer-27 score of 0.250 is at the floor rather than above it.Source: H The Sparse-Damage Experiment · p. 16

I Pre-Softmax Carrier: Preliminary Numbers

  • Residual-stream sharing remains preliminary: it continues to work to roughly cosine 0.90, unlike key substitution, but the protocols and reference anchors are unmatched.The controlled comparison is specified for matched norm, cosine, and corresponding layers.Source: I Pre-Softmax Carrier: Preliminary… · p. 16

J Fitted Bridges: Scope of the Negative Claim

  • The negative claim excludes fitted bridges; cited within-family results retain quality, while cross-family transfer remains open.The passage distinguishes receiver-cache reconstruction from methods optimized against receiver behavior.Source: J Fitted Bridges: Scope of the… · p. 16

Related papers

Loading 2610.22016v1…