PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, Bita Darvish Rouhani

arXiv:2608.03893v1cs.LG

TL;DR

Model swaps make receivers prefill accumulated context again, and within-model prefix caching does not cover cross-model reuse. The paper proposes a closed-form per-head ridge mapper for transferring KV caches; four matched-KV pairs retain 73–98% of standalone accuracy, while two degrade sharply.

  • Problem

    Model swaps in long sessions require the receiver to prefill accumulated context again, while prefix caching mitigates this only within a single model.

    Source: §1 · p. 1

  • Method

    A closed-form per-head ridge mapper selects top-k source layers and maps KV using RoPE-stripped keys and a small calibration set.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Four matched-KV pairs retain 73–98% of standalone accuracy across five benchmarks, while two pairs fall to 42–44% Avg.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Attention-output cosine predicts HellaSwag retention better than R2 across 12 matched-KV pair evaluations, with Pearson r=+0.57 versus r=−0.20.

    Source: §1 · p. 2, §6 · p. 10

    All six evaluated pairs are matched-KV; transfer to mismatched-KV pairs is untested.

    Source: §5 · p. 9

Abstract

from arXiv · show

Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-model KV cache transfer, where the receiver reuses the source's KV cache, skipping prefill. We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension. On Qwen3 14B->32B, one source layer explains 56% of variance in the target's keys and 32% in values, rising to 79% and 65% with multiple source layers. Building on this, we design a closed-form ridge mapper that operates per head and proceeds in three steps. First, for each target layer we select the top-k most predictive source layers and concatenate their KV as input. Second, we strip RoPE from the keys before mapping, so the fit is position-free and reusable across context lengths. Third, we fit ridge regression on a small calibration set of 500 FineWeb-Edu sequences of 1,024 tokens each. Surprisingly, across six pairs in three families, this linear mapper retains 73-98% of the receiver's standalone-prefill accuracy on four pairs, while two degrade sharply. A nonlinear MLP recovers up to +37 pp HellaSwag retention on the failures. The mapper runs 2.7-25x faster than re-prefill and remains stable across multi-turn handoff, making cross-model KV cache transfer practical.

1 Introduction

Model swaps force receivers to prefill accumulated context again, while existing prefix caching applies only within one model. The paper maps a source model’s prefilled KV cache into the target’s format, with strong retention on four evaluated pairs.

1 Introduction

  • Cross-model KV cache transfer addresses the repeated prefill incurred when model swaps require receivers to process accumulated context again.The study focuses on within-family transfer, which can be bidirectional: small-to-large for quality or large-to-small for cost.Source: §1 · p. 1
  • Prior cross-model approaches use gradient-based training or strong architectural assumptions, whereas this work investigates a small-sample, gradient-free linear fit.The proposed transfer maps cache values from one model to another.Source: §1 · pp. 1–2
  • A closed-form per-head ridge mapper uses top-k predictive source layers and RoPE-stripped keys to make its fit reusable across context lengths.The approach fits from a small calibration set and maps the source cache into the target’s expected format.Source: §1 · p. 2, Figure 1 · p. 2
  • Four matched-KV pairs retain 73–98% of standalone accuracy across five benchmarks, with 2.7–25× lower prefill latency than re-prefill in the small-to-large direction.Two other pairs degrade sharply.Source: §1 · p. 2

2 Background and motivation

The section frames transfer as mapping source KV cache representations into a target model’s cache space and evaluates success by downstream behavior. It motivates a linear mapper with evidence of cross-model KV structure and complementary information across source layers.

2.1 Problem formulation

  • The formulation seeks a mapping from the source cache to an estimated target cache that preserves target outputs, measuring transfer by downstream accuracy rather than reconstruction metrics alone.Within a family, source and target share a tokenizer, so the input sequence has the same length for both.Source: §2.1 · pp. 2–3

2.2 Prior approaches

  • Prior cross-model reuse methods rely on gradient-based training or architectural constraints, including neural fusers, latent adapters, attention-pattern mapping, and same-architecture KV sharing.The paper asks whether cross-model KV relationships permit a closed-form, training-free mapping.Source: §2.2 · p. 3

2.3 Linear structure in cross-model KV

  • At heatmap peaks, Kstripped R2 reaches 0.81 on Qwen3 14B→32B and 0.65 on Qwen3 8B→32B; RoPE stripping generally sharpens diagonals.The probes fit token-level single-source linear regressions for raw keys, RoPE-stripped keys, and values, then visualize head-averaged R2 by source and target layer.Source: §2.3 · pp. 3–4, Figure 2 · p. 3
  • On Qwen3 14B→32B, k=1 captures 66% of the k=all R2 for Kstripped and 42% for V, with R2 close to its k=all value by k=6.Greedy forward selection adds source layers that increase R2, indicating complementary information across layers and motivating top-k selection.Source: §2.3 · p. 4

3 Mapper design

The mapper fits independent per-head linear projections from selected source-layer KV features to each target layer’s KV. It selects sources by predictive value and maps keys in position-free content space so the fit can transfer across context lengths.

3 Mapper design

  • For each target layer and head, selected source keys and values are concatenated and passed through independent ridge regressions to produce target KV.The mapper repeats this closed-form process independently across all target layers and KV heads.Source: §3 · p. 4

3.1 Per-head ridge regression

  • The mapper fits an independent linear mapping for each target layer and head, using ridge regression to map concatenated source features to target keys and values.The ridge fit uses λ = 0.01 for numerical conditioning; calibration uses 500 FineWeb-Edu sequences of 1,024 tokens, stride-4 subsampled to approximately 128K tokens per target head.Source: §3.1 · pp. 4–5

3.2 Cross-layer source selection

  • Each target layer selects its top-k most predictive source layers, concatenates their KV features, and fits a multi-source ridge regression.Source selection is based on head-averaged R² over RoPE-stripped keys and values, and is the largest single contributor among the three mapper components.Source: §3.2 · p. 5

3.3 Content-space mapping (RoPE factoring)

  • Keys are mapped in RoPE-stripped content space and then re-encoded with target RoPE, keeping the fitted weights position-free.The position-free weights remain valid across different RoPE configurations and positions supported by the target; values are mapped directly because they carry no positional encoding.Source: §3.3 · p. 5

4 Experiments

The experiments assess transfer quality across matched-KV pairs and mapper variants, then test whether transfer remains stable across turns and reduces prefill latency.

4.2 Main results

  • Four of six pairs retain 73–98% of target accuracy averaged across five benchmarks, while two Ministral pairs fall to 42–44% Avg and 11–15% floor-normalized.The higher-retention pairs are Qwen3 14B→32B, Qwen3 8B→32B, Llama 3.1 8B→70B, and Ministral 3B→8B.Source: Table 1 · p. 6, §4.2 · p. 6

4.1 Setup

  • Transfer is evaluated on five accuracy benchmarks plus WikiText-2 perplexity, with k selected using four accuracy benchmarks and GSM8K, CoQA, and latency held out.The k criterion averages log-likelihood benchmark accuracies; per-pair selected k values are used for evaluation.Source: §4.1 · p. 6

4.3 Ablation

  • Cross-layer selection mattered most: reducing k from 8 to 1 lowers K R2 from 0.79 to 0.56.The fully RoPE-coupled variant stays within noise at the 1,024-token fit context, but content-space mapping is preferred for cross-RoPE and longer-context generalization. Calibration is robust across λ and sample counts; CodeAlpaca costs 5.24 pp on HellaSwag while Wikipedia stays within noise.Source: §4.3 · p. 6, Table 2 · p. 7

4.4 Substituting MLP for ridge

  • On ridge failure pairs, the MLP raises HellaSwag retention by 24.3–36.8 pp and places all four evaluated pairs above 90%, but slightly underperforms ridge on its successful pairs.The MLP is a drop-in replacement evaluated on four Qwen3 and Ministral 3 pairs, with the mapper functional form as the changed factor.Source: §4.4 · p. 7, Table 3 · p. 7

4.5 What determines transfer quality?

  • Across 12 matched-KV evaluations, attention-output cosine correlates with HellaSwag retention at r=+0.57, whereas calibration-domain R2 has essentially no correlation (r=−0.20).On ridge failure pairs, the MLP lowers K-concentration and raises attention-output cosine; those changes alone do not guarantee gains when ridge already succeeds.Source: §4.5 · p. 8

4.6 Multi-turn handoff

  • In one Qwen3 14B↔32B CoQA evaluation, the S→L gap widens 1.7 pp over ten turns and L→S drift grows 0.33 pp per turn, without cascading failure within ten turns.Tuning k on the multi-turn task changes drift by at most 2.0 pp.Source: §4.6 · p. 8, Figure 4 · p. 9

4.7 Prefill latency

  • Transfer runs 4–25× faster small-to-large and 3–7× faster large-to-small than re-prefill across sequence lengths from 64 to 32,768 tokens.The large-to-small speedup widens with sequence length because mapper wall time grows more slowly than receiver re-prefill.Source: §4.7 · p. 8, Table 5 · p. 9

5 Broader impact, future work, and limitations

The paper frames cross-model KV transfer as a way to reduce serving costs for agentic workloads with frequent model swaps, and identifies calibration, selection, and architectural scope boundaries. It also outlines methodological and transferability questions for future study.

5 Broader impact, future work, and limitations

  • Skipping receiver prefill can reduce serving cost, energy, and tail latency in agentic workloads with frequent model swaps.The motivating settings include cost-quality cascading, mid-conversation switching, and routing.Source: §5 · p. 9
  • Future work includes nonlinear mapper comparisons, pre-fit transferability signals, cross-family transfer, hidden factors, and hybrid architectures.The authors note that training-data overlap and fine-tuning recipe may influence transferability, and leave systematic study of these factors for future work.Source: §5 · p. 9
  • The evaluation uses FineWeb-Edu calibration, selects k using reported log-likelihood benchmarks, and tests only matched-KV pairs within dense full-attention models.The authors do not test mismatched-KV pairs; hybrid attention and attention-recurrent architectures are out of scope.Source: §5 · p. 9

6 Conclusion

Within-family cross-model KV transfer admits a closed-form, training-free fit, with strong retention on four matched-KV pairs and faster prefill than re-prefill. The conclusion also points to attention-sensitive error placement as a useful evaluation lens and reports gains from a nonlinear mapper on harder pairs.

6 Conclusion

  • 73–98% of standalone accuracy is retained on the four highest-retention matched-KV pairs, with 2.7–25× lower prefill latency than re-prefill.The mapper remains stable across multi-turn handoff.Source: §6 · p. 10
  • Attention-output cosine predicts cross-pair retention better than R2, with r=+0.57 versus r=−0.20 across 12 matched-KV pair evaluations.The conclusion attributes retention to where residual error lands relative to attention-sensitive subspaces, rather than its magnitude.Source: §6 · p. 10
  • A nonlinear MLP adds up to +37 pp HellaSwag retention on harder pairs by redistributing error toward attention-irrelevant directions..Source: §6 · p. 10
  • The conclusion argues that cross-model mapper evaluation should move from raw reconstruction metrics toward subspace-aware diagnostics and attention-aligned objectives..Source: §6 · p. 10

A Cross-model KV cache transfer methods

The appendix compares cross-model KV transfer methods across four criteria: gradient-free fitting, cross-scale operation, KV-value transfer, and closed-form mapping. It reports that the proposed method alone meets all four criteria.

A Cross-model KV cache transfer methods

B Linear structure analysis

The appendix uses greedy forward selection to measure how many source layers contribute to target-layer predictability. Its curves show that multiple layers contribute complementary information, while gains approach the all-layer value by six selected layers.

B Linear structure analysis

  • The appendix presents the curves as empirical evidence for the linear-structure claim and as a measure of how many source layers the mapper needs.Pairwise R2 heatmaps are in Figure 2 of §2.3, while production-ridge R2 values for k=1, 8, and all are in Appendix C.Source: B Linear structure analysis · p. 13
  • Greedy forward selection adds the source layer with the largest joint R2 increase at each step, and the curves climb steeply through k=4 before nearing their k=all values by k=6.The appendix distinguishes this analysis from the production mapper’s fixed top-k selection.Source: B Linear structure analysis · pp. 13–14, Figure 5 · p. 13
  • For Qwen3 14B→32B, k=6 reaches 92.3% of the k=all R2 value for Kstripped and 87.7% for V.Figure 5 shows curves for Kstripped and V across Qwen3 8B→32B and 14B→32B, with nine representative target layers per curve.Source: Figure 5 · p. 13
  • On Qwen3 14B→32B, averaging across target layers, k=1 reaches R2=0.56 for Kstripped and 0.32 for V, while k=8 reaches 0.79 and 0.65.Complementary information across source layers motivates top-k cross-layer selection; the production mapper uses fixed top-k by single-source head-averaged R2 for tractability, rather than greedy joint selection.Source: B Linear structure analysis · p. 14

C Extended ablation

The extended ablations show that using multiple predictive source layers is important, while the gains saturate and vary across pairs. RoPE handling and calibration choices also affect results, with calibration domain showing a distinct cost.

Source layer count

  • Increasing k raises fit R2 with diminishing returns: the largest K R2 gain is from k=1 to k=8, with smaller gains through k=all.The source-layer-count table reports K and V in-sample R2 for Qwen3 14B→32B.Source: C Extended ablation · p. 14, Table 7 · p. 14
  • Across pairs, k=1 is insufficient; Qwen3 14B→32B reaches within 0.3 pp of peak HellaSwag by k=8, while Llama 3.1 8B→70B improves until k=24.Closer pairs saturate earlier; every pair drops sharply on at least one benchmark.Source: Figure 6 · p. 14, C Extended ablation · p. 14

Sequential component removal

  • Removing cross-layer selection costs most on ARC-C and HellaSwag, while removing inference-time RoPE handling costs most on WinoGrande, MMLU and GSM8K.At the 1,024-token fit context, fitting and inferring on RoPE-coupled keys stays within noise of the decoupled pipeline; decoupling is preferred for generalization across RoPE configurations and longer contexts.Source: C Extended ablation · p. 14, Figure 7 · p. 15

Calibration protocol sensitivity

  • Across four orders of magnitude in λ and N=50–1000 calibration sequences, results have wide flat regions; N=50 remains within ∼1.6 pp of production and performance flattens after N=200.Collapse appears only at extremes, including λ=1, which lowers HellaSwag by 15.79 pp.Source: C Extended ablation · p. 15, Table 8 · p. 15

Calibration domain

  • On seven log-likelihood benchmarks, retention stays above 90% in every calibration-domain cell; Wikipedia averages 99.4%, while CodeAlpaca averages 95.9% and varies by 11.1 pp.CodeAlpaca matches FineWeb-Edu on MMLU and BoolQ but costs the most on coreference and hard science QA.Source: C Extended ablation · pp. 15–16, Table 9 · p. 16

D Experimental setup details

The setup specifies matched-KV transfer pairs, benchmark protocols and coverage, and the calibration and run configuration used to evaluate the mappers.

D Experimental setup details

  • The experimental-setup appendix collects architectural axes for the studied transfer pairs, all of which are matched-KV.The table lists the transfer pairs and their architectural characteristics.Source: Table 10 · p. 16, D Experimental setup details · p. 16
  • Evaluation covers five accuracy benchmarks, WikiText-2 perplexity and CoQA for multi-turn handoff; MMLU uses 5-shot prompts, GSM8K uses 8-shot chain-of-thought, and the other accuracy benchmarks are zero-shot.All five accuracy benchmarks are evaluated on all six pairs at their selected k, while transfer perplexity is reported only for Qwen3 and Llama 3.1.Source: D Experimental setup details · p. 16, Table 11 · p. 16
  • Ministral 3 transfer perplexity is omitted because its reasoning-tuned standalone WikiText-2 perplexity is orders of magnitude above the other families, making it an unsuitable comparison.The setup reports missing entries as “—” in the per-family tables.Source: D Experimental setup details · p. 16
  • WikiText-2 perplexity uses non-overlapping 2,048-token chunks, scoring the second 1,024 tokens from a 1,024-token prefix cache to isolate prefix-KV quality.Standalone and transfer conditions score identical continuation tokens.Source: D Experimental setup details · p. 16
  • Mappers need not remain GPU-resident: their size does not grow with sequence length or cache size, and host-to-device paging is estimated at ∼80–480 ms once per active pair.The estimate assumes a 25–50 GB/s PCIe Gen4/Gen5 link and is calculated from size and bandwidth, not measured; directional mappers yield up to P(P−1) ordered pairs for P models.Source: D Experimental setup details · p. 17

E Run information

  • Calibration uses 500 FineWeb-Edu sequences of 1,024 tokens with ridge λ=0.01; experiments use one 8×H100 node and bfloat16 forward computation.The appendix also specifies float32 covariance and float64 analysis, and reports roughly one hour per pair for end-to-end mapper fitting.Source: E Run information · p. 17

F Extended main results

The appendix provides full per-family standalone and transfer accuracies and examines how floor normalization changes benchmark-level retention comparisons.

Per-family results

  • The per-family results table lists target standalone accuracy followed by transfer accuracy from each source, while Figure 8 visualizes retention on ARC-C, HellaSwag and MMLU.The appendix also provides per-benchmark floor-normalized retention for the six matched-KV pairs.Source: F Extended main results · p. 17, Table 13 · p. 18, Table 15 · p. 18

Floor-normalized retention

  • 48.5% WinoGrande accuracy on the ablated Qwen3 8B→32B mapper corresponds to 69.2% retention but −7.7% floor-normalized retention against a 50% chance floor.The target’s own ground-truth cache is the 100% anchor.Source: Table 14 · p. 18, F Extended main results · p. 18
  • 87.1% to 58.5% is the WinoGrande retention change for Llama 3.1 8B→70B after floor normalization; GSM8K is unchanged because its floor is already approximately zero.On the strongest pair, floor normalization makes GSM8K outrank WinoGrande.Source: F Extended main results · p. 18, Table 15 · p. 18

G Prefill latency details

The appendix reports latency comparisons and details how source-layer count k is selected and evaluated. The mapper is faster in the measured cells, while selection uses a defined benchmark set and is checked through re-selection and held-out evaluation.

G Prefill latency details

  • The mapper is faster than target re-prefill in all 70 measured cells across seven pairs and ten sequence lengths, with re-prefill above the mapper at every sequence length shown.Measurements use an 8×H100 bf16 setup; the latency figure plots mapper application and target re-prefill across sequence length.Source: Table 16 · p. 19, G Prefill latency details · p. 19, Figure 9 · p. 20
  • Latency comparisons use synthetic inputs, exclude shipping the mapped cache to the target process, and time Ministral 3 re-prefill without its vision tower.The mapper is measured in eager mode without torch.compile or CUDA graphs.Source: G Prefill latency details · pp. 19–20

H Source-layer count selection

  • For each pair, k is selected by maximizing mean log-likelihood accuracy over ARC-Challenge, HellaSwag and MMLU; GSM8K, CoQA and latency are excluded from selection.WinoGrande is absent at most k values, and Ministral 3B→8B uses k=all rather than its k=20 argmax, which it trails by 0.38 pp.Source: H Source-layer count selection · p. 20
  • Leave-one-benchmark-out re-selection changes k in 6 of 18 folds, and the held-out benchmark’s accuracy moves by at most 2.49 pp, with a 0.30 pp mean decrease when it moves.The largest shift is 2.49 pp on Ministral 8B→14B; across four Tier 1 pairs it is at most 1.45 pp.Source: H Source-layer count selection · pp. 20–21, Table 17 · p. 21
  • On three benchmarks never used for selection, at each pair’s selected k, retention was at least in-sample for every pair; four Tier 1 pairs stayed above 96% and two Tier 2 pairs below 64%.Retention is against the target’s standalone accuracy on the same benchmark. PIQA and ARC-Easy are easier than the selection benchmarks, so absolute levels are not comparable across the two sets.Source: Table 18 · p. 21, H Source-layer count selection · p. 21
Loading 2608.03893v1…