PaperlayerView exact revision on arXiv

Source-linked AI summary

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel

arXiv:2610.11358v1cs.AIcs.LG

TL;DR

Cross-model KV-cache reuse is limited because model-specific caches force a full target prefill, while linear-map accuracy degrades across large capacity gaps. RaReCache uses rank disagreement to select tokens for target-side recomputation; with small recompute budgets, it recovers near-native accuracy and reduces prefill latency.

  • Problem

    Cross-model cache reuse is limited because KV caches are tied to model architecture, and closed-form linear-map accuracy degrades as model-capacity gaps widen.

    Source: §1 · p. 1

  • Method

    RaReCache uses rank disagreement to identify critical mapped-cache tokens for selective target-side recomputation, without target forward passes for selection.

    Source: §1 · p. 2

  • Results

    Across two model families, 30% recomputation retains 95–99% of target accuracy on a 23× gap; prefill is up to 3.04× faster, with median and p99 TTFT reduced by 5.0× and 6.4×.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RaReCache decouples transfer quality from source-model capacity and delivers prefill speedups in the reported settings.

    Source: §5 · p. 10

    The fitted linear map requires calibration data that closely matches the downstream evaluation distribution, limiting fully zero-shot use without generalized maps or broadly representative calibration data.

    Source: D LIMITATIONS AND FUTURE WORK · p. 16

Abstract

from arXiv · show

Cross-model KV-cache reuse remains a key challenge in modern LLM serving. Coding agents and multi-model systems increasingly route a shared context across models: a user may switch models mid-session, or a cascade may escalate a difficult query. Because KV caches contain model-specific representations, each switch typically forces the receiving model to prefill the entire context from scratch. Recent work shows that closed-form linear maps can translate KV caches between models in the same family, but transfer accuracy degrades as the model-size gap widens. In this paper, we establish that these transfer failures are concentrated in a small subset of information-dense tokens. To bridge this gap, we introduce RaReCache, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation. RaReCache identifies these critical positions using a novel rank disagreement metric, scoring each token by the energy of its mapped KV in output directions weakly supported by the calibration data. Across two model families and five benchmarks, on a 23x parameter gap (Qwen3-0.6B to 14B) recomputing just 30% of positions retains 95-99% of the target accuracy, whereas on a 8.8x gap (Llama3-8B to 70B), recomputing 40% retains 96.5% of the target accuracy. RaReCache largely removes sensitivity to source-model size, and achieves up to a 3.04x prefill speedup. For online serving, it handles 1.8x the request throughput of target prefill on a single GPU, and at the target's saturation load, reduces median and 99th-percentile time-to-first-token (TTFT) by 5.0x and 6.4x respectively, with a 30% recompute budget. RaReCache establishes an efficient serving paradigm where small models prefill on behalf of massive targets, enabling large models to recompute only critical tokens, drastically reducing prefill latency.

1 INTRODUCTION

Model hand-offs make cross-model KV-cache reuse valuable, but mismatched representations force a full target prefill. RaReCache selectively recomputes critical tokens to bridge transfer-accuracy gaps while reducing prefill cost.

1 INTRODUCTION

  • Model switches force the receiving model to prefill the full context because a KV cache is tied to its model’s architecture.The paper situates these hand-offs in cascades, dynamic routing, and coding agents that upgrade models mid-session.Source: §1 · p. 1
  • Existing reuse methods are largely limited to identical architectures, while cross-model linear-map accuracy degrades as model-capacity gaps widen.Other cross-model approaches described here focus on improving accuracy rather than saving prefill compute.Source: §1 · p. 1
  • Rank disagreement selects critical tokens by measuring mapped-KV energy outside the calibration subspace, without target-model forward passes.It outperforms attention and raw mapping-error heuristics in closing the accuracy gap.Source: §1 · p. 2
  • On a 23× gap, recomputing 30% of positions retains 95–99% of target native accuracy on GSM8K and ARC; on an 8.8× gap, a 40% budget retains 96.5% on GSM8K.The results cover Qwen3-0.6B→14B and Llama 3 8B→70B, respectively.Source: §1 · p. 2
  • At a 30% recompute budget, RaReCache reduces median TTFT by 5.0× and p99 TTFT by 6.4× under live serving, and reaches up to 3.04× faster prefill.The evaluation uses dynamic request batching.Source: §1 · p. 2

2 MOTIVATIONAL CASE STUDY

The case study finds that linear mapping alone leaves a substantial accuracy gap that grows as the source model shrinks. Attention and raw mapping error also fail to reliably select tokens whose recomputation improves transfer.

2.2 CAN ATTENTION IDENTIFY WHICH TOKENS TO RECOMPUTE?

  • Figure 3 compares top-20% token selections and per-token scores across template, question, instruction, and header regions for different selectors.The plotted scores include relative mapping error and source and target attention; random selection has no associated score.Source: Figure 3 · p. 3, §2.2 · p. 3

2.1 THE CAPACITY BOTTLENECK OF LINEAR MAPPING

  • The mapped fraction of target key variance barely changes with source size even as accuracy drops, and mapping more source layers to one target layer yields negligible gains while increasing parameters.The authors hypothesize that the shortfall reflects the source model’s representational capacity, so the target must recompute some positions.Source: §2.1 · p. 3
  • 71.6% GSM8K accuracy results from mapping Qwen3-0.6B KV to Qwen3-14B, versus 95.1% for the 14B model.This is a 23× parameter gap, and the remaining accuracy gap grows as the source model shrinks.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 CAN ATTENTION IDENTIFY WHICH TOKENS TO RECOMPUTE?

  • On the illustrated GSM8K prompt, attention concentrates on sink and template tokens; at ρ=0.2 it selects no question tokens, and attention selectors score below random.The 14B assigns 72% of attention to the <|im start|> sink, 24% to the rest of the template and instruction, and 4% to the question.Source: §2.2 · p. 4
  • The ℓ2 mapping-error selector also does no better than random, so neither attention nor raw error reliably identifies tokens whose recomputation improves downstream accuracy.The case study compares selectors on Qwen3-0.6B→14B with 30% of prompt positions recomputed.Source: Figure 4 · p. 3, §2.2 · p. 4

3 METHODOLOGY

RaReCache maps source-model KV caches into target space, then selectively recomputes prompt positions whose mapped representations depart from calibration-supported directions. A closed-form linear map supplies cache predictions, while a recompute budget limits target-side work to selected positions.

3.1 MAPPING THE CACHE

  • For each target layer and key or value tensor, the map uses concatenated entries from the top-K source layers selected by single-source regression R².Keys are de-rotated before fitting and re-rotated after mapping, allowing transfer across context lengths.Source: §3.1 · p. 4
  • A centered ridge-regression fit estimates the target cache entry in closed form, with an L2 penalty regularizing the near-singular source-feature matrix.The calibration pass accumulates XᵀX and XᵀY for a single linear solve, without gradient descent.Source: §3.1 · p. 4

3.2 RARECACHE: SELECTIVE RECOMPUTATION VIA RANK DISAGREEMENT

  • Given recompute budget ρ, RaReCache selects the top ⌈ρT⌉ positions using one shared set across target layers.The selected positions are recomputed by the target while the mapped cache is used elsewhere.Source: §3.2 · p. 4, Figure 5 · p. 5

3 METHODOLOGY

  • The pipeline combines linearly mapped cache entries with target-recomputed entries at the selected prompt positions.The source model prefills the prompt; the target runs forward passes only for positions selected by rank disagreement.Source: §3 · p. 4, Figure 5 · p. 5

3.2 RARECACHE: SELECTIVE RECOMPUTATION VIA RANK DISAGREEMENT

  • Rank disagreement scores each token by the squared gap between full-rank and reduced-rank predictions, summed across mapped layers and cache tensors.The reduced-rank map retains the leading calibration-supported directions; positions with the largest scores are recomputed.Source: §3.2 · pp. 4–5
  • Calibration-familiar text such as templates scores near zero, whereas item-specific entities, quantities, and relations score high.The score targets representations in directions rarely explored by the calibration set, rather than selecting by raw error magnitude.Source: §3.2 · p. 5
  • The calibration-derived basis is computed once per map, and inference scoring uses one projection per layer and cache side.The method reuses de-rotated source-layer blocks across target layers to limit mapping and selection overhead.Source: §3.2 · p. 5

4 EXPERIMENTS

Across Qwen3 and Llama 3, the experiments test accuracy recovery, selector quality, reverse transfer, and serving efficiency. Selective recomputation recovers much of the large target’s accuracy while reducing prefill cost, with results varying by transfer direction and load.

4.1 SETUP

  • Evaluation covers Qwen3 and Llama 3 model pairs and five benchmarks, with calibration disjoint from evaluation and recompute budgets from 0 to 0.5.The fixed rank is r=128; calibration uses 128k–350k tokens from each task’s training subset.Source: §4.1 · p. 6

4.2 Q1: HOW MUCH OF THE TARGET MODEL’S ACCURACY CAN RARECACHE RECOVER?

  • At a 23× gap, recomputing 30% of positions recovers 94.8% of Qwen3-14B accuracy on GSM8K, 96.9% on ARC-Challenge, and 98.6% on ARC-Easy.At ρ=0.5, GSM8K transfer reaches 95.3% versus 95.1% for the 14B alone.Source: §4.2 · p. 6, Figure 6 · p. 6
  • Plain-transfer accuracy depends strongly on source size, but recomputation narrows the source spread, including LongBench-E from 23.6 to 1.9 F1 at ρ=0.5.At ρ=0, the 0.6B source retains 60.3% of target accuracy on MMLU-Redux and 75.3% on GSM8K; LongBench-E transfer is 8.2 F1 below the 0.6B alone.Source: §4.2 · pp. 6–7
  • At an 8.8× Llama 3 gap, 40% recomputation reaches 92% accuracy, or 96.5% of the 70B target, about 10 pp above the 8B alone.The 8B→70B accuracies use the first 300 GSM8K test items.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Q2: HOW DOES RANK DISAGREEMENT COMPARE TO STANDARD TOKEN SELECTION HEURISTICS?

  • At a matched 30% budget, rank disagreement closes 79% of the 14B accuracy gap, versus 42% for random selection, and selects content words rather than template tokens.The 14B-attention and largest-mapping-error selectors are analysis-only oracles; question content words receive under 2% of 14B attention.Source: §4.3 · p. 7, Figure 7 · p. 7

4.4 DOES RARECACHE GENERALIZE TO THE LARGE-TO-SMALL TRANSFER?

  • In large-to-small transfer, the 14B mapped cache improves MMLU-Redux by 14.4 pp for the 1.7B and 3.1 pp for the 4B, but loses 7.8 pp at the 23× gap.Recomputation changes reverse-transfer accuracy by at most 0.7 pp at ρ=0.2; the 14B cache is reconstructed well, with R² of 0.90–0.91.Source: §4.4 · pp. 7–8

4.5 COMPARISON WITH EXISTING REUSE STRATEGIES

  • At ρ=0.3 on GSM8K, RaReCache reaches 90.3% accuracy, versus 80.7% for the ℓ2-error oracle and 79.0% for layerwise recomputation.The ℓ2-error oracle performs no better than random selection; layer-prefix recomputation scales poorly because errors occur across depths.Source: §4.5 · p. 8, Figure 8 · p. 8

4.6 SYSTEM ANALYSIS: END-TO-END INFERENCE PERFORMANCE

  • At T=2048, recomputing 50%, 30%, and 20% yields 1.38×, 1.95×, and 2.31× speedups; batching raises peak speedup to 3.04× at ρ=0.2.At ρ=0.3, recomputation takes 38%–63% of full-prefill latency because target-weight loading and attention over all prompt keys remain necessary.Source: §4.6 · pp. 8–9, Figure 9 · p. 9

5 CONCLUSION

RaReCache uses rank disagreement to identify information-dense tokens for selective recomputation, enabling large targets to reuse caches from smaller sources with near-native reasoning accuracy. The authors report substantial prefill gains and identify custom sparse causal kernels as future work.

5 CONCLUSION

  • RaReCache targets cross-model KV-transfer accuracy loss by selectively recomputing a minority of information-dense tokens, without target-side prefill.The conclusion says recomputing as little as 30% of the prompt can recover near-native reasoning accuracy.Source: §5 · pp. 9–10
  • The reported efficiency uses standard attention kernels; custom sparse causal kernels are future work for attaining peak theoretical gains.The authors state that they reviewed AI-assisted content and take responsibility for the final text, claims, and artifacts.Source: §5 · p. 10, AI USE STATEMENT · p. 10
  • The authors report that experimental setups, model configurations, and evaluation metrics are detailed, and that source code will be made public after de-anonymization and publication.The listed code includes linear mapping, rank-disagreement scoring, system optimization, and evaluation scripts.Source: REPRODUCIBILITY STATEMENT · p. 10

A RANK DISAGREEMENT DERIVATION

Rank disagreement measures mapped-cache energy outside the output directions represented by calibration data. The rank-r map retains leading directions of fitted outputs and refits its bias to preserve the calibration means.

A RANK DISAGREEMENT DERIVATION

  • The calibration-supported subspace is formed from the leading eigenvectors of the covariance of fitted calibration predictions.The first r eigenvectors define the projection onto the well-described subspace.Source: A RANK DISAGREEMENT DERIVATION · p. 12
  • The rank-r map keeps leading directions of its fitted outputs and refits the bias so the average input maps to the average output.This is reduced-rank regression, not a truncated SVD of W; the two coincide only for isotropic source features.Source: A RANK DISAGREEMENT DERIVATION · p. 12
  • The bias terms cancel in the prediction difference, so the score does not depend on either model's mean cache.The derivation starts from the centered prediction z_t=(x_t−x̄)W.Source: A RANK DISAGREEMENT DERIVATION · p. 12
  • The score is the energy of a centered mapped prediction outside the calibration-supported subspace, summed independently across layers and sides.It can be computed with one d_t × r projection per layer and side.Source: A RANK DISAGREEMENT DERIVATION · pp. 11–12
  • At rank zero the score is the centered prediction's squared norm, while at full rank it is zero for every token.At intermediate ranks, it measures only energy outside directions covered by calibration data.Source: A RANK DISAGREEMENT DERIVATION · p. 12

B.1 MAPPING IMPLEMENTATION AND RUNTIME EFFICIENCY

RaReCache's smaller kernels gain throughput as batching improves GPU occupancy, while BF16 mapping and shared source blocks reduce implementation overhead. At T=2048, these optimizations leave target-side recomputation as the main remaining latency source.

B.1 MAPPING IMPLEMENTATION AND RUNTIME EFFICIENCY

  • BF16 mapping raises relative performance to 0.78–1.38× by using Tensor Cores instead of the FP32 path's vector units.The unoptimized FP32 map makes cache transfer slower than native target prefill, at 0.63–0.85×.Source: §B.1 · p. 12
  • Figure 10 tracks phase runtimes across implementation variants and context lengths, with bar labels reporting overall speedups.BF16 execution plus source-block sharing reduces mapping and selection overhead to a small fraction of total latency.Source: Figure 10 · p. 13
  • Caching 50 de-rotated source blocks once per prompt avoids reconstructing them 640 times and yields 1.16–1.95× overall speedup.The reference implementation redundantly reconstructs those blocks across target layers and KV heads.Source: §B.1 · p. 13
  • At T=2048, mapping latency falls from 195 ms to 27 ms (7.2×) and token selection from 70 ms to 17 ms (4.2×).After these optimizations, target-side recomputation is the main remaining latency source.Source: §B.1 · p. 13

B.2 BATCHING AND PEAK THROUGHPUT

  • 12,378 tokens/s for RaReCache versus 6,256 tokens/s for native target prefill as batching raises RaReCache throughput nearly twofold.Native target prefill plateaus beyond batch size 6, while RaReCache benefits from improved SM occupancy.Source: §B.2 · p. 13

B.3 LIVE SERVING EVALUATION AND QUEUING METHODOLOGY

The live-serving evaluation uses FCFS dynamic batching under Poisson arrivals and measures wall-clock TTFT, including queueing and dispatch overhead. RaReCache sustains greater throughput and lower queue buildup than target prefill under the tested load.

B.3 LIVE SERVING EVALUATION AND QUEUING METHODOLOGY

  • The evaluation uses an FCFS dynamic-batching scheduler that dispatches up to 32 queued requests when the GPU becomes idle.Requests arrive through a homogeneous Poisson process; each sweep covers 500 requests and discards the first 50.Source: §B.3 · p. 13
  • Figure 11 compares throughput scaling at T=149 and speedup across batch sizes for three prompt lengths.Its throughput panel shows target-only prefill saturating near batch size 4 while RaReCache continues to scale.Source: Figure 11 · p. 14
  • TTFT is measured in wall-clock time, capturing both queueing delay and framework-level eager-dispatch overhead.The live-serving comparison uses identical Poisson arrival traces.Source: §B.3 · pp. 13–14
  • At ρ=0.3, RaReCache sustains 77.0 requests/s versus target prefill's 42.7 requests/s, a 1.80× capacity advantage.The load sweep caps at 42 requests/s because the baseline queue diverges at or above its saturation rate.Source: §B.3 · p. 14
  • At 42 requests/s, mean batch size is 24.1 requests for target prefill versus 5.9 for RaReCache, which clears requests faster and limits queue buildup.The baseline approaches the 32-request batch cap, while RaReCache's shorter service time prevents head-of-line blocking.Source: §B.3 · p. 14

B.4 ANALYTICAL SPEEDUP CEILING AND ATTENTION SCALING

The analytical speedup ceiling compares total FLOPs for native target prefill and the RaReCache pipeline. With standard attention kernels, the ceiling falls as context length increases; skipping causally masked query-key pairs could improve long-context speedups.

B.4 ANALYTICAL SPEEDUP CEILING AND ATTENTION SCALING

  • With mapper overhead eliminated, target-side recomputation accounts for 73–79% of total RaReCache latency; recomputed query tokens attend over the entire context T via an explicit attention mask, evaluating all ρT² query-key pairs.Source: §B.4 · p. 14
  • The ceiling compares target-prefill FLOPs with the map, probe, and target-side recomputation costs in RaReCache.The cost expression includes the parameter counts of the target, map, and rank-disagreement selector, as well as recomputation attention.Source: §B.4 · p. 15
  • At ρ=0.3, the theoretical speedup ceiling falls from 2.74× at T=2,048 to 2.28× at T=32,768, approaching 1.67× as context length grows.Quadratic attention FLOPs dominate linear feed-forward terms at long contexts.Source: §B.4 · p. 15
  • A dedicated attention kernel that skips causally masked query-key pairs could raise the asymptotic ceiling to 3.33× at ρ=0.3.The paper benchmarks standard, unspecialized attention kernels and leaves custom sparse causal implementations to future work.Source: §B.4 · p. 15

C.1 MAPPING.

This section specifies the mapping and calibration setup, then compares rank disagreement’s shared recompute set with layerwise selectors and examines which prompt tokens it selects.

C.1 MAPPING.

  • The mapping uses λ=0.01, selects source layers for each target layer by R2, and maps both keys and values.Qwen3 uses k=8, mapping 8192 to 1024 dimensions per token over 40 target layers, for 671M map parameters.Source: §C.1 · p. 15
  • Calibration uses disjoint, similarly formatted task data; stride-subsampling is intended to cover diverse contexts across each corpus.The supplied pools include task training data for GSM8K and ARC, MMLU validation for MMLU-Redux, and LongBench-E data filtered against evaluation items.Source: §C.1 · p. 15
  • Table 5 reports Qwen3-0.6B to Qwen3-14B accuracy or F1 and retention across five benchmarks as a function of the recomputed-position fraction.The caption identifies plain transfer at ρ=0 and says each recompute budget improves over it, but provides no cell values.Source: Table 5 · p. 16
  • 0.86-0.97 layer overlap: rank disagreement’s shared set exceeds the ℓ2 mapping-error oracle at every Qwen3-14B layer except the last.The gap averages 4.8 points in layers 0-25 and 1.7 points in layers 30-38.Source: §C.1 · p. 15, Figure 12 · p. 16
  • Rank disagreement selects mostly question or content tokens, while other selectors spend more of their budget on template tokens that yield negligible accuracy gains when recomputed.Figure 13 compares the top 20% selected by rank disagreement and mapping error across prompt windows.Source: §C.1 · p. 16

D LIMITATIONS AND FUTURE WORK

The section identifies calibration-distribution sensitivity and a shared-tokenizer assumption as current boundaries, and points to task-agnostic calibration and heterogeneous-model transfer as future work.

D LIMITATIONS AND FUTURE WORK

  • The closed-form mapping remains sensitive to calibration-data distribution, so current transfer requires calibration closely matching the downstream evaluation distribution.The paper gives use of the specific benchmark’s training split as an example.Source: D LIMITATIONS AND FUTURE WORK · p. 16
  • Task-agnostic maps or a diverse representative calibration corpus are proposed to enable selective recomputation without prior knowledge of deployment workload.Source: D LIMITATIONS AND FUTURE WORK · p. 16
  • RaReCache assumes source and target share a tokenizer and vocabulary; transfer across heterogeneous model families with differing tokenizers remains open.Source: D LIMITATIONS AND FUTURE WORK · p. 16

Related papers

Loading 2610.11358v1…