PaperlayerView exact revision on arXiv

Source-linked AI summary

CacheBridge: Efficient Cross-Model KV Cache Transfer

Xingyu Qu, Siyuan Lu, Zhiyu Chen, Sheng Wang, Tao Lin

arXiv:2609.00891v1cs.AI

TL;DR

Cross-model KV transfer avoids repeated receiver-side prefill, but Full-Head Mapping is sensitive to architecture and incurs storage, application, and construction costs. CacheBridge addresses these issues with architecture-indexed head support, attention-aligned calibration, and fused bounded construction while retaining affine online deployment. Across three transfer directions, it recovers degraded Ministral 3 transfers and preserves 99.83% mean target retention on Qwen3, with substantial Qwen3 efficiency gains.

  • Problem

    Model-specific KV caches force the receiving model to prefill shared prefixes, while Full-Head Mapping remains sensitive to architectural differences and scales in cost with layer support.

  • Method

    CacheBridge restricts each target head to an architecture-indexed source-head block, weights fitting by receiver attention sensitivity, and uses fused bounded sufficient-statistics construction while retaining an affine interface.

  • Results

    Across three transfer directions, CacheBridge recovers the two degraded Ministral 3 directions and preserves 99.83% mean target retention on Qwen3 14B →32B.

  • Takeaways & Limitations

    CacheBridge reduces coefficient count by 8× while repairing or preserving transfer quality and lowering Qwen3 mapper storage, application latency, and construction time.

  • Takeaways & Limitations

    The evaluation covers same-family models with dense grouped-query attention and matched KV-head counts, and does not establish generalization to other attention mechanisms or open-ended multi-turn quality.

Abstract

from arXiv · show

Sharing context between LLMs in a multi-model system requires the receiving model to prefill the shared prefix because KV caches are model-specific. Recent closed-form cross-model KV transfer, hereafter Full-Head Mapping, avoids this replay by fitting a training-free affine mapper from source to target caches. However, its full-head design maps each target KV head from every source KV head in the selected layers, making transfer quality sensitive to architectural differences and causing mapper storage and application cost to grow with layer support. To this end, we introduce CacheBridge, which co-designs architecture-indexed mapper support, attention-aligned calibration, and bounded mapper construction while retaining a closed-form affine interface for online deployment. CacheBridge restricts each target head to a matched source head, weights reconstruction errors by causal attention sensitivity, and uses a fused GPU kernel to construct weighted sufficient statistics without materializing full observation tensors. Across three transfer directions, CacheBridge recovers the two Ministral 3 transfer directions where Full-Head Mapping loses substantial accuracy while preserving 99.83\% mean target retention on Qwen3. On Qwen3 $14\mathrm{B}\to32\mathrm{B}$, it reduces mapper storage by $8\times$, accelerates application by up to $3.0\times$, matches \fullhead with one tenth of the calibration data, and reduces 500-sequence construction from 92.63 to 8.63 seconds ($10.7\times$).

1. Introduction

CacheBridge addresses the model sensitivity, attention mismatch, and efficiency costs of Full-Head Mapping by co-designing mapper support, calibration, and construction while retaining affine online deployment. Across three transfer directions, it recovers degraded Ministral 3 performance, preserves Qwen3 quality, and reduces mapper costs.

  • Motivation: Cross-model KV transfer avoids repeated receiver-side prefill, which otherwise grows with context length in multi-model applications.The receiver cannot directly reuse the sender’s model-specific cache and must prefill the shared prefix before decoding resumes.
  • Prior approach: Full-Head Mapping fits a training-free affine mapper offline and applies it online to materialize a decode-ready target cache.Its full-head design maps every target KV head from every source KV head in selected layers.
  • Limitations: Full-Head Mapping is model-sensitive: under the same 500-sequence protocol, it stays near Qwen3 target accuracy but falls to 52.2% and 44.4% on two Ministral 3 directions.This contrast indicates that the shared KV interface can hide architecture-dependent head correspondence.
  • Limitations: Full-Head Mapping also incurs layer-dependent application cost and fragmented construction because generic top-k fitting repeatedly materializes scattered, centered, and weighted tensors.On Ministral 3 3B →14B, latency approaches target re-prefill at k=16 and exceeds it from k=20 onward.
  • CacheBridge: CacheBridge co-designs architecture-indexed support, attention-aligned calibration, and bounded construction to address these coupled failure modes while keeping online deployment affine.HEAD-LOCAL constrains head fan-in, ATTN-REPAIR weights residuals by receiver attention sensitivity, and FUSED-FIT constructs weighted sufficient statistics in bounded panels.
  • Results: 20.4 and 31.6 HellaSwag points are recovered on the two Ministral 3 directions, while Qwen3 14B →32B retains 99.83% mean target retention.On Qwen3, mapper storage falls from 4.296 to 0.538 GB, application is up to 3.0× faster, and 500-sequence construction takes a median of 8.63 seconds.

2. Background and Motivation

Cross-model KV transfer avoids target-side prefill but faces architecture-sensitive head correspondence, receiver-sensitive errors, and support-dependent construction costs. CacheBridge addresses these coupled issues by co-designing mapper support, calibration, and construction while retaining affine deployment.

  • Cross-Model Prefix-State Transfer: Cross-model transfer requires a separate directional mapper because source and target models use different internal representations.The receiver otherwise must prefill the shared prefix before decoding resumes.
  • Cross-Model Prefix-State Transfer: Full-Head Mapping predicts each target head from every source KV head across selected layers, yielding feature width kHsds per target head.It removes source RoPE before regression and restores target RoPE after mapping.
  • Cross-Model Prefix-State Transfer: Architecture differences can undermine full-head transfer despite a shared KV interface, because model families expose different residual-stream scaling paths and head-ownership structures.The contrast is consistent with scale-specific cross-head correlations not surviving residual-stream changes, but does not isolate a single cause.
  • Layerwise Error Propagation in Cache Transfer: Full-head support raises feature dimension from pℓ = kds to pf = kHsds, retaining weakly identified cross-head directions that increase propagated estimation error.This motivates restricting support to control errors injected at each target layer.
  • Layerwise Error Propagation in Cache Transfer: Receiver-composed attention and downstream computation determine which cache errors matter, whereas ordinary KV regression and R2 use an identity metric.Mapped-layer errors are transformed by downstream receiver computation and can accumulate across layers.
  • Construction and Deployment Costs: Mapper support couples statistical flexibility to deployed affine cost, while varying selected source sets create irregular offline layouts that can dominate construction through repeated tensor materialization.Static support can instead be compiled into the construction path using sufficient statistics rather than full intermediate observations.

3. Our Approach: CACHEBRIDGE

CACHEBRIDGE co-designs mapper support, attention-aligned calibration, and bounded construction while preserving affine online deployment. HEAD-LOCAL narrows support, ATTN-REPAIR weights errors by receiver attention sensitivity, and FUSED-FIT constructs sufficient statistics without full observation materialization.

  • CACHEBRIDGE keeps online deployment affine while changing mapper support, calibration weights, and offline sufficient-statistics construction.The deployed artifact remains a collection of affine maps, and the added calibration logic introduces no extra inference stages.
  • 3.2. HEAD-LOCAL—Head-Level Structural Locality: HEAD-LOCAL replaces all-head support with a deterministic architecture-indexed source-head assignment for each target head.Aligned KV groups use a(h) = h; other group-count relationships use a fixed map derived from architecture metadata.
  • 3.2. HEAD-LOCAL—Head-Level Structural Locality: 8× fewer coefficients reduce mapper storage and leading affine multiply–add work at fixed prefix length.The reduction factor equals the number of source KV heads and is independent of target head count and selected-layer count.
  • 3.3. ATTN-REPAIR—Attention-Aligned Calibration: ATTN-REPAIR derives calibration weights from causal prefix-boundary sensitivities of the receiver’s future queries.The weights use squared Jacobian-block traces aggregated over query heads sharing a KV head, while discarding off-diagonal blocks and forming a local first-order surrogate.
  • 3.3. ATTN-REPAIR—Attention-Aligned Calibration: ATTN-REPAIR shrinks concentrated raw weights toward uniform weights and enforces an effective-sample-size floor without changing support or the online interface.The same centered ridge solver is retained; only coefficient values differ.
  • 3.4. FUSED-FIT: FUSED-FIT uses a two-pass fused GPU kernel and bounded chunks to gather, center, weight, and write contiguous per-head panels for batched sufficient-statistics accumulation.This avoids repeated materialization of irregularly gathered, centered, and weighted tensors while preserving the solver and mapper artifact.

4. Evaluation

Across three transfer directions, CacheBridge evaluates receiver-aligned support, attention-weighted calibration, and fused construction against matched Full-Head Mapping protocols. It recovers both Ministral 3 failures, preserves Qwen3 retention, and reduces mapper cost and construction time.

  • Structural attribution: 8× fewer coefficients reduce the local input width without changing selected source layers.HEAD-LOCAL uses one architecture-indexed source-head block per target head.
  • Calibration efficiency: With one tenth of the calibration data, CacheBridge reaches 99.89% mean target retention versus 99.44% for Full-Head Mapping with 500 sequences.The frozen Qwen3 budget sweep uses 50 versus 500 calibration sequences.
  • Structural attribution: At matched capacity, every non-identity cyclic assignment performs worse than the aligned assignment, while compactness alone does not explain the operating point.The mean non-identity cyclic assignment lowers primary retention by 15.50 points.
  • Attention attribution: Attention weighting improves primary retention by 1.25 points and reduces NLL@4K from 2.446 to 2.350 despite nearly unchanged K/V R2.Mapper support, coefficient count, and application code remain unchanged in this comparison.
  • FUSED-FIT efficiency: On Qwen3 14B →32B, mapper storage falls from 4.296 to 0.538 GB and application speedups reach 3.0× at 1,024-token prefixes.Application latency falls from 65.12 to 21.66 ms at that prefix length.
  • Scope of measurements: Latency measurements isolate mapper application from source prefill, transport, loading, decoding, and scheduling, while construction timing excludes upstream trace collection and evaluation.End-to-end TTFT and throughput remain outside the reported scope.
  • FUSED-FIT efficiency: FUSED-FIT reduces 500-sequence mapper construction from 92.63 to a median of 8.63 seconds, a 10.7× speedup.The fused path preserves mapper support, weights, the ridge solve, and the serialized artifact.

5. Related Work

CacheBridge extends closed-form affine cache transfer rather than replacing it with learned communication modules. Its distinction is co-designing fixed mapper support, calibration, and construction for efficient online deployment.

  • Closed-form cache transfer: CacheBridge starts from a training-free affine mapper that transfers source to target caches without target-side prefill or task-specific adaptation.The baseline maps every target KV head from every source KV head in selected layers.
  • Learned latent communication: Unlike learned latent-communication approaches, CacheBridge does not train additional communication modules or shared representation spaces.Related methods use neural projection, fusion, adapters, latent channels, or token-gated translators.
  • Reuse, repair, and architectural co-design: Other related methods reuse attention patterns, transmit semantic codes, or selectively recompute layers, whereas CacheBridge targets cross-model KV-value transfer through a closed-form affine interface.These alternatives address reuse or repair under different architectural and training assumptions.

6. Limitations

The evaluation is bounded to same-family dense GQA models with matched KV-head counts and immediate continuation metrics. It does not establish behavior for broader architectures, cross-family transfer, or repeated multi-turn handoffs.

  • Evaluation boundaries: All evaluated directions connect models from the same family, so cross-family transfer may expose weaker head correspondence and larger representation shifts.The paper identifies same-family transfer as the first evaluation boundary.
  • Attention mechanisms: The support rule is unestablished for mismatched head counts and sparse, sliding-window, linear, or hybrid attention.All evaluated models use dense grouped-query attention with matched KV-head counts.
  • Multi-turn quality: The study measures immediate continuation through multiple-choice accuracy and teacher-forced NLL, not open-ended multi-turn quality after transfer-driven decoding.Repeated generation and model handoffs may accumulate errors not captured by these evaluations.

7. Conclusion

CacheBridge co-designs mapper support, attention-aligned calibration, and construction while retaining affine online deployment. Across three directions it improves the quality–cost trade-off of closed-form cache transfer without adding a learned translator.

  • Conclusion: CacheBridge combines HEAD-LOCAL, ATTN-REPAIR, and FUSED-FIT to address model sensitivity, overprovisioning, and construction and application cost.The components constrain head fan-in, align fitting with receiver computation, and handle irregular gathers with bounded intermediate memory.
  • Conclusion: Across all three transfer directions, CacheBridge reduces coefficient count by 8× while repairing or preserving transfer quality.On Qwen3 14B →32B, it also lowers mapper storage and application latency and constructs the 500-sequence mapper in 8.63 seconds.
Loading 2609.00891v1…