PaperlayerView exact revision on arXiv

Source-linked AI summary

A Universal Context-Reuse Layer for Cross-Model KV Sharing

Yi Li, Dongming Jiang, Yi Zhao, Bingzhe Li

arXiv:2608.30963v1cs.LGcs.AI

TL;DR

LLM serving systems repeatedly prefill shared context because conventional KV caches are model-specific. The paper translates source-model KV states into target-compatible representations across model sizes, architectures, tokenizers, and families. Experiments provide initial evidence of useful state transfer with reduced target-side prefill and latency.

  • Problem

    Existing KV-cache reuse generally assumes identical producer and consumer models, leaving repeated prefill across heterogeneous models unresolved.

  • Method

    The approach translates a source model’s KV representation into a target-compatible state for subsequent inference in a different model.

  • Results

    Translated KV states preserve meaningful downstream utility while reducing redundant target-side prefill across evaluated within-family and cross-family configurations.

  • Takeaways & Limitations

    KV states can serve as transferable computational representations rather than strictly model-local caches, motivating context mobility for heterogeneous LLM workflows.

  • Takeaways & Limitations

    Runtime integration, synchronization, and other system overheads remain important optimization targets, and translated states need not exactly reproduce native target caches.

Abstract

from arXiv · show

Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own prefill computation even when another model has already processed the same input. Existing KV-cache reuse mechanisms substantially reduce redundant computation within a single model, but generally assume that the producer and consumer of a cache are identical. We study \emph{cross-model KV sharing}, which translates the KV state produced by a source model into a representation that can be consumed by a different target model, including models that differ in scale, architecture, attention configuration, tokenizer, and model family. We evaluate the approach in both within-family and cross-family settings. For Qwen2.5-7B $\rightarrow$ Qwen2.5-1.5B, translated KV states improve LongBench2 accuracy from 27.59\% to 34.48\%, a gain of 6.89 percentage points over the native 1.5B baseline, while reducing handoff cost relative to native target prefill. For the cross-family Qwen2.5-1.5B $\rightarrow$ Gemma-2-2B setting, KV handoff reduces target-side prefill cost by up to 67.05\% at 4K context length while maintaining decoding perplexity close to native-model baselines. In a more heterogeneous Llama3.1-70B $\rightarrow$ Qwen2.5-7B setting, cross-family handoff achieves 44.0\% accuracy compared with 45.7\% for native Qwen2.5-7B inference, while reducing measured latency from 899ms to 138ms. These results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate \emph{context mobility} as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.

1 Introduction

LLM applications increasingly combine heterogeneous models over shared context, but existing KV reuse is largely confined to identical models. Cross-model KV sharing translates previously computed state across model boundaries to reduce repeated contextual computation.

  • Heterogeneous LLM applications route, cascade, or coordinate multiple models over shared task state, conversation history, or retrieved information.
  • Repeated processing occurs because each newly invoked model independently transforms the same context into model-specific internal representations.
  • Long-context prefill can impose substantial GPU computation, memory traffic, and latency before the first output token.
  • Existing prefix-caching systems reduce redundant prefill when the same model encounters an already processed prefix.
  • KV states are generally unusable across models because hidden representations, attention configurations, layers, tokenizers, and architectural parameters differ.
  • Cross-model KV sharing translates one model’s KV representation into a form consumable by a different model, including different-sized or different-family models.

2 Background and Related Work

KV caching converts prefill computation into reusable runtime state, while prior work expands reuse through context composition, transport, architectural variants, and within-family translation. Cross-family translation extends this trajectory toward heterogeneous model representations.

  • KV Caching and Context Reuse: During prefill, an LLM processes the complete input sequence and generates layer-wise KV states that later decoding can reuse.
  • KV Caching and Context Reuse: For long contexts, constructing KV state requires substantial matrix computation, attention processing, memory movement, and possibly inter-device communication.
  • Existing KV-Reuse Directions: Prefix caching, paged KV management, offloading, distributed storage, and cache-aware scheduling optimize KV reuse, placement, capacity, or retrieval.
  • Cross-Context Reuse: CacheBlend and KVCOMM broaden reuse beyond exact-prefix matching by combining cached chunks or correcting context-induced KV offsets.
  • KV Placement and Transport: SwiftCache moves KV states across hardware resources, but transport alone leaves representations tied to their generating model.
  • Cross-Model Reuse: DroidSpeak reuses KV states across fine-tuned variants with the same architecture, whereas prior linear mappings primarily target related models within one family.
  • Heterogeneous Translation: MoT and this work formulate KV reuse across heterogeneous architectures as representation translation rather than direct cache reuse.

3 The Redundant Context Computation Problem

Heterogeneous AI workflows repeatedly process shared context because each model constructs its own internal representation. The resulting redundant prefill grows with context length and the number of participating models.

  • Shared Context: Shared workflow context can include system prompts, user requests, conversation history, retrieved documents, tool outputs, and intermediate agent state.
  • Conventional Inference: Under conventional inference, each participating model independently performs prefill over the common context.
  • Redundant Computation: Although separate representations are necessary for individual models, the application repeatedly processes substantially the same information.
  • Redundant Computation: Redundant prefill becomes more severe as context length or the number of participating models increases.
  • Workload Examples: Model escalation, multi-agent processing, routing, and verification all share logical information while using separate underlying computational representations.

4 Cross-Model KV Sharing

Cross-model KV sharing translates a source model’s cached representation into a target-compatible state, allowing target inference to reuse contextual computation instead of repeating full prefill. Its value depends on translation cost, runtime overhead, and output-quality requirements.

  • Core Mechanism: Cross-model sharing reuses contextual computation by translating a source model’s KV state into a representation consumable by a different target model.
  • Translation Challenge: The translation must adapt across differing representation spaces, attention structures, layer organizations, hidden dimensions, or tokenizers while preserving useful context.
  • Design Objective: A translated state need not exactly reproduce the target model’s native KV cache if it supports acceptable quality while avoiding meaningful target-side prefill.
  • Scope: The approach generalizes KV reuse across differences in model size, configuration, architecture, tokenizer, and model family, with increasing differences making translation more challenging.
  • Efficiency Condition: Cross-model reuse is advantageous when translation and overhead cost less than native target prefill and the translated state meets application quality requirements.
  • Context Mobility: Cross-model KV sharing reframes KV states as reusable computational assets that can be stored, transferred, translated, and consumed across models.

5 Understanding the Translation Layer

The translation layer converts source KV states into target-suitable representations, with objectives ranging from representation matching to downstream behavioral performance. It supports flexible translation granularity while enabling evaluation of within-family KV sharing.

  • The translation layer produces representations suitable for a target model from observed source KV states.
  • The general translation objective is parameterized by θ, which represents the translation mechanism’s parameters.
  • One learning objective minimizes the difference between translated and natively generated target states using a representation-distance function D(·, ·).
  • Table 1 reports LongBench2 accuracy for within-family Qwen2.5 KV sharing.
  • Downstream outputs, attention behavior, logits, or task accuracy may provide more task-oriented objectives than raw KV distance.
  • Translation can operate across layers, KV heads, shared intermediate representations, compressed states, or low-dimensional latent adapters.

6 Experimental Evaluation

The evaluation tests cross-model KV sharing across within-family and cross-family transfers, including differing model scales, architectures, tokenizers, and capacities. Across these settings, handoff improves or nearly preserves accuracy, reduces prefill latency, and remains usable during decoding, while some runtime overhead and long-context fidelity challenges remain.

  • Experimental Setup: The experiments test one within-family and two cross-family transfer directions spanning comparable-scale and large-to-small model pairs.The handoff protocol uses a source-prefilled cache, a learned transport module, and frozen target-model decoding without target-side full prefill.
  • Within-Family Results: 6.89 percentage points: translated Qwen2.5-7B KV states improve Qwen2.5-1.5B LongBench2 accuracy from 27.59% to 34.48%.The handoff remains below 45.69% native Qwen2.5-7B accuracy but recovers 38.1% of the native 1.5B-to-7B accuracy gap.
  • Within-Family Results: 8.16 percentage points: within-family handoff accuracy falls from 40.74% at 8K–16K contexts to 32.58% at 16K–32K contexts.The translated state remains informative through the 32K-token bucket, although representation fidelity becomes harder to preserve as transported context grows.
  • Within-Family Results: 5.4×: handoff reduces incremental latency versus Qwen2.5-1.5B target prefill for 16K–32K contexts, with 53.8 ms versus 288.3 ms.For 8K–16K contexts, handoff is 34.5 ms versus 158.7 ms, a 4.6× reduction; relative to 7B prefill, reductions reach 16.7× and 14.5×.
  • Cross-Family Results: 67.04%: Qwen2.5-1.5B → Gemma-2-2B handoff reduces incremental latency at 4K tokens, taking 59.897 ms versus 181.706 ms for native Gemma prefill.Transferred-state perplexity remains comparable to native baselines across decoding horizons, although differences vary by horizon.
  • Cross-Family Results: 44.0% accuracy and 138 ms latency: Llama3.1-70B → Qwen2.5-7B handoff trails native Qwen accuracy by 1.7 percentage points while reducing latency from 899 ms.The handoff retains 96.3% of native target accuracy and is 6.5× faster than target self-inference.

7 Conclusion

The work provides initial evidence that cross-model KV sharing can transfer contextual state across heterogeneous models while preserving downstream utility and reducing redundant target-side prefill. It frames this capability as context mobility for inference across models and stages.

  • Cross-model KV sharing translates a source model’s contextual representation into a form usable by a different target model.
  • Experiments provide initial evidence that state transfer is feasible within model families and across distinct families with differences in scale, architecture, and internal representation.
  • Translated KV states preserve meaningful downstream utility while reducing redundant target-side prefill across the evaluated configurations.
  • The findings support viewing KV caches as transferable computational state rather than only model-local execution artifacts.
  • The results motivate context mobility, where previously computed context retains value as execution moves across models and inference stages.
Loading 2608.30963v1…