DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving

1 Introduction

• Cross-model prefix reuse addresses a gap in distributed inference: existing systems share KV caches for repeated prefixes on the same LLM, not across different models.Compound AI settings commonly involve multiple LLMs processing shared context, including agent handoffs, model updates, and personalized assistants.Source: §1 · p. 1
• The study asks whether one LLM can reuse another’s context KV cache to accelerate prefill when the models have different weights but the same architecture.Naively sharing across models can greatly reduce generation quality, motivating selective recomputation and reuse.Source: §1 · pp. 1–2
• Across eight representative model pairs, often around 10% of layers are sensitive to cross-model KV-cache differences, and those layers are largely consistent across inputs for a given pair.The identities of these critical layers vary between model pairs.Source: §1 · p. 2
• DroidSpeak profiles critical layers offline, recomputes them while reusing other layers’ KV cache, and pipelines cache loading with recomputation to hide remote loading delay.The profiling selects layers for recomputation on a held-out training set, aiming to preserve accuracy while reusing as much KV cache as possible.Source: §1 · p. 2
• Unlike prior selective-recomputation work for a single LLM, DroidSpeak targets cross-LLM reuse and recomputes or reuses groups of layers rather than selected tokens.CacheBlend updates reused KV cache but feeds it to the same LLM that generated it.Source: §1 · p. 2
• Latency falls by up to 3.1× and throughput rises by up to 4×, with negligible quality drop across six datasets, three tasks, and eight model pairs.Quality is measured using F1, Rouge-L, or code similarity score; the comparison is against baselines that do not allow sharing across models.Source: Figure 1 · p. 1, §1 · p. 2
• DroidSpeak’s primary contribution is making KV-cache translation practical in distributed systems by efficiently transferring and computing the translated cache.The implementation is integrated with LMCache and vLLM and has been tested in enterprise settings.Source: §1 · p. 2

2.1 Basic Transformer Concepts

• Each transformer layer produces E/Q/K/V tensors; the paper calls the K and V tensors together the KV cache and the embedding tensors the E cache.Prefill processes the input context to produce embeddings and KV caches; decode uses the cache to generate output tokens autoregressively.Source: §2.1 · pp. 2–3

2.2 Context Sharing Across LLMs

• Compound AI workloads share prefixes across agent workflows, personalized assistants, and multi-LLM or multi-LoRA serving, where models reuse conversation histories or other contexts.The paper calls the model producing a context's KV cache the sender and the model reusing it, with limited recomputation, the receiver.Source: §2.2 · p. 3

2.3 Distributed LLM Inference Systems

• Distributed inference systems transfer KV caches for shared prefixes when requests use the same model, but do not optimize cache sharing across different models.The systems described include vLLM Production Stack, NVIDIA Dynamo, and ByteDance AIBrix.Source: §2.3 · p. 4

2.4 Prefill interference

• With a 3-second latency SLO, increasing synthetic input length from 5K to 20K tokens reduced goodput by as much as 32×.The Figure 5 setup uses Llama-3-8B on one A100 GPU.Source: Figure 5 · p. 3, §2.4 · p. 4

3 Reusing KV cache across LLMs

• Reusing another model’s KV cache for a 40K-token input reduced prefill latency from 4s to 0.08 seconds in the stated A100 and Llama-3.1-8B-Instruct example.The section frames this reuse as a motivation and asks how direct reuse affects generation quality.Source: §3 · p. 4

3.1 Building the benchmarks and datasets

• The benchmark used eight model pairs across six datasets, requiring shared foundation models and a receiver fine-tuned for the dataset task that outperformed the sender.The study focuses on the challenging case where a less accurate sender produces intermediate states reused by the more accurate receiver.Source: §3.1 · p. 4

3.2 Empirical insights of KV cache

• On HotpotQA, direct full-cache reuse lost more than 50% of accuracy points across most model pairs, although it remained better than the sender model alone.Figure 6 compares receiver quality, receiver quality with the sender's KV cache, and sender quality alone.Source: §3.2 · pp. 4–5, Figure 6 · p. 5
• Across model pairs, an average of 11% of layers were critical, meaning KV-cache deviations in those layers significantly reduced F1 score.The study defines critical layers as the small subset sensitive to deviation; their identities vary by model pair.Source: §3.2 · pp. 5–6
• For the studied HotpotQA model pair, input-to-input F1 variation was notable for critical layers but insignificant for non-critical layers.Layer 23, identified as most critical for that pair, showed a wider spread, with many inputs seeing an F1-score change over 50%.Source: §3.2 · p. 6

4.1 Challenges with Selective KV Cache Reuse

• Recomputing contiguous layer groups reduces output error relative to recomputing only critical layers, while transition-layer E-cache differences can propagate through later layers.At each reuse-to-recompute transition, the sender’s E cache seeds recomputation; multiple transitions can introduce multiple deviations.Source: §4.1 · pp. 6–7

4.2 Profiling for re-computation configuration

• Per model pair, offline profiling on example inputs selects a recomputation configuration that balances generation quality against the number of recomputed layers.The resulting Pareto frontier records the highest F1 score attainable for each layer group; profiling has O(l^2) overhead, and two-layer grouping reduces profiling time by about 3×.Source: §4.2 · p. 7

4.3 DroidSpeak runtime design

• DroidSpeak uses the profile to select recomputation based on the latency SLO and pipelines remote KV-cache loading with recomputation.In the example, pipelining reduces total TTFT to 17, versus 30 when reusable KV is loaded before recomputation and 47 when all caches are loaded first.Source: §4.3 · pp. 7–8

4.4 Implementation

• The vLLM and LMCache implementation fetches reused KV and transition-layer E caches from remote GPU nodes while overlapping transmission with recomputation.It places cache transfers on a CUDA stream separate from the default computation stream.Source: §4.4 · p. 8

5 Evaluation

• Across three datasets and eight model pairs, DroidSpeak reduces prefill latency 1.7–3.1× without compromising accuracy, with average prefill speedup of 2.1× and up to 4× throughput improvement.The evaluation reports robustness of recomputation-layer profiling across datasets and model types.Source: §5 · p. 8

5.1 Experiment Setup

• Evaluation covers eight model pairs, six datasets, and two nodes with eight A100 GPUs each, using 50 HotpotQA contexts for offline profiling.Quality metrics include F1 for QA, Rouge-L for summarization, and code similarity for code completion.Source: §5.1 · pp. 8–9

5.2 Lower Latency with Preserved Accuracy

• Across eight model pairs and three datasets, DroidSpeak reduces prefill delay 1.7–3.1× versus full prefill without compromising quality, and has 5–33% higher quality than CacheBlend at similar latency.Against full KV reuse, it preserves the receiver model’s improved quality at slightly higher delay.Source: Figure 14 · p. 9, §5.2 · p. 10

5.3 Throughput and Latency Improvement

• Under the stated SLO, DroidSpeak supports 2–4× higher throughput than full prefill; reduced TTFT also brings down TBT and end-to-end latency.The figure reports TTFT, TBT, and E2E across request arrival rates.Source: Figure 15 · p. 10, §5.3 · p. 11

5.4 Robustness across datasets

• Training-dataset profiles closely match profiles obtained directly from test data, with score differences of at most 4 points and 2 points on average.This supports applying an offline profile to other benchmark datasets.Source: §5.4 · p. 11, Figure 17 · p. 11

5.5 Case study of other tasks and other models

• On HumanEval problems sent at various rates, DroidSpeak improves TTFT by 2.7× while maintaining pass@1 of 52.5; on Mixtral-8x7B/Mixtral-8x7B-Instruct, it reduces prefill latency.For GSM8K on MAmmoTH2, its Pareto frontier follows a pattern very similar to LongBench.Source: §5.5 · p. 11

5.6 Comparison against a smaller model

• Llama-3.1-8B achieves about 4× lower prefill delay than Llama-3.1-70B-Instruct but about half its original F1 score; DroidSpeak adjusts recomputation as resources change.Unlike switching model sizes, this adjustment avoids loading a different model onto the GPU.Source: §5.6 · pp. 11–12

5.7 Impact of different network bandwidth

• DroidSpeak outperforms non-pipelined baselines under almost all tested bandwidth conditions, though its absolute TTFT reduction is smaller at high bandwidth.The reported comparison is between pipelined recomputation and cache loading and baselines without pipelining.Source: §5.7 · p. 12, Figure 20 · p. 12

6 Limitation

• DroidSpeak as-is does not support sharing KV cache across LLMs from different foundation models, whose caches have different sizes.Source: §6 · p. 12
• Recomputation adaptation considers system load, not changing network bandwidth; extending it for bandwidth changes is left to future work.The proposed example is to expand the range of critical layer groups when bandwidth is limited.Source: §6 · p. 12
• Significant drift between test data and profiling data may degrade quality; periodic re-profiling is proposed to update the configuration.The enhancement is left to future work.Source: §6 · p. 12

7 Related Work

• Parameter-efficient fine-tuning methods such as LoRA and LISA reduce the memory and computation needed for fine-tuning.The paper notes fine-tuned LLM agents in question answering, tool learning, and personalization.Source: §7 · p. 12
• DroidSpeak targets communication delays in multi-agent systems and is described as faster than LoRA-serving work through eliminating prefill computation [40] [47].Other serving work addresses scheduling [56], LoRA memory management, and KV-cache offloading [52].Source: §7 · p. 12
• Single-model KV-cache compression, offloading, and non-prefix blending are orthogonal to DroidSpeak’s cross-model sharing and can be used in conjunction [52].Source: §7 · p. 12

• DroidSpeak targets repetitive computation in systems where multiple models work on a shared context.Source: §8 · pp. 12–13
• DroidSpeak is a framework for KV cache sharing in compound AI systems.Source: §8 · p. 13
• The framework recomputes only a subset of layers to maintain quality and is robust across diverse model pairs, model types, and datasets.Source: §8 · p. 13

• Table 2 reports the sizes of the datasets and their context lengths.Source: A Detail statistics of the datasets · p. 18

• DroidSpeak reduces SLO violation rate compared with DroidSpeak without adaptation and full prefill for the ultrachat-8B and fingpt pair.Source: Figure 21 · p. 18
• When requests are queued, DroidSpeak selects the fewest recomputed layers above an accuracy target; without a queue, it selects the highest recomputation ratio that satisfies each request's latency SLO.It estimates prefill latency from request token counts using parameters obtained during profiling.Source: B Dynamic Re-computation Configuration… · p. 18