Source-linked AI summary
KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, Yiran Chen
TL;DR
Multi-agent systems repeatedly recompute KV-caches for shared contexts, while diverging prefixes prevent direct reuse. KVCOMM addresses this with training-free, prompt-adaptive cache sharing and online offset estimation, demonstrating efficient communication across multi-agent workloads with task-performance trade-offs.
Problem
Multi-agent agents redundantly recompute KV-caches for overlapping conversation contexts, and diverging prefixes limit traditional KV sharing.
Method
KVCOMM is a training-free, prompt-adaptive framework that estimates cache offsets online and reuses shared-context KV-caches through an anchor pool.
Results
Across retrieval, math, and programming workloads, KVCOMM reduces average latency as agent count increases; moderate reuse reaches 98.2% with a 3.3% accuracy cost.
Takeaways & Limitations
KVCOMM provides an effective trade-off between multi-agent prefilling efficiency and system accuracy while adapting to changing interaction patterns.
Takeaways & Limitations
The original TTFT calculation omitted the first token’s decoding latency, which the final version rectifies.
Abstract
from arXiv · showhide
Multi-agent large language model (LLM) systems are increasingly adopted for complex language processing tasks that require communication and coordination among agents. However, these systems often suffer substantial overhead from repeated reprocessing of overlapping contexts across agents. In typical pipelines, once an agent receives a message from its predecessor, the full context-including prior turns-must be reprocessed from scratch, leading to inefficient processing. While key-value (KV) caching is an effective solution for avoiding redundant computation in single-agent settings where prefixes remain unchanged, it cannot be directly reused in multi-agent scenarios due to diverging prefixes introduced by agent-specific context extensions. We identify that the core challenge lies in the offset variance of KV-caches across agents. To address this, we propose KVCOMM, a training-free framework that enables efficient prefilling in multi-agent inference by reusing KV-caches and aligning cache offsets of overlapping contexts under diverse prefix contexts. KVCOMM estimates and adjusts KV-caches for shared content by referencing a pool of cached examples-termed anchors-that store observed cache deviations under varying prefixes. The anchor pool is maintained and updated online, allowing dynamic adaptation to distinct user requests and context structures. KVCOMM achieves over 70% reuse rate across diverse multi-agent workloads, including retrieval-augmented generation, math reasoning, and collaborative coding tasks, all without quality degradation. Particularly, when each fully-connected agent receives 1K input tokens with 512 prefix tokens and 512 output tokens under a five-agent setting, KVCOMM achieves up to 7.8x speedup compared to the standard prefill pipeline, reducing TTFT from ~430 ms to ~55 ms.
1 Introduction
Multi-agent LLMs redundantly recompute overlapping contexts because prefix-dependent KV-cache offsets prevent direct reuse. KVCOMM addresses this with training-free, online anchor-based offset alignment and achieves substantial prefilling speedups across workloads.
- Motivation: Multi-agent systems redundantly recompute KV-caches for overlapping contexts during every prefill, creating a multi-context redundancy bottleneck.Agents encode full conversation histories even when retrieved passages or peer outputs are shared.
- Motivation: Different prefix contexts can induce substantially different KV deviations for the same shared text, limiting fixed KV-cache reuse policies.The paper motivates prompt-adaptive reuse because system roles and upstream-agent output lengths vary across requests.
- KVCOMM: KVCOMM is presented as a training-free, prompt-adaptive framework that requires no additional training, profiling, or model modifications.Its anchor-based cache sharing is designed for dynamic adaptation across distinct requests and agent workloads.
- KVCOMM: KVCOMM reuses shared-context KV-caches by estimating context-aware offsets from similar cached examples and aligning them to new prefixes.The framework treats reuse as approximate translation and maintains an online anchor pool for offset estimation.
- Results: ∼6.7× average prefill speed-up is achieved across retrieval-augmented generation, math reasoning, and programming workloads with Llama-3.1-8B-Instruct on an NVIDIA H100.The same experiments report comparable GSM8K performance with less than 2.5% accuracy drop at a 95% reuse rate across 1,319 samples in a four-agent system.
2 Related Work
Prior KV-cache reuse methods target fixed prefixes, selective recomputation, or other constrained settings, while multi-agent systems repeatedly prefill shared contexts. KVCOMM instead targets adaptive reuse under changing agent prefixes.
- LLM-Based Multi-Agent Systems: Multi-agent workloads repeatedly perform full prefill passes over shared contexts, with repeated computation scaling quadratically as agent graphs grow.This motivates addressing prefilling rather than only improving agent coordination or routing.
- KV-cache Acceleration and Reuse: KV-cache reuse has distinct multi-request, multi-turn, and multi-context patterns, but reuse generally requires compatible cache structure or prefixes.Shared-prefix copying works when only request tails differ, whereas changing agent contexts create a harder reuse setting.
- KV-cache Acceleration and Reuse: KVCOMM differs by reusing all shared-context KV-caches and applying context-aware offsets to align them with changed prefixes.Its overview combines prefix-cache reuse, anchor matching, offset approximation, and direct decoding after cache construction.
- KV-cache Acceleration and Reuse: Existing acceleration paradigms include prompt-level reuse, selective recomputation, cache compression, and kernel-level optimization, each designed for particular scenarios.Prompt-level reuse, for example, is restricted to fixed prompt structures.
3 Proposed Approach
KVCOMM addresses cross-context KV-cache reuse by aligning positional and contextual offsets, using online anchors to approximate reusable caches across agent-specific prefixes.
- KV-cache reuse can avoid the O(N^2d) prefilling cost that dominates long-context inference, but multi-agent prefixes create positional and contextual mismatches.Agents compose prompts from fixed prefixes and runtime placeholders, while RoPE-induced position differences can overwhelm the contextual deviations relevant to reuse.
- The framework maintains online reuse and fallback: agents reuse caches when conditions are met, otherwise perform standard prefilling and update anchor pools.This update process enriches future reuse opportunities without requiring model training.
- KVCOMM first aligns positions by de-rotating stored keys, removing RoPE rotations caused by shifts between token positions.Without alignment, raw key differences may be much larger than the contextual deviation being estimated.
- KV-cache proximity tracks embedding proximity: near token pairs have smaller cache distances under identical prefixes and smaller offset distances under distinct prefixes.The reported experiments find consistently closer caches for near pairs and high correlation between embedding distance and cache or offset proximity.
- KVCOMM uses anchor pools for each placeholder to predict reusable offsets from similar, length-compatible samples and approximate the placeholder’s K/V cache.Embedding-distance weights favor nearby anchors, while length compatibility supports positional alignment.
4 Experiments
KVCOMM is evaluated across multi-agent retrieval, math, and programming workloads against Original and CacheBlend baselines, measuring accuracy, cache reuse, and latency. It maintains competitive task performance while delivering adaptive reuse and substantial prefill speedups across agent counts and context lengths.
- Main results: KVCOMM maintains competitive accuracy on MMLU at 64.7%–69.9%, consistently matching or outperforming CacheBlend while closely tracking the original baseline.The comparison covers multiple agent configurations summarized in Table 1.
- Main results: On GSM8K, KVCOMM declines only 1.9% from 81.5% to 79.6% across two to five agents, remaining within ±2% of the original baseline while CacheBlend falls from 82.0% to 57.1%.This result tests robustness as the number of collaborating agents increases.
- Main results: KVCOMM achieves stable HumanEval Pass@1 scores of 81.4%–83.2%, exceeding CacheBlend by an average margin of 53%.The reported comparison indicates stronger preservation of coding-task performance under cache sharing.
- Reuse and scalability: 70%–87.6% reuse rates are achieved adaptively, while mean speedup scales from 2.24× to 6.72× as prefix and output lengths increase.Reuse declines with more diverse agent contexts, whereas the scalability study varies prefixes from 64 to 1K tokens and outputs from 128 to 1K tokens.
- Latency and scalability: 7.82× speedup is achieved for the fifth agent, reducing subsequent-agent TTFT to 26.9–38.6 ms under 512-token prefixes and outputs.The first agent achieves only 1.11× acceleration because it lacks upstream caches.
- Robustness and ablations: KVCOMM remains robust to request-order changes and reaches up to 98.2% reuse at minimal accuracy cost, with an anchor pool of 20 balancing efficiency and task performance.The request-order study reports consistent or slightly improved accuracy versus baseline; hyperparameter sensitivity is evaluated on GSM8K with four agents.
5 Conclusion
KVCOMM enables efficient KV-cache sharing and context-aware offsetting for collaborative multi-agent LLM systems, achieving substantial prefilling speedups across RAG, math reasoning, and programming tasks.
- KVCOMM achieves approximately 6.7× average prefilling speedup under a three-agent setting on a single H100 GPU.
- The framework combines shared KV-cache reuse with context-aware cache offsetting to improve communication efficiency as the number of agents increases.
- Experiments across retrieval-augmented generation, math reasoning, and programming demonstrate an efficiency–accuracy trade-off for collaborative multi-agent systems.
6 Appendix
The appendix provides supplementary material covering limitations, method details, experimental settings, theoretical analyses, and additional experimental visualizations.
- The appendix includes sections on limitations and broader impacts, method explanation, experimental details, and further experimental analysis.
- Supplementary method material covers a glossary, theoretical proofs, placeholder naming, KV-cache management, and the KVCOMM algorithm.
- Additional experiments analyze benchmark prompts, matching criteria, approximation methods, long-context matching overhead, memory cost, anchor distributions, and offset visualizations.
6.1 Limitations and Broader Impacts
KVCOMM is currently evaluated only for text-input multi-agent systems and accelerates prefilling but not decoding, leaving broader modalities and end-to-end latency for future work.
- KVCOMM is evaluated on LLM-based multi-agent systems that process text inputs, while multimodal inputs remain outside the current evaluation scope.
- KVCOMM accelerates each agent’s prefilling process but does not accelerate decoding latency, another bottleneck in agent collaboration.
6.2 Method Explanation
KVCOMM models cache reuse as an offset-alignment problem: it estimates context-dependent deviations from anchors, updates shared caches, and reuses them when anchor criteria permit.
- KV-cache offsets measure how shared text’s cache under a new prefix differs from its base cache, with Keys requiring RoPE alignment before offsetting.
- The offset variance problem makes static reuse unreliable because identical shared text can have substantially different cache offsets across contexts.
- Anchors store representative placeholder and prefix offsets, which KVCOMM interpolates to approximate offsets for new contexts.
- Placeholders represent user inputs, tool results, and agent responses, using category-specific names to address shared caches consistently across agents.
- The system initializes placeholder-specific anchor pools, computes missing base caches, stores reusable response caches, and adds new anchors when reuse is unavailable.
- For each request, KVCOMM retrieves base caches and matched anchors, computes softmax distance weights, approximates placeholder and prefix caches, and concatenates the updates for decoding.
6.3 More Experimental Details
The experiments use role-specific multi-agent prompt templates across MMLU and GSM8K, with agents exchanging current and previous outputs. Additional templates define specialized roles for knowledge retrieval, critique, mathematical analysis, and final decision-making.
- Additional analysis: The experimental analysis evaluates KV-cache proximity and offset proximity against token-embedding proximity using sampled vocabulary-token pairs.This evaluation tests whether embedding-closest anchors can approximate token KV-cache offsets.
- MMLU: MMLU cycles through specialized multi-agent roles, including Knowledge Expert, Wiki Searcher, Critic, Mathematician, and FinalRefer.FinalRefer selects one answer option using the other agents’ analyses.
- Prompt structure: Agent prompts incorporate the user question, current outputs from other agents, and selected outputs from the previous dialogue round.These templates include role-specific fields such as Wikipedia explanations, prior answers, and internal-testing conditions.
- GSM8K: GSM8K uses Math Solver, Mathematical Analyst, Programming Expert, Inspector, and FinalRefer roles within its prompt templates.The Math Solver is instructed to reason step by step and place only the final result on the last line.
6.4 More Experimental Analysis
Additional experiments show that KVCOMM remains competitive on harder reasoning benchmarks while balancing reuse and accuracy through a combined matching criterion. The appendix also examines memory growth with longer sequences and more anchors.
- Harder reasoning benchmarks: KVCOMM achieves superior or comparable performance to dense computation on MATH500, while reuse declines more rapidly as agent count increases on harder reasoning tasks.On Deepseek-Qwen, KVCOMM achieves both higher reuse and higher accuracy than the comparison baseline.
- Harder reasoning benchmarks: KVCOMM maintains comparable AIME accuracy to dense prefill while keeping reuse above 70%, although decoding length is reduced by its extra memory cost.The reported accuracy drop is attributed to the token-length constraint during decoding.
- Memory analysis: Memory cost increases with longer input and output sequences and with larger anchor counts because anchor-specific KV-cache deviations require additional storage.The deviations are reported to be sparse across anchors, averaging about 50% of elements.
- Matching criterion: Combining length matching with embedding-distance matching preserves performance while enabling effective anchor reuse; length matching alone increases reuse but degrades accuracy.The ablation is conducted on MMLU in a four-agent setting.
6.4.3 Analysis on the Approximation Method
The approximation analysis compares anchor-based offset methods and measures the overhead of matching long-context KV-caches. Cosine similarity performs comparably to the ℓ2 method, while CPU offloading can dominate latency.
- Offset approximation methods: Cosine-similarity matching achieves performance comparable to the ℓ2-norm method with slightly higher reuse rates on HumanEval.The comparison uses a four-agent setting with Qwen-Coder-2.5-7B and an 84.45% baseline accuracy.
- Matching overhead: Softmax matching takes about 18 ms with 25 anchors and 4096 tokens per anchor in simulation.Its latency scales with the number of anchors and sequence length because softmax operates across anchor weights.
- Long-context overhead: CPU offloading raises average softmax latency above 100 ms and total per-agent offloading to above 1260 ms for 4K-token contexts.The analysis identifies data movement for long-context KV as the main overhead and suggests pipelining as an orthogonal mitigation.
- Memory analysis: The memory analysis reports increasing cost with longer input and output sequences and more anchors, reflecting storage for anchor-specific KV deviations.The reported deviations are sparse across anchors, averaging about 50% of elements.
6.4.6 Visualization of Responses Generated by Different Combinations of Alignment Strategies
Response visualizations show that partial alignment can produce a correct answer while damaging the reasoning chain, whereas combining all three alignment processes preserves a coherent explanation path. Anchor analyses further examine activation patterns and offset variance.
- Alignment strategies: The partial-alignment response omitted a formal definition, repeated keywords, and analyzed nonexistent agents, disrupting the link between evidence, inference, and conclusion.The response still produced the correct option “C,” showing that answer correctness did not guarantee coherent reasoning.
- Alignment strategies: Combining all three alignment processes is essential for a coherent explanation path comparable to the original response.Key Rotation plus Prefix KV Offsets alone produced fragmented reasoning despite eventually selecting the correct option.
- Anchor distribution: In the four-agent GSM8K analysis, anchor token length is positively correlated with activation frequency across conversational placeholders.The visualization covers user_question and three agent-response placeholders.
- Offset distributions: Prefix KV offsets are often larger than placeholder offsets, but their variance is relatively smaller, especially in deeper layers.The comparison is based on ten samples in a fully connected four-agent MMLU setting.
6.4.9 Visualization of Distance between Approximated and Real Offsets
The visualization compares KV-cache offset distributions across agents and evaluates approximation quality against real caches. L2NORM maintains high similarity and low offset error, while simpler reuse strategies degrade more substantially in deeper layers.
- The figures track key and value cache offset distributions for placeholder and prefix segments across the first, second, third, and fourth agents.The distributions are measured on ten MMLU samples in a four-agent setting.
- Approximately 0.92 key-cache and 0.95 value-cache cosine similarity is maintained across layers with minimal offset errors under L2NORM approximation.These similarities are comparable to the cosine-based approach.
- Nearest-reusing exceeds 0.003 mean offset error beyond layer 25, whereas plain reuse falls below 0.8 cosine similarity and exceeds 0.004 offset error.The comparison shows stronger degradation for simpler reuse strategies, particularly in deeper transformer layers.