Source-linked AI summary
LatentRemedy: Efficient Remedy For Non-Prefix KV Cache Reuse in Multi-Agent Systems
Zhiwei Chen, Zhongtao Rao, Waikit Xiu, Yuxiang Zhang, Shuting Cui, Jian Chen, Tianxiang Zhao
TL;DR
Conflicting tasks can make non-prefix KV-cache reuse across agents perform poorly, and existing repairs recover only a limited fraction of the loss. LatentRemedy adds target-conditioned latent tokens to reused context and trains them through full-prefill distillation. It consistently outperforms other reuse baselines on conflicting tasks while keeping TTFT close to direct reuse as the reused block grows.
Problem
Non-prefix KV-cache reuse in multi-agent systems is underexplored for conflicting objectives, where representations computed under another task can interfere with the target agent’s interpretation of shared context.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The method provides a way to guide generation toward the target task while reusing the shared block without recomputing it.
Source: §1 · p. 2Evaluation covers conflicting-task cache reuse with Qwen3 models at three scales; generalization to other model families and latency configurations remains unexplored.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Non-prefix KV cache reuse is a promising approach for improving inference efficiency in multi-agent systems as agents share the same context blocks. However, when agents have conflicting tasks, directly reusing non-prefix KV caches across agents can severely degrade model performance, causing up to a 90% drop. Existing remedies, including selective recomputation and offset-based calibration, recover a limited fraction of this loss. We argue that the problem lies not only in contaminated KV representations but also in the agent's difficulty interpreting reused context under its own instructions. To this end, we introduce LatentRemedy, which uses a few task-conditioned latent tokens to help the target agent leverage reused context without recomputing the shared block. Specifically, we use an MLP to map the last hidden state of the target prefix into a small number of latent tokens, which are placed after the reused KV block to interact with the shared context. To guide subsequent predictions toward the target task, we train the MLP through full-prefill distillation. Experiments show that LatentRemedy consistently outperforms existing reuse baselines on conflicting tasks while maintaining TTFT close to direct reuse as cache size grows. Code and data are available at https://github.com/chenzhwsysu57/kvreuse.
1 Introduction
Non-prefix KV-cache reuse can save prefill computation when agents share context, but conflicting tasks leave the target agent interpreting context through representations computed for another task. LatentRemedy uses target-conditioned latent tokens to interact with reused context, improving reuse performance while keeping TTFT close to direct reuse.
1 Introduction
- Conflicting-agent reuse is underexplored despite its potential to reduce redundant prefill computation when agents share context blocks.The target agent may have to use a shared block whose cache was computed under a different instruction.Source: §1 · p. 1
- Existing selective-recomputation and offset-calibration remedies recover only a limited fraction of performance loss, while reused representations can interfere with target-task interpretation.The authors suggest an effective remedy should let the target instruction interact with reused context and guide subsequent generation.Source: §1 · p. 1
- LatentRemedy maps the target prefix’s last hidden state into a few latent tokens placed after the reused block, then trains the mapping through full-prefill distillation.The tokens can interact with the prefix and reused context without recomputing the shared block.Source: §1 · pp. 1–2
- LatentRemedy substantially outperforms existing reuse methods on conflicting tasks while maintaining TTFT close to direct reuse.The paper identifies conflicting-task cache reuse, target-specific contextualization disruption, and the latent remedy as its central contributions.Source: §1 · p. 2
2 Related Work
Prior non-prefix KV-cache repair work mainly addresses cooperative settings, while soft-prompt tuning learns fixed task-level prompts. LatentRemedy instead generates request-specific latent tokens from the target prefix.
2 Related Work
- CacheBlend, KVCOMM, RelayCaching, and EPIC repair cached states through selective recomputation or offset recalibration, mainly for cooperative scenarios (Geng et al., 2026b).Non-cooperative scenarios have rarely been explored.Source: §2 · p. 2
- Soft-prompt tuning optimizes continuous embeddings prepended to input as an efficient form of full-model fine-tuning.Source: §2 · p. 2
- Unlike fixed task-level prompts for general adaptation, LatentRemedy generates request-specific latent tokens from the target prefix to guide task-specific generation.Source: §2 · p. 2
3 Preliminary Study
The preliminary study tests cache reuse across agents with conflicting objectives and finds that existing repairs often leave large accuracy losses. Its failure analysis points to task confusion, while repeating the target instruction helps mitigate it.
3.1 How do existing KV cache reuse remedies perform under conflicting agents?
- The study tests ArgKP, Deal or No Deal, and Job Interview by reusing agent A’s shared-block cache for agent B and comparing Full, direct reuse, and four repair methods.The agents have different or conflicting task interpretations of the shared context.Source: §3.1 · p. 2
Without reasoning, conflicting objectives make
- Without reasoning, repairs recover at most 13.50% of lost performance; best-repair accuracy drops are 72.75% on Deal or No Deal and 86.18% on Job Interview.On ArgKP, Full reaches 85.91% accuracy, direct reuse 2.27%, and the best repairs, CacheBlend and EPIC, 6.82%.Source: Figure 2 · p. 2, Without reasoning, conflicting… · pp. 2–3
3.1 How do existing KV cache reuse remedies perform under conflicting agents?
- With reasoning, direct reuse reaches 79.09% on Deal or No Deal versus Full’s 100%, while CacheBlend and EPIC reach 92.73% and 89.09%.Reasoning narrows but does not eliminate the gap between Full and reuse-based methods.Source: Figure 2 · p. 2, Without reasoning, conflicting… · p. 3
3.2 What causes the performance degradation under conflicting objectives?
- On Deal or No Deal, reuse-method reasoning often mixes source-agent utility and shares; this occurred in 50% of EPIC traces.Other observed failures included using source-agent shares or omitting shares.Source: Figure 3 · p. 3, §3.2 · p. 3
- Repeating the target instruction improves all six tested settings over direct reuse, reaching 99.09% on reasoning-enabled Deal or No Deal; a generic restatement does not consistently help.Inspection found no remaining source–target task mixture in the corrected Deal or No Deal traces.Source: §3.2 · p. 3
4 The Remedy Prompt Tuning
LatentRemedy adds task-conditioned latent tokens after a reused KV block so the target agent can interact with shared context without recomputing it. Full-prefill distillation trains the latent mapping to guide predictions toward the target task.
Problem Definition
- The remedy seeks to guide target-specific generation while reusing another agent’s shared-context KV cache.The formulation distinguishes full-prefill generation from generation using another agent’s cache, then introduces a learned latent remedy.Source: §4 · p. 3
Latent Tokens as Remedy
- A lightweight MLP maps the target prefix’s last hidden state into latent tokens placed after the reused KV block, leaving that cache unchanged.Causal attention lets the latent tokens incorporate information from both the target prefix and reused context; subsequent tokens can attend to them.Source: §4 · p. 4
Training Objective
- Full-prefill distillation trains the latent mapping to align remedy next-token predictions with full-prefill predictions and steer generation toward the target task.The objective minimizes KL divergence between the full-prefill and remedy prediction distributions.Source: §4 · p. 4
5 Experiments
Experiments evaluate LatentRemedy on curated and existing conflicting-task benchmarks, comparing accuracy and time-to-first-token (TTFT) with reuse and full-prefill baselines. LatentRemedy generally leads reuse-based methods in accuracy while keeping TTFT near direct reuse, with results varying by task and reasoning setting.
5.1 Setup
- ConflictInstruct contains up to 240 curated conflicting scenes, split 75:25 for training and evaluation; ArgKP, Deal or No Deal, and Job Interview test generalization.Experiments use Qwen3-1.7B, 4B, and 8B, with accuracy and TTFT measured against full prefill, direct reuse, prompt repetition, and cache-repair baselines.Source: §5.1 · pp. 4–5
5.2 Results
- 66.58% vs. 44.91% accuracy: Latent-16 remains the strongest reuse-based method across most settings on ConflictInstruct, including Qwen3-8B with reasoning.Across model sizes, the performance gap over the strongest reuse-based competitor widens; Latent-16 also generalizes to Job Interview, where its advantage reaches 53.64 percentage points.Source: Table 2 · p. 5, §5.2 · p. 5
- Reasoning narrows LatentRemedy’s advantage on Task Switch and Boolean Logic, but competing methods still struggle on Label Mapping; Priority Check approaches perfect accuracy when reasoning is enabled.The results indicate that task-specific guidance and reasoning play complementary roles, with reasoning important for tasks requiring more complex inference.Source: §5.2 · pp. 5–6
- Across tasks and model scales, LatentRemedy delivers higher accuracy at TTFT close to direct reuse, while existing repair methods often add latency without comparable accuracy gains.Figure 5 compares the TTFT–accuracy trade-off across tasks, model scales, and reasoning settings.Source: Figure 5 · p. 6, §5.2 · p. 6
- The latent adapter adds about 0.2 ms across KV lengths, compared with approximately 18–20 ms of recomputation overhead for EPIC-16.The decomposition reports target-prefix, method-specific, KV relocation and splicing, suffix-forward, and first-token sampling costs; measurements are means over 10 runs on one RTX 5090.Source: §5.2 · p. 6, Table 4 · p. 7
- As shared KV length grows, LatentRemedy’s TTFT increases modestly and tracks direct reuse more closely than KVCOMM, CacheBlend, or RelayCaching.Figure 6 compares TTFT across shared-context lengths and three model scales; EPIC-16 maintains a consistent overhead.Source: §5.2 · p. 6, Figure 6 · p. 7
- In a Deal or No Deal case, direct reuse confuses the target agent with the source agent, while Latent-16 restores the target identity and selects the correct allocation.The trace describes Agent A applying its own utility values and correctly selecting E, where direct reuse produces the incorrect answer C.Source: Figure 8 · p. 8, §5.2 · p. 8
- 43.61% vs. 16.00% accuracy: with no reasoning, Latent-8 outperforms direct reuse in the latent-token ablation; three training pairs yield 42.36% without reasoning and 61.61% with reasoning.For the training-pair comparison, direct reuse reaches 16.00% and 50.94%, respectively; even one training pair improves on direct reuse.Source: §5.2 · p. 7, Table 5 · p. 8, Table 6 · p. 8
6 Conclusion
The conclusion studies conflicting-task non-prefix KV reuse and introduces LatentRemedy, which remaps prefix hidden states into latent tokens; experiments find it consistently outperforms baselines while staying close to direct-reuse TTFT as shared blocks grow.
6 Conclusion
- The method remaps prefix hidden states into latent tokens to counteract task interference, and the paper introduces ConflictInstruct to test conflicting-task reuse.The latent remedy is intended to serve conflicting tasks without recomputing the shared block.Source: §6 · p. 8
- The paper identifies conflicting-task non-prefix KV reuse in multi-agent systems as an underexplored scenario.It also examines why existing methods struggle to recover performance under conflicting objectives.Source: §6 · p. 8
- LatentRemedy is consistently better than other baselines on conflicting-task reuse while remaining close to direct-reuse TTFT as the reused block grows.The conclusion describes the method as effective across conflicting-task scenarios.Source: §6 · pp. 8–9
Limitations
The evaluation and latency results cover specific models, tasks, hardware, and shared-block lengths, leaving broader generality and configurations unresolved.
Limitations
- Evaluation focuses on conflicting-task cache reuse with Qwen3 models at three scales, so generality to other model families remains unexplored.The authors also leave open whether training on a large corpus can serve the same goal as training on conflicting tasks.Source: Limitations · p. 9
- Whether training the latent adapter on a large corpus instead of conflicting tasks can serve the same goal remains unanswered.Source: Limitations · p. 9
- Latency is measured using a single RTX 5090, leaving results across other hardware configurations unexplored.Source: Limitations · p. 9
- Latency measurements cover shared blocks of up to 4,096 tokens, leaving longer blocks unexplored.Source: Limitations · p. 9
A Additional Performance Results
Additional results show Latent-16 performs strongly across tasks and model scales, including settings where direct reuse degrades severely; the TTFT decomposition reports the components measured for reuse-based methods.
A Additional Performance Results
- Across model scales and tasks, Latent-16 performs strongly, particularly when direct KV reuse suffers severe accuracy degradation.The appendix adds ArgKP, Deal or No Deal, and Job Interview results for Qwen3-1.7B and Qwen3-8B alongside the main-text Qwen3-4B results.Source: A Additional Performance Results · p. 10
- 74.28% on Job Interview with reasoning exceeds direct reuse at 46.82% and full prefill at 50.45% for Qwen3-1.7B.Latent-16 ranks second on ArgKP and Deal or No Deal and first on Job Interview in the reasoning settings.Source: A Additional Performance Results · p. 10
- 76.44 percentage points on ArgKP and 63.91 points on Job Interview are Latent-16's gains over Reuse without reasoning at Qwen3-8B.The method ranks among the top two in five of six settings; its results are also close to full prefill across the three no-reasoning tasks.Source: A Additional Performance Results · p. 10
B TTFT Decompositions
- The TTFT decomposition reports target-prefix and suffix forward passes, method-specific computation, KV relocation and splicing, and first-token sampling for reuse-based methods.Full prefill is reported by total TTFT because it processes the complete prompt in one forward pass.Source: B TTFT Decompositions · p. 10
Across both model scales, our method consistently achieves total TTFT closest to direct
LatentRemedy stays close to direct reuse in total TTFT as shared KV length grows, with nearly constant adaptation costs and slightly increased suffix latency from added latent tokens.
Across both model scales
- 1.30 ms and 0.40 ms are the total-TTFT gaps from direct reuse for Qwen3-1.7B at 256 and 4,096 shared tokens; the Qwen3-8B gaps are 3.76 ms and 4.22 ms.The full-prefill versus reuse-based execution gap grows rapidly with shared KV length.Source: Across both model scales, our method… · p. 10
- 0.17–0.19 ms on Qwen3-1.7B and approximately 0.25–0.26 ms on Qwen3-8B are the latent adapter's nearly KV-length-constant overheads.EPIC-16 costs roughly 13–14 ms on Qwen3-1.7B and 22 ms on Qwen3-8B, while KVCOMM, RelayCaching, and CacheBlend overheads rise substantially with shared KV length.Source: Across both model scales, our method… · p. 10
- The suffix forward pass is the main remaining overhead, because added latent tokens make suffix latency slightly higher than direct reuse, especially on larger models.This remains small compared with the method-specific computation required by competing reuse-based approaches.Source: Across both model scales, our method… · p. 10
C TTFT-accuracy tradeoff in other tasks
The appendix provides additional TTFT-accuracy figures for other tasks and further reasoning-trace cases across methods and tasks.
C TTFT-accuracy tradeoff in other tasks
- The section presents TTFT-accuracy figures for more tasks in Figures 9, 10, 11, 12, 13, and 14.Source: C TTFT-accuracy tradeoff in other tasks · p. 11
D More Cases
- Figures 15–22 add reasoning-trace cases across methods in Argument Key Point and Deal or no deal.Source: D More Cases · p. 11
E Dataset Overview
ConflictInstruct benchmarks multi-agent KV-cache reuse when agents share context but pursue different goals. It contains paired task instances across diverse task families, with family-disjoint development and test splits.
E Dataset Overview
- ConflictInstruct evaluates shared-context KV-cache reuse when agents receive different goals and must produce different outputs.The benchmark targets multi-agent reuse beyond cooperative tasks such as MMLU, HumanEval, and RAG discussion.Source: E Dataset Overview · p. 11
- The corpus contains 240 task families with 30 paired instances each, totaling 7,200 pairs.Tasks were generated from manually created seed tasks expanded with LLM-proposed operators, scenarios, and task forms; full-prefill evaluation filtered formatting and answer errors.Source: E Dataset Overview · p. 11
- Mean instruction and shared-block lengths are 88.3 and 260.6 tokens, respectively, across varied algorithmic, reasoning, role-conflict, and scenario-based tasks.Instruction lengths range from 19–254 tokens and shared-block lengths from 5–2,200 tokens.Source: E Dataset Overview · p. 12, Table 11 · p. 13
- The family-level split assigns 180 families to train75 and 60 unseen families to test25, yielding 5,400 training and 1,800 test pairs.The split uses fixed random seed 20261005; optional real-world OOD evaluations include ArgKP, Deal or No Deal, and Job Interview.Source: E Dataset Overview · p. 12, Table 12 · p. 13