PaperlayerView exact revision on arXiv

Source-linked AI summary

KVCMAS: Efficient KV cache Correction for Shared Context in Multi-Agent Systems

Hyesung Jeon, Hyeongju Ha, Seoyoung Lee, Beomseok Kang, Jae-Joon Kim

arXiv:2609.34060v1cs.LG

TL;DR

Agent-specific prefixes make shared-context caches diverge, forcing repeated prefill or approximate reuse. KVCMAS uses compact low-rank online corrections chained along the workflow, and reports strong accuracy and serving performance across language and vision-language workloads.

  • Problem

    Different agent prefixes produce different KV caches for the same shared context, while existing methods face limits in dynamic-context correction, memory use, or reference construction.

    Source: §1 · p. 1, §2.2 · p. 4

  • Method

    KVCMAS stores cross-agent cache deviations in compact low-rank states and chains corrections using caches produced along the agent workflow, without a separate reference prefill.

    Source: §3 · p. 4, §3.2 · p. 5

  • Results

    At 32K shared tokens and 8 QPS, KVCMAS provides a 2.0× TTFT speedup over NonShared; across language and vision-language workloads, it achieves the highest overall accuracy and lowest serving latency among KV cache sharing works at high request rates.

    Source: §5 · p. 9

  • Takeaways & Limitations

    KVCMAS offers an accurate, scalable KV cache sharing approach for prompt-specialized multi-agent serving under high request rates.

    Source: §5 · p. 9

    Full-dimensional online anchor states scale memory with context length, anchor count, and agent count, while non-chained correction requires a separate reference prefill for first-seen context.

    Source: §2.2 · p. 4, §3.2 · p. 5

Abstract

from arXiv · show

Prompt-specialized multi-agent systems enable multiple agents to share a model while performing complementary roles to solve complex tasks. However, agent-specific prefixes change the KV cache generated for the same shared context, causing each agent to repeatedly prefill the growing context and construct a separate cache with high computation and memory overhead. Selective recomputation reduces this redundancy but still retains substantial model execution, while existing delta correction methods either support only recurring context relations or maintain memory-intensive online correction states for dynamically changing context. For first seen shared context, these methods also construct a reference cache outside the agent workflow, and an approximate correction at the first agent affects the outputs passed to subsequent agents. We present KVCMAS, an online KV cache correction framework that represents cross-agent cache deviations using compact low-rank states and seamlessly chains corrections along the agent workflow without an additional reference prefill. This design supports dynamically changing shared context while preserving an exact first-agent cache. Across multiple language and vision-language workloads, KVCMAS matches or improves the accuracy of prior KV cache sharing methods while achieving the lowest TTFT under highly concurrent serving. Under controlled serving traces, it provides a 2.0x TTFT speedup over inference without KV cache sharing and reduces peak GPU memory by up to 3.7x relative to a prior KV cache correction method. These results establish KVCMAS as an accurate and scalable KV cache sharing approach for prompt-specialized multi-agent serving.

1 INTRODUCTION

Prompt-specialized agents share model weights, but different prefixes make shared-context KV caches diverge and repeated prefill costly. KVCMAS uses compact online corrections chained along the workflow, preserving an exact first-agent cache without a separate reference prefill.

1 INTRODUCTION

  • Prompt-specialized agents share model weights while taking complementary roles, but their growing trajectories repeatedly process shared context across turns, agents, and modalities.Source: §1 · p. 1
  • Different agent prefixes change the hidden states and KV caches for the same shared context, so separate prefills and caches are otherwise needed.Direct cache reuse avoids redundancy but degrades accuracy; selective recomputation recovers some accuracy with additional prefill computation.Source: §1 · pp. 1–2
  • Existing correction methods often depend on recurring context relations or retain full-dimensional online states; first-seen contexts also require an outside-workflow reference prefill.An inaccurate first-agent correction can affect outputs passed to subsequent agents.Source: §1 · p. 2
  • KVCMAS stores cross-agent cache deviations in compact low-rank states, densely prefills the first agent, and uses each corrected shared-context cache as the next edge’s reference.This chaining removes the separate context-free reference prefill while preserving an exact first-agent cache.Source: §1 · p. 2, Figure 1 · p. 2
  • Across text and vision workloads, KVCMAS retains competitive multi-agent accuracy while reducing correction memory and improving serving efficiency.Source: §1 · p. 2

2 BACKGROUND

Shared context is reusable across agents, but agent-specific preceding prefixes cause its KV cache to differ. Existing sharing methods address this through selective recomputation or delta correction, each with trade-offs in computation, dynamic-context support, memory, or reference construction.

2.1 CONTEXT SHARING CONVENTION

  • Agent trajectories interleave agent-specific prefixes with shared PH segments containing user queries, task observations, and agent outputs; differing preceding prefixes make identical PH segments yield different caches.Source: §2.1 · p. 3

2.2 KV CACHE SHARING METHODS

  • Selective recomputation replaces a chosen cache subset with target-context entries, but deviations remain outside that subset and expanding coverage requires more model execution.Source: §2.2 · p. 3
  • Delta correction estimates the target cache’s deviation from a context-free reference and applies it without model execution over corrected entries; unreliable corrections fall back to dense prefill.Source: §2.2 · pp. 3–4
  • Figure 2 defines effective rank as the minimum rank retaining 90% of singular-value energy; annotations average across layers and the dotted line marks rank 32.Source: Figure 2 · p. 4
  • GraphFlow and Kamera reuse corrections for recurring context relations, while KVComm matches dynamic contexts with online anchors whose full-dimensional states increase memory with context length, anchor count, and agent count.First-seen contexts also require an outside-workflow reference prefill, and first-agent error can propagate downstream.Source: §2.2 · p. 4

3 METHODOLOGY

KVCMAS stores online anchor representations and corrections in low-rank form, and defines corrections relative to the source-agent cache already produced in the workflow. It densely prefills the first agent, then chains correction across agents.

3.2 CHAINED CORRECTION ALONG THE AGENT WORKFLOW

  • Chaining correction from a densely prefetched first-agent cache reduces aggregate reconstruction error across evaluated layers and agents.Figure 3 compares the reconstructed cache with the exact cache obtained through full prefill.Source: Figure 3 · p. 5, §3.2 · p. 6

3.1 COMPACT ONLINE CORRECTION VIA LOW-RANK ANCHOR POOLS

  • Online correction states occupy a low-dimensional feature space, so KVCMAS stores base caches and agent-specific corrections in low-rank form.Full-dimensional online anchor states can require substantial memory; one stated example requires approximately 50 GiB for a 4K-token pool with 20 anchors and four agents.Source: §3.1 · p. 5

3.2 CHAINED CORRECTION ALONG THE AGENT WORKFLOW

  • Unlike context-free correction, chained correction uses the source-agent cache as its reference, avoiding a separate reference prefill for first-seen shared context.Existing non-chained correction can approximate the first agent's cache, whose error then affects outputs passed to subsequent agents.Source: §3.2 · p. 5

3.3 KVCMAS

  • For each shared slot, anchors store low-rank base and correction factors; similarity-weighted matching estimates transitions, and an entropy threshold determines whether to correct or densely prefill.The reconstructed correction is applied to the source-agent cache; accepted results become the reference for the next workflow edge.Source: §3.3 · p. 6

4 EXPERIMENTS

KVCMAS is evaluated across language and vision-language benchmarks and controlled serving traces. It combines strong accuracy with low latency and memory use, with larger TTFT gains under long-context, high-concurrency serving.

4.1 EXPERIMENTAL SETUP

  • Evaluations cover five language and vision-language workloads, use three task-specific agents followed by reflection, and measure accuracy over three runs alongside controlled four-agent serving traces.Accuracy, latency, TTFT, and peak memory were measured on an A100 80 GB at 1 QPS; concurrent serving traces varied shared context and QPS.Source: §4.1 · pp. 6–7

4.2 BENCHMARK ACCURACY

  • KVCMAS achieves the highest accuracy among KV cache sharing methods on three of five benchmarks and ranks second to KVComm on GSM8K and HumanEval within 0.7% points.It also has the lowest end-to-end latency and TTFT among delta correction methods across workloads, and remains near the upper-left accuracy–latency Pareto frontier.Source: Table 1 · p. 7, §4.2 · pp. 7–8, Figure 4 · p. 8

4.3 SERVING EFFICIENCY

  • At 32K shared tokens and 8 QPS with ρ = 0.8, KVCMAS provides a 2.0× TTFT speedup over NonShared and 27% lower TTFT than GraphFlow.Its median TTFT advantage over other cache-sharing methods grows with shared-context length; the comparison also includes ρ = 0.9 versus 0.8 and ρ = 0.8 versus 0.6 settings.Source: Figure 5 · p. 8, §4.3 · pp. 8–9

4.2 BENCHMARK ACCURACY

  • Selective recomputation partly recovers FullShared’s accuracy loss but requires additional model execution for broader coverage, while GraphFlow’s preconstructed corrections degrade on dynamic requests.KVComm recovers much of that loss through online anchor matching, but its full-dimensional anchor pool uses up to 3.7× more peak GPU memory than KVCMAS.Source: §4.2 · p. 8

4.3 SERVING EFFICIENCY

  • With other procedures fixed, chained correction reduces median TTFT by 34% at 32K shared tokens and 8 QPS versus non-chained correction.The non-chained variant uses a separate context-free reference, whereas chained correction follows the preceding agent cache.Source: Figure 6 · p. 9, §4.3 · p. 9
  • Under single-stream execution, KVCMAS uses substantially less memory than KVComm while keeping peak memory comparable to other methods; r = 32 and V = 10 capture most accuracy gains.It retains the lowest TTFT and highest throughput among cache-sharing works as agent count and interaction rounds increase.Source: §4.3 · p. 9

5 CONCLUSION

KVCMAS uses low-rank states to represent cross-agent cache deviations and chains corrections along the workflow. Across language and vision-language workloads, it reports strong accuracy and serving efficiency, including lower TTFT and memory use.

5 CONCLUSION

  • Across language and vision-language workloads, KVCMAS achieves the highest overall accuracy and lowest serving latency among KV cache sharing works at high request rates.At 32K shared tokens and 8 QPS, it provides a 2.0× TTFT speedup over NonShared.Source: §5 · p. 9
  • KVCMAS stores cross-agent cache deviations in compact low-rank form and chains corrections using caches produced along the workflow, preserving an exact first-agent cache and removing reference prefill.Chained correction reduces TTFT by 34% compared with non-chained correction.Source: §5 · p. 9
  • KVCMAS reduces peak GPU memory by up to 3.7× relative to KVComm and retains the lowest TTFT and highest throughput as agent count and interaction rounds increase.The paper concludes that KVCMAS is an accurate and scalable KV cache sharing approach for multi-agent serving.Source: §5 · p. 9

A ANALYSIS OF KV CACHE REUSE

Selective recomputation often leaves substantial cache-reuse error unreconstructed, while KVCMAS represents recurring PF-induced deviations compactly and manages online corrections through anchor matching and reliability controls.

A.1 RECONSTRUCTION COVERAGE OF SELECTIVE RECOMPUTATION

  • At least half of cache-reuse error remains outside recomputed subsets; on MathVista and Video-MME, deviation and attention selection remove only about 20%, close to random selection.KVCMAS instead corrects the full reused cache without model execution over individual entries, yielding higher accuracy with lower latency in long-context, high-load regimes despite a lower reuse ratio.Source: Table 2 · p. 13, §A.1 · p. 13

A.2 LOW-RANK STRUCTURE OF PF-INDUCED DELTA

  • PF-induced cache deviations have effective rank around 4–6, and rank 8 retains over 90% of singular-value energy across evaluated workloads.PF corrections are profiled once and stored in low-rank form for reuse; unlike dynamically changing PH, recurring PF relations do not require online anchor matching.Source: Figure 7 · p. 14, §A.2 · p. 14

A.3 ANCHOR POOL MANAGEMENT

  • KVCMAS gates corrections by normalized entropy, handles partial anchor coverage with the longest covered prefix, and uses dense prefill when a match is unreliable.Dense prefill yields an exact target cache, resets propagated error, and supplies source caches and observed corrections for new anchors; accepted corrections are not reinserted.Source: §A.3 · p. 15

B.1 EXPERIMENTAL SETUP

The evaluation uses five language and vision-language workloads with task-specific multi-agent roles, applying cache correction across several workflow structures and comparing implementations under shared measurement conventions.

Agent roles

  • MMLU, GSM8K, and HumanEval use distinct three-agent role sequences followed by a FinalRefer reflection agent; MathVista and Video-MME adapt the GSM8K and MMLU role structures.The evaluated framework follows KVComm (Ye et al., 2025).Source: §B.1 · p. 16

Workflow structures

  • KVCMAS applies correction to sequential, loop, fan-in, and fan-out transitions, correcting each incoming fan-in segment separately.Dense fallback produces an exact cache for the current segment and resets propagated cache error for later transitions.Source: §B.1 · p. 16

Workloads and settings

  • The evaluation reports three-run accuracy variation; MathVista uses variable-length any-resolution visual context, while Video-MME uses 32 resized frames pooled to 196 visual tokens each.Thresholds target reuse ratios of about 0.8 for LLM workloads and 0.6 for VLM workloads; KVCMAS uses an anchor pool of 10.Source: §B.1 · p. 16

Baselines and measurement

  • Baselines run in the same evaluation framework with key-position alignment; GraphFlow represents preconstructed correction, KVComm uses online anchor matching, and selective-recomputation baselines follow layer and token criteria.Anchor construction time is excluded from the generating hop’s TTFT but included in end-to-end latency and throughput; peak GPU memory includes transient factorization state.Source: §B.1 · p. 16

B.2 ACCURACY EVALUATION

The accuracy evaluation summarizes performance on method-generated trajectories, alongside trajectory-length statistics and variation across repeated runs.

Evaluation outcomes

  • Mean accuracy and latency are measured on trajectories generated by each method.Table 4 reports average and maximum accumulated trajectory lengths in model tokens, including visual tokens for VLM workloads.Source: §B.2 · p. 17, Table 4 · p. 17

Trajectory lengths

  • Trajectory lengths are reported as average and maximum accumulated model-token counts, including visual tokens for VLM workloads.The trajectory-length summaries are presented in Table 4.Source: Table 4 · p. 17, §B.2 · p. 17

Run variation

  • Accuracy variation is reported as the standard deviation across three evaluation runs, in percentage points.The variation summary is presented in Table 5.Source: Table 5 · p. 17, §B.2 · p. 17

B.3 THRESHOLD

The reliability threshold trades reuse against accuracy, with a Video-MME transition routed to dense prefill throughout the tested threshold sweep.

Threshold mechanism

  • A higher τ accepts a broader set of corrections and generally increases realized reuse ratio ρ.The threshold applies to normalized entropy of the anchor weights.Source: §B.3 · p. 18

Threshold results

  • Increasing τ generally raises reuse ratio while gradually lowering accuracy; the selected thresholds reach target reuse regimes without entering the more aggressive range associated with larger accuracy loss.Table 6 reports accuracy and reuse ratio across threshold settings.Source: Table 6 · p. 18, §B.3 · p. 18

Video-MME behavior

  • On Video-MME, one of three reuse-eligible transitions is consistently routed to dense prefill, keeping ρ near 0.65.KVCMAS evaluates transitions independently and avoids forcing reuse when anchor correction is unreliable.Source: §B.3 · p. 18

B.4 SERVING EFFICIENCY EVALUATIONS

At 32K shared tokens and 8 QPS, KVCMAS substantially lowers median and tail TTFT versus both NonShared and the next-fastest sharing work. Its advantage grows with shared-context length in concurrent serving.

B.4 SERVING EFFICIENCY EVALUATIONS

  • The evaluation uses Llama-3.1-8B on an A100 80 GB GPU, replaying accumulated 128-token agent outputs through the first generated token to measure TTFT.Selective recomputation at ρ = 0.9 and 0.8 is compared with delta correction at ρ = 0.8 and 0.6; QPS 0 is single-stream execution.Source: §B.4 · p. 19
  • At 32K shared tokens and 8 QPS, KVCMAS delivers 2.0× p50 and 2.1× p90 TTFT speedups over NonShared, and reduces both TTFTs by 27% versus GraphFlow.The advantage increases with shared-context length; KVComm exceeds GPU memory capacity at 8K and 32K because it stores full-dimensional anchor states.Source: §B.4 · p. 19, Table 7 · p. 20

B.5 CHAINED CORRECTION EFFICIENCY EVALUATIONS

Chained correction improves accuracy and reduces TTFT while preserving an exact first-agent cache and avoiding separate context-free reference prefill. KVCMAS maintains strong TTFT speedups even with longer accumulated agent outputs.

B.5 CHAINED CORRECTION EFFICIENCY EVALUATIONS

  • Chained correction improves accuracy by 6.44% points on MMLU and 1.53 points on Video-MME.An exact first-agent cache and use of each materialized cache for the next transition prevent first-agent correction error from entering downstream shared context.Source: Table 8 · p. 21, §B.5 · p. 21
  • At 32K shared tokens and 8 QPS, chained correction reduces both p50 and p90 TTFT by 34%.It removes the separate reference prefill; benefits grow with longer shared contexts and higher request rates.Source: Table 9 · p. 21, §B.5 · p. 21, Figure 9 · p. 21

B.6 SERVING EFFICIENCY WITH LONG AGENT OUTPUTS

  • The long-output evaluation adds 0.375K, 1.5K, or 6K accumulated output tokens to a fixed 2K non-generated shared context in a four-agent workflow.The final agent receives outputs from the three preceding agents; Figures 10 and 11 show p50 and p90 trends, and Table 10 reports numerical results for 1.5K and 6K.Source: §B.6 · p. 22, Figure 10 · p. 22, Figure 11 · p. 22
  • At 6K shared generated tokens and 8 QPS, KVCMAS provides 2.3× p50 and 2.4× p90 TTFT speedups over NonShared.It retains the lowest TTFT among KV cache sharing works under high request rates at 0.375K, 1.5K, and 6K generated-token settings; KVComm exceeds GPU memory capacity at 6K.Source: §B.6 · p. 22, Figure 10 · p. 22, Figure 11 · p. 22

C.1 PER-REQUEST EFFICIENCY

Across per-request efficiency and scaling evaluations, KVCMAS achieves strong TTFT, throughput, and memory results, while compact low-rank states support larger ranks, anchor pools, and longer agent workflows.

C.1 PER-REQUEST EFFICIENCY

  • At 2K, 8K, and 32K shared tokens, estimated KVComm peak memory is 2.5×, 6.2×, and 14.9× KVCMAS, respectively.KVCMAS memory remains comparable to selective recomputation and nearly unchanged across reuse ratios; KVComm runs out of memory at 2K due to fragmentation.Source: Table 11 · p. 24, §C.1 · p. 24
  • At 8K and 32K shared tokens, KVCMAS achieves the lowest TTFT and highest throughput among KV cache sharing works; at 2K, it remains comparable to the fastest work.The evaluation measures reuse-request TTFT and end-to-end trajectory throughput under single-stream execution.Source: §C.1 · p. 24, Table 12 · p. 25, Figure 12 · p. 25

C.2 RANKS OF BASE CACHE AND DELTA CORRECTION

  • Increasing both ranks from 16 to 32 improves accuracy on MMLU and Video-MME; beyond 32, accuracy changes marginally while memory continues to increase.The (128, 128) configuration exceeds GPU capacity, so KVCMAS uses rC = r∆ = 32 to capture most full-dimensional accuracy with a compact anchor pool.Source: Figure 13 · p. 26, §C.2 · p. 26

C.3 ANCHOR POOL SIZE

  • At matched V = 20, KVCMAS uses 31.94 GB versus KVComm’s estimated 435.6 GB; V = 10 captures most accuracy gain, with only marginal additional gain at V = 20.TTFT and throughput change little beyond V = 5, making V = 10 the balanced default.Source: V TTFT (s) Throughput Memory MMLU… · p. 27, Table 13 · p. 27

C.4 SCALING WITH THE NUMBER OF AGENTS

  • Across agent counts, KVCMAS has the lowest TTFT among KV cache sharing works, with a larger advantage as more agents reuse the shared context.It has the highest throughput at N = 4 and N = 8, remains close to the highest at N = 2, and peak memory increases only marginally.Source: Table 14 · p. 28, §C.4 · p. 28

C.5 SCALING WITH THE NUMBER OF ROUNDS

  • Across evaluated round counts, KVCMAS achieves the lowest TTFT and highest throughput among KV cache sharing works.Additional rounds amortize initial dense prefill across more reuse requests; peak memory increases slightly as new agent outputs add compact low-rank anchor states.Source: Table 15 · p. 29, §C.5 · p. 29
Loading 2609.34060v1…