PaperlayerView exact revision on arXiv

Source-linked AI summary

PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction

Hyesung Jeon, Hyeongju Ha, Jae-Joon Kim

arXiv:2609.34054v1cs.LG

TL;DR

Multi-LoRA agents repeatedly prefill shared context, and existing KV-cache sharing methods either have training or architecture constraints or retain substantial computation. PReCache precomputes agent-specific LR caches and reconstructs the shared base cache from adapter-free hidden states. Across four backbone–benchmark combinations, ReBaseShared stays closest to NonShared in accuracy, with an average drop of 1.1% points.

  • Problem

    Multi-LoRA agents repeatedly process growing shared trajectories, while existing cache-sharing methods have training or architecture constraints or retain substantial computation.

    Source: §1 · p. 2

  • Method

    PReCache precomputes compact agent-specific LR caches as context is first processed and reconstructs the shared base cache from adapter-free hidden states.

    Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 5

  • Results

    Across four backbone–benchmark combinations, ReBaseShared remains closest to NonShared overall, with an average accuracy drop of 1.1% points.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    ReBaseShared achieves accuracy comparable to BaseShared while eliminating most of its repeated prefill computation.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Multi-LoRA agent systems enable efficient role specialization by sharing a common backbone model. However, each agent repeatedly processes the growing shared trajectory and constructs its own KV cache, introducing substantial memory and computation redundancy in long-horizon tasks. Existing KV cache sharing methods reduce this repeated prefill, but they either require additional training or architectural constraints or retain substantial model computation. Moreover, direct cache reuse causes the current agent to rely on cache states generated by the previous agent's adapter, weakening the role-specific behavior encoded by its own LoRA. We present PReCache, a training-free KV cache sharing framework with two designs, namely PreLRShared and ReBaseShared, that share the base cache computed using the pretrained weights and precompute a compact agent-specific low-rank (LR) cache. To remove repeated prefill, PreLRShared precomputes each agent's LR cache when the shared context is first processed, allowing the current agent to use its own LR cache without reprocessing context processed by previous agents. To improve sharing accuracy, ReBaseShared reconstructs the shared base cache from adapter-free hidden states, reducing the remaining error caused by the previous agent's adapted representation. To minimize its reconstruction cost, we propose two inference schemes tailored to single-stream inference and concurrent serving, performing the same reconstruction after each agent's turn or alongside its execution, respectively. Across multiple models and agent benchmarks, PreLRShared achieves up to a 3.1x TTFT speedup and a 2.3x improvement in per-request throughput over inference without KV cache sharing. ReBaseShared best preserves accuracy overall among the evaluated cache-sharing methods, with an average drop of only 1.1 points relative to inference without cache sharing.

1 INTRODUCTION

Multi-LoRA agents share a pretrained backbone but repeatedly prefill the growing shared trajectory and build separate KV caches. PReCache precomputes each agent’s compact LR cache as context first arrives and reconstructs the shared base cache from adapter-free hidden states.

1 INTRODUCTION

  • Each agent independently prefills the expanding shared trajectory and builds a separate KV cache, increasing computation and memory redundancy.The trajectory accumulates requests, retrieved information, tool observations, and earlier agents’ outputs.Source: §1 · p. 2
  • BaseShared preserves an agent-specific LR cache but still makes each agent process the accumulated context to construct it, leaving most repeated prefill computation unreduced.Its computational cost can be comparable to or higher than token-wise recomputation methods.Source: §1 · p. 2
  • Existing sharing approaches either require additional training or architectural constraints, or retain substantial recomputation and can degrade accuracy with heterogeneous adapters.PReCache targets both reduced repeated prefill and KV-cache memory without substantial accuracy degradation for existing multi-LoRA agents.Source: §1 · p. 2
  • PReCache precomputes compact agent-specific LR caches as each context segment is first processed, so later agents need not reprocess the accumulated trajectory.ReBaseShared additionally reconstructs the shared base cache from adapter-free hidden states to reduce its dependence on the previous agent’s adapted representation.Source: §1 · p. 2

2 BACKGROUND

Multi-LoRA agents combine a shared pretrained backbone with role-specific adapters, but adapter-conditioned hidden states make their KV caches agent-dependent. Prior sharing methods trade accuracy, computation, memory, or compatibility, motivating PReCache’s cache construction designs.

2.1 MULTI-LORA BASED AGENT SYSTEMS

  • LoRA adds a low-rank, agent-specific adapter contribution to the shared backbone projection, while the base contribution uses the same frozen projection.The adapter contribution differs across agents and propagates through later layers, making even their base contributions agent-dependent.Source: §2.1 · pp. 2–3
  • Because hidden states vary with the adapter, agents must separately process shared context to construct their own KV caches, increasing computation and memory use.These costs grow as the shared trajectory grows.Source: §2.1 · p. 3

2.2 MULTI-AGENT KV CACHE SHARING

  • Directly reusing another agent’s full cache can degrade accuracy; learned or calibrated methods have training, model-pair, or architecture requirements, while prefix-based corrections provide no adapter-specific correction.With matched prefixes and token positions, those correction variables remain unchanged across agents.Source: §2.2 · p. 3
  • Selective recomputation targets sensitive layers or tokens, but limited recomputation can leave accuracy degradation, while larger ratios reduce computation and memory savings.DroidSpeak, CacheBlend (Yao et al., 2025), and RelayCaching (Geng et al., 2026) select layers or tokens for recomputation.Source: §2.2 · p. 3
  • LRAgent (Jeon et al., 2026) BaseShared retains separate agent-specific LR caches but still reprocesses accumulated context; BaseLRShared avoids this by requiring a shared down-projection during training.Figure 1 overviews PReCache’s cache construction across consecutive agent turns.Source: §2.2 · p. 3, Figure 1 · p. 4

3 METHODOLOGY

PReCache precomputes agent-specific LR caches when context is first processed, then shares a base cache across agent turns. ReBaseShared reconstructs that base cache from adapter-free hidden states, with schedules that place reconstruction after a turn or alongside execution.

3.1 PRELRSHARED: LR CACHE PRECOMPUTATION

  • PreLRShared constructs every agent’s rank-r LR cache from hidden states when each context segment is first processed, replacing later full-backbone processing over that accumulated context.The current agent reuses its precomputed LR cache and shared base cache for prior context, processing only newly added context with its adapted backbone.Source: §3.1 · p. 5

3.2 REBASESHARED: SHARED BASE CACHE RECONSTRUCTION

  • ReBaseShared constructs the shared base cache as XneutW0 from adapter-free hidden states, reducing dependence on the previous agent’s adapter-conditioned representation.The neutral hidden states also supply other agents’ precomputed LR caches, while the current agent uses its adapted hidden states for its own LR cache.Source: §3.2 · pp. 5–6
  • Figure 2 reports lower layer-wise hidden-state and base-cache relative errors for ReBaseShared than PreLRShared, including at the last layer across agent transitions.The reference is the base cache the current agent constructs; FullShared has larger error because it reuses the previous agent’s entire KV cache.Source: §3.2 · p. 5, Figure 2 · p. 5
  • ReBaseSharedDB runs adapted and adapter-free paths together during prefill and decoding, while keeping neutral-cache reconstruction from affecting the current agent’s generation.The adapted path produces next-token logits; the adapter-free path extends the neutral base cache for subsequent agents.Source: §3.2 · p. 6
  • ReBaseSharedLP runs the adapted path during the agent’s turn, then performs one contiguous adapter-free prefill over newly added context after decoding.LP and DB produce logically equivalent cache states but differ in when reconstruction occurs; LP suits single-stream edge inference and DB concurrent server-side serving.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments evaluate PReCache’s accuracy and efficiency in a three-agent setup across two backbones and benchmarks. ReBaseShared stays close to NonShared in accuracy, while the designs reduce repeated prefill and retain efficiency as trajectories grow and serving load rises.

4.1 IMPLEMENTATION SETUP

  • The evaluation uses three role-specific agents, two 8B instruction-tuned backbones, HotpotQA and ScienceQA, and controlled trajectories from 1.9k to 66.4k tokens.Single-stream tests use an A6000; concurrent serving uses vLLM on an A100 with a fixed 17.3k-token trajectory.Source: §4.1 · pp. 6–7

4.2 BENCHMARK ACCURACY EVALUATION

  • ReBaseShared’s average accuracy drop is 1.1 percentage points versus NonShared across four backbone–benchmark combinations, while remaining comparable to BaseShared.PreLRShared’s average degradation ranges from 1.0 to 3.3 percentage points; ReBaseShared reduces it through reconstruction from adapter-free hidden states.Source: Table 2 · p. 7, §4.2 · p. 7
  • Natural-trajectory tests place both designs in a favorable accuracy–latency region, and 20 complete benchmark runs show low standard deviations.Source: §4.2 · p. 7

4.3 SYSTEM EFFICIENCY

  • On a single A6000 GPU, across both backbones, PreLRShared reduces TTFT by up to 3.1× and improves per-request throughput by up to 2.3× over NonShared.ReBaseSharedLP reduces TTFT by up to 3.0× and improves per-request throughput by up to 1.6× over NonShared.Source: Figure 3 · p. 8, §4.3 · p. 8
  • Under unsaturated concurrent load, PreLRShared and ReBaseShared provide 1.6× and 1.3× higher per-request throughput than NonShared, respectively.At higher request rates, PreLRShared and ReBaseSharedDB sustain lower TTFT over a wider range as more computationally intensive methods saturate earlier.Source: Figure 4 · p. 8, §4.3 · p. 8
  • LP performs reconstruction after decoding, yielding lower TTFT and higher throughput than DB in single-stream inference; DB sustains lower TTFT and higher throughput than LP as concurrent request rates increase.LP keeps reconstruction outside the next agent’s TTFT, whereas DB incorporates the adapter-free path into the serving batch during the current turn.Source: §4.3 · pp. 8–9, Figure 5 · p. 9
  • At 66.4k tokens, PreLRShared and ReBaseShared remain within 2% of FullShared’s peak memory and use 17–23% less than selective recomputation baselines.Both store one shared base cache and compact per-agent LR caches rather than full-dimensional KV cache replications.Source: Figure 6 · p. 9, §4.3 · p. 9

5 CONCLUSION

PReCache shares KV caches across existing multi-LoRA agents through two complementary designs. Across models and benchmarks, ReBaseShared stays close to NonShared in accuracy while reducing repeated prefill, and PreLRShared improves single-stream efficiency.

5 CONCLUSION

  • PReCache is a training-free KV cache sharing framework for existing multi-LoRA agents, comprising PreLRShared and ReBaseShared.Source: §5 · p. 9
  • ReBaseShared maintains accuracy within an average of 1.1 percentage points of NonShared, comparable to BaseShared, while avoiding most repeated prefill computation.Source: §5 · p. 9
  • PreLRShared achieves up to a 3.1× TTFT speedup and 2.3× higher per-request throughput than NonShared in single-stream inference.Both designs keep peak memory within 2% of FullShared at the longest trajectory and retain efficiency advantages under concurrent serving.Source: §5 · p. 9

A.1 DERIVATION

The derivation compares relative cache error when the current agent uses the previous agent’s base cache versus a neutral base cache.

A.1 DERIVATION

  • At a fixed layer and token position, the derivation compares base caches built from the previous agent’s, current agent’s, and adapter-free hidden states.The comparison concerns context previously processed by one agent and received by another.Source: §A.1 · p. 15
  • The relative-error comparison uses Euclidean norms over the cache vector at that layer and token position.It contrasts using the previous agent’s base cache with using the neutral base cache.Source: §A.1 · p. 15
  • Γerr > 1 means the neutral base cache has lower relative error; at ρ = 2 cos θ, the two base caches have equal relative error.The condition also applies to hidden states, where strict inequality means adapter-free reconstruction has strictly lower error against the current agent’s state.Source: §A.1 · p. 15

A.2 MEASURED GEOMETRY

Across the measured layers, the neutral representation has lower average relative error than the previous-agent representation. ReBaseShared reduces base-cache error without repeated backbone processing.

A.2 MEASURED GEOMETRY

  • Measurements cover LLaMA-3.1-8B-Instruct and Ministral-8B-Instruct across plan-to-action, action-to-reflect, and reflect-to-plan transitions on HotpotQA.For each layer, token-wise quantities are averaged across tokens, prompts, and agent transitions.Source: §A.2 · p. 16
  • Γerr averages remain above one at every measured layer for both backbones and for both hidden states and base caches.The layer-wise averages summarize token-wise quantities across prompts and agent transitions.Source: Figure 7 · p. 16, §A.2 · p. 16
  • ReBaseShared reduces base-cache error without repeated backbone processing and yields accuracy comparable to BaseShared through a different cache construction.BaseShared uses the same previous-agent base cache as PreLRShared, so base-cache geometry alone does not explain its primary accuracy benefit.Source: §A.2 · p. 16
  • Across four backbone–benchmark pairs, ReBaseShared reduces PreLRShared’s average degradation, though the improvement varies across difficulty groups.The analysis targets the shared base cache; agent-specific LR caches remain separate, with small differences after agent-specific down-projections.Source: §A.2 · p. 16

B.1 BENCHMARK LATENCY

On actual HotpotQA trajectories, PreLRShared and ReBaseSharedLP have the most favorable overall accuracy-latency trade-offs. Methods with greater accuracy degradation generally incur more agent steps and longer latencies.

B.1 BENCHMARK LATENCY

  • Latency measurements use actual LLaMA-3.1-8B HotpotQA trajectories; TTFT is summed across invocations, while model latency includes prefill, decoding, and cache reconstruction.End-to-end latency additionally includes tool calls and other agent-side overhead; Table 3 reports latency and trajectory lengths.Source: §B.1 · p. 17, Table 3 · p. 17
  • PreLRShared and ReBaseSharedLP remain closest to Figure 8’s upper-left region, indicating the most favorable overall accuracy-latency trade-offs.The figure compares accuracy with TTFT, model latency, and end-to-end latency; higher accuracy and lower latency are preferred.Source: Figure 8 · p. 17, §B.1 · p. 17
  • Methods with greater accuracy degradation generally require more agent steps and longer trajectories, increasing model and end-to-end latency even when TTFT falls.The gap between average and maximum trajectory lengths indicates that some requests grow substantially longer than average.Source: §B.1 · p. 17

B.2 ACCURACY DEVIATION

Repeated evaluations show small run-to-run accuracy variation, while controlled long traces characterize single-stream efficiency beyond benchmark trajectory lengths.

B.2 ACCURACY DEVIATION

  • Accuracy standard deviations are computed over 20 complete evaluations for each method and benchmark setting on HotpotQA and ScienceQA.Table 4 reports the standard deviations in percentage points.Source: §B.2 · p. 18, Table 4 · p. 18
  • Accuracy standard deviations stay below 0.60 percentage points; FullShared and selective-recomputation degradation generally exceeds run-to-run variation, while BaseShared and ReBaseShared differ little.Variation in external search results and model generation contributes to remaining differences across runs.Source: §B.2 · p. 18, Table 4 · p. 18

C.1 SINGLE-STREAM EDGE INFERENCE

  • At longer controlled trajectories, PreLRShared achieves higher throughput than ReBaseSharedLP because LP reconstruction adds inference time.The traces span 1.9k–66.4k tokens on LLaMA-3.1-8B and Ministral-8B with one A6000 GPU; Table 5 reports TTFT and throughput.Source: §C.1 · p. 19, Table 5 · p. 19
  • The controlled three-agent traces extend beyond naturally generated benchmark lengths and retain short decoding segments to reflect prefill-dominated execution.ReBaseSharedLP reconstruction is included in completion time but occurs before the next invocation, outside that invocation’s TTFT.Source: §C.1 · p. 19

C.2 CONCURRENT SERVER-SIDE SERVING

In concurrent serving, PreLRShared has the highest per-request throughput and lowest p50 TTFT among the paper’s methods across the request-rate sweep. LP and DB scheduling suit different serving settings, while both sharing designs remain close to FullShared and BaseShared in peak memory as trajectories grow.

C.2 CONCURRENT SERVER-SIDE SERVING

  • PreLRShared maintains the highest per-request throughput and lowest p50 TTFT across the request-rate sweep; ReBaseSharedDB also improves both metrics over BaseShared.Differences in p90 and p99 results grow after saturation as queueing accumulates.Source: Table 6 · p. 20, §C.2 · p. 20

C.3 LP AND DB COMPARISON

  • LP suits single-stream inference, while DB suits concurrent serving by integrating reconstruction into the continuous batch rather than delaying queued requests with a separate prefill.LP performs reconstruction as a contiguous post-turn prefill, which can overlap with tool execution.Source: §C.3 · p. 21, Table 7 · p. 21, Table 8 · p. 21

C.4 MEMORY USAGE

  • PreLRShared and ReBaseShared remain close to FullShared and BaseShared in peak memory as trajectory length increases.They store one shared base cache and compact per-agent LR caches; reconstruction hidden states are released layer by layer rather than retained as a second persistent full-dimensional cache.Source: Table 9 · p. 21, §C.4 · p. 21

C.5 EFFECT OF AGENT NUMBER

As agent count doubles from four to eight at approximately the same 64k-token trajectory, PreLRShared and ReBaseSharedLP show smaller TTFT and throughput declines than the compared baselines. Their peak memory increases are also smaller, as both methods use shared base and compact per-agent LR caches.

C.5 EFFECT OF AGENT NUMBER

  • From N = 4 to N = 8, TTFT rises 3% for PreLRShared and 7% for ReBaseSharedLP, versus 32–53% for NonShared, BaseShared, and selective recomputation.The comparison uses eight turns with 8k retrieved-context tokens per turn, increasing agent count while holding total trajectory length approximately constant.Source: Table 10 · p. 22, §C.5 · p. 22
  • From N = 4 to N = 8, throughput decreases approximately 6% for PreLRShared and ReBaseSharedLP, versus 20–30% for the baselines.ReBaseSharedLP has lower throughput than PreLRShared because LP reconstruction remains part of wall-clock completion time.Source: §C.5 · p. 22
  • As N doubles, peak memory rises approximately 1–2% for PreLRShared and ReBaseShared, compared with 16–20% for selective recomputation and 26% for NonShared.The sharing methods store one shared base cache and compact per-agent LR caches, whereas NonShared maintains a full KV cache per agent.Source: §C.5 · p. 22, Table 10 · p. 22

D.1 LORA RANK

For LoRA rank, accuracy increases from r = 4 to r = 8 and changes little above that, while throughput costs rise modestly with rank. Under parameter-matched QKVO adaptation, PReCache retains long-context efficiency advantages, supporting QV as the default while also accommodating key adaptation.

D.1 LORA RANK

  • ReBaseShared stays within 1.0% points of NonShared across r = 8–32, with accuracy varying by less than 0.3% points; accuracy changes little above r = 8.Accuracy increases from r = 4 to r = 8 for all methods, and higher ranks do not reduce the accuracy gap from KV cache sharing.Source: §D.1 · p. 23, Table 11 · p. 23
  • As rank increases from 8 to 64, throughput decreases 2% for PreLRShared and 4% for ReBaseSharedLP, while PreLRShared remains within 9% of FullShared across ranks.The reported rank-related costs come from LR cache construction and Flash-LoRA-Attention; r = 8 is described as a suitable balance with accuracy saturation above that rank.Source: §D.1 · p. 23, Table 12 · p. 23

D.2 LORA PROJECTION CONFIGURATION

  • At 66.4k tokens under parameter-matched QKVO adaptation, PreLRShared and ReBaseSharedLP retain TTFT and throughput improvements over NonShared, while BaseShared does not.QKVO with r = 4 matches the LoRA parameter count of QV with r = 8, but adds key-cache and LR-cache computation; the results support QV as the default while showing PReCache remains effective with key adaptation.Source: §D.2 · p. 24, Table 13 · p. 24
Loading 2609.34054v1…