PaperlayerView exact revision on arXiv

Source-linked AI summary

Shared-Prefix KV Reuse Across Standard LoRA Adapters: Quality and Serving Tradeoffs

Dushyant Rajput

arXiv:2609.17109v1cs.AIcs.CL

TL;DR

Serving several LoRA specialists over a shared context can repeat prefill, and evidence is limited for reusing a base cache with already-trained standard adapters. The paper sweeps the specialist takeover boundary and measures quality and serving cost. Reuse improves warm-cache TTFT but does not achieve physical prefix sharing, while quality differences are not consistent.

  • Problem

    The paper asks how much quality inference-time base-cache reuse preserves for already-trained standard LoRA adapters, and what serving cost it entails.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates two LoRA specialists on a shared Qwen3-1.7B backbone, reusing base-computed prefix KV while sweeping the specialist takeover boundary.

    Source: §2 · p. 2, §4 · p. 6

  • Results

    Reuse trades quality for warm-cache TTFT: the GSM8K quality difference varied across settings, and the two-branch implementation never physically shared prefix storage.

    Source: §3.2 · p. 4, §8 · p. 9

  • Takeaways & Limitations

    The measured serving benefit is warm-cache latency, not shared-cache memory savings; persistent shared storage remains unimplemented.

    Source: §3.2 · p. 4, §8 · p. 9

    The n = 500 boundary evaluation overlaps development data, and the authors parked the boundary sweep pending a clean evaluation.

    Source: §6 · p. 8

Abstract

from arXiv · show

A common small-model deployment runs one shared backbone with several LoRA specialists that answer over the same context. Serving them naively re-prefills that shared context once per specialist. We study a narrow, practical question: for already-trained standard LoRA adapters -- not adapters retrained for cache compatibility -- how much task quality is preserved if the backbone's prefill KV cache is computed once and reused across specialists, and what does that buy in serving cost? On a Qwen3-1.7B backbone with two adapters (extractive QA on HotpotQA, arithmetic reasoning on GSM8K), we sweep the boundary at which the specialist takes over from the reused base cache and measure paired quality differences and serving cost. Full-prefix reuse had the lowest prefill cost and a small quality difference on held-out GSM8K (Delta = -4.6 EM at a 160-token budget; -3.0 at 320 tokens; -0.8 under a second training seed -- all favoring native, only the first excluding zero, and the magnitude not consistent). Partial recomputation provided no demonstrated advantage. Neither quality equivalence nor a general boundary-selection rule is established. We also report a closed-form ridge KV translator that did not beat direct reuse, and specialist-dependence contrasts whose intervals all include zero. The measured serving benefit is warm-cache time-to-first-token, which grows with context (~16x at 8K); two-branch peak memory was only 12% lower and, on inspection, the prefix was never physically shared across branches -- this implementation reuses KV values but copies their storage, so shared-cache memory savings are not achieved.

1 Introduction

Shared-context serving can repeat prefill for each LoRA specialist, while existing cache-reuse approaches target same-model prefixes, learned KV mappings, or adapters designed for cache compatibility. This study asks whether already-trained standard LoRA adapters can reuse a base cache and what serving cost that entails.

1 Introduction

  • Each specialist can trigger another prefill over the same retrieved passages, conversation history, or demonstrations.The adapters differ while the shared context remains the same.Source: §1 · p. 1
  • Prior approaches address same-model prefix caching, learned KV-space transfer between models, or adapters modified for exact base-cache reuse.These approaches include PagedAttention [5], RadixAttention [6], linear KV transfer [7], CacheBridge [8], and Activated LoRA [9].Source: §1 · p. 1
  • The study targets already-trained standard LoRA adapters on a shared backbone, without a cache-compatibility training objective.Source: §1 · p. 2
  • It asks how much task quality inference-time base-cache reuse preserves and what serving cost it incurs.Source: §1 · p. 2
  • The paper reports paired quality intervals, serving-cost accounting, a rejected translator, and unresolved specialist-dependence contrasts, while distinguishing exploratory measurements from trusted ones.It also flags the difference between context lengths used for quality and serving-cost measurements.Source: §1 · p. 2

2 Setting and method

The study uses a shared Qwen3-1.7B backbone with QA and math LoRA specialists. It reuses base-computed prefix KV and varies the takeover boundary while holding the prompt and greedy decoding fixed.

2 Setting and method

  • A Qwen3-1.7B backbone hosts rank-16 LoRA specialists for HotpotQA extractive QA and GSM8K math; the serving-cost accounting also considers M specialists sharing a context.The two trained adapters are used for quality evaluation; M enters the adapter-agnostic cost accounting.Source: §2 · p. 2
  • The backbone computes prefix KV with adapters disabled, then the specialist processes the remaining prompt and generates without a key/value mapping.The specialist processes at least one prompt token, so the first answer-token logits always come from the specialist.Source: §2 · p. 2
  • Four takeover boundaries vary how many prompt tokens are reused and which content the specialist re-encodes, with the full prompt and greedy decoding held fixed.Full-prefix reuse sets b = T − 1, leaving the specialist to process the final prompt token before generation.Source: §2 · p. 2, Figure 1 · p. 2, Figure 2 · p. 3
  • Evaluation uses deterministic token-level F1 for extractive QA and exact match for GSM8K, with paired-bootstrap 95% confidence intervals over per-example score differences.Intervals use 10,000 resamples; runs use bf16, greedy decoding, and a 160-token generation cap.Source: §2 · pp. 2–3

3 Serving cost

Reuse reduces warm-cache time-to-first-token, with a larger measured benefit at long context, but the two-branch implementation copies prefix storage rather than physically sharing it. Serving-cost accounting separates prefill reuse from storage and latency measurements.

3 Serving cost

  • With M specialists, reuse computes the shared prefix once, while each specialist still runs its non-reused suffix and generation.The single-prefix cost accounting distinguishes KV footprint from prefill; specialist prefill exceeds base prefill due to adapter matrix-multiply overhead.Source: §3 · p. 3, Table 3 · p. 4

3.1 Warm-cache latency (measured)

  • At 8K, warm-cache TTFT is 30 ms vs. 486 ms, a ≈16× speedup; the measure excludes the one-time base prefill and includes the specialist’s final-token forward.Table 1 reports latency across three matched settings; the one-time base prefill is amortized across specialists.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 Two-branch memory: logical reuse, no physical sharing

  • Two-branch peak allocation is 12% lower at 8K and 5% lower at 2K under reuse, but prefix KV is never physically shared because storage is copied across branches.Persistent shared-cache storage would require an implementation that preserves shared storage during generation; a paged cache is one route.Source: §3.2 · p. 4, Table 2 · p. 4
  • The reported benefit is warm-cache latency, not peak-memory reduction from sharing; quality was evaluated at ≈655-token GSM8K prompts and 2K/8K QA contexts, without claiming equivalence at any scale.Source: §3.2 · p. 4

4 Quality of reused-KV inference

Full-prefix reuse favored native on the held-out GSM8K evaluation, but the observed quality difference varied with generation budget, seed, and QA context length. In the overlapping boundary sweep, partial recomputation showed no demonstrated advantage.

4 Quality of reused-KV inference

  • Own-KV control scored 49.2 versus 48.3 math EM (n = 120), with 3/120 disagreements; numerical divergence was not instrumented.The authors describe this discrepancy as consistent with greedy sensitivity to cache-reconstruction differences, but do not diagnose it further.Source: §4 · p. 5
  • On 500 untouched GSM8K examples, full-prefix base-KV reuse reduced accuracy from 54.4% to 49.8%, a −4.6-percentage-point difference (paired CI [−8.8, −0.4]).Source: §4 · p. 5
  • Base-only trailed native by 46 EM on GSM8K, but hit the 160-token cap on every example, bounding adapter-necessity evidence to that decoding budget.The matched comparisons also reported base-only differences of −26.6 and −29.2 on QA.Source: §4 · p. 5, Table 4 · p. 5
  • At 320 tokens, the first seed's reuse penalty was −3.0 EM; at 160 tokens, the second seed's was −0.8 EM, with only the first configuration's interval excluding zero.The direct paired budget and seed contrasts included zero; two seeds do not characterize seed variability.Source: §4 · pp. 5–6
  • The boundary sweep used overlapping GSM8K examples, whereas the −4.6 penalty came from untouched examples; the overlapping result is insufficient confirmation.The two intervals overlap, so the authors do not claim that overlap caused the difference.Source: §4 · pp. 5–6
  • At 700 tokens, reuse was +0.3 F1 versus native (CI [−1.5, +2.2]); at 2K and 8K, its QA differences were −6.6 and −4.5 F1.The 700-token supplied-context result used truncated distractor paragraphs, and truncation could drop answer-bearing text.Source: §4 · pp. 5–6
  • In the 500-example overlapping boundary sweep, full-prefix reuse was closest to native and cheapest; question takeover had the worst estimate, but its CI reached zero.The sweep was non-monotonic: recomputing more of the prefix was not uniformly better, and no general boundary-selection rule was established.Source: Table 6 · p. 6, §4 · pp. 6–7

5 Comparisons and negative results

The tested ridge KV translator did not improve task quality over direct reuse and added per-token computation. Comparisons also leave both the aLoRA result and specialist dependence unresolved.

Standard LoRA vs Activated LoRA

  • The preliminary aLoRA comparison is uninterpretable because activation correctness was unverified.Its native F1 was 42.6 versus 53.0 for standard LoRA, but the gap cannot be attributed to training quality, capacity, or architecture.Source: §5 · p. 7

A ridge KV translator does not earn its cost

  • The ridge translator did not beat direct reuse on task quality and added per-layer matrix-multiply cost per reused token.The tested ridge map is rejected, but translation as a class is not.Source: §5 · pp. 7–8

Specialist dependence

  • Both specialist-dependence contrasts include zero, so the comparisons do not establish that the math specialist is specifically fragile under reuse.The base-reuse penalty difference was −9.2pp (CI [−21.7, +2.5]); the takeover-seam difference-in-differences was +2.6pp (CI [−3.6, +8.6]).Source: §5 · p. 8

6 Limitations and future work

The reported quality–latency tradeoff is bounded by decoding-budget effects, workload differences, evaluation overlap, and a narrow model and task scope. The boundary sweep is parked, with its exploratory history in Appendix B.

6 Limitations and future work

  • The reuse penalty is partly truncation-related, but the residual negative point estimate leaves a budget-independent penalty neither established nor excluded.At 320 tokens the penalty estimate shrank from −4.6 to −3.0, with a confidence interval including zero.Source: §6 · p. 8
  • Full-prefix reuse on identical prompts does not establish reuse when specialists share background passages across different questions.That setting corresponds to a mid-prompt boundary, where the observed penalty was uncertain.Source: §6 · p. 8
  • The n = 500 boundary evaluation overlaps earlier development data, so it is not an independent held-out confirmation.A clean result requires a frozen harness and primary comparison followed by a fresh, precision- or noninferiority-powered evaluation.Source: §6 · p. 8
  • One 1.7B backbone, two tasks, and two seeds leave seed variability and cross-backbone generality uncharacterized.The penalty magnitude is not shown unstable; replication on another backbone and with more seeds remains needed.Source: §6 · p. 8
  • The boundary sweep is parked, and its exploratory measurement history is in Appendix B.Source: §6 · p. 8

7 Related work

This work distinguishes same-model prefix caching from reuse across different specialists sharing a backbone. It studies already-trained standard LoRA adapters, unlike prior approaches that map KV spaces or modify adapters for exact reuse.

Cross-model KV transfer

  • Cross-model KV transfer maps KV between different-size models whose key/value spaces differ; Heo et al. [7] and CacheBridge [8] target that setting.The same-backbone, different-LoRA-head setting is complementary, and the paper reports a negative ridge-map result there.Source: §7 · p. 9

Adapter-aware caching

  • Activated LoRA [9] modifies adapters for exact base-prefix reuse, while this work measures reuse with already-trained standard LoRA adapters.The paper also reports a partial aLoRA comparison.Source: §7 · p. 9

Prefix caching

  • PagedAttention [5] and RadixAttention [6] cache identical prefixes across requests under one model; this work reuses one prefill across different specialists.Source: §7 · p. 9

8 Conclusion

The conclusion frames base-cache reuse across already-trained standard LoRA specialists as a quality–latency tradeoff, not an equivalence result. Its measured benefit is warm-cache TTFT, while physical shared-cache storage remains unimplemented.

8 Conclusion

  • For composable serving with already-trained standard LoRA specialists, reusing a shared-backbone prefill KV cache favored native quality: held-out GSM8K was −4.6 EM at 160 tokens; QA losses were −6.6/−4.5 F1 at 2K/8K.GSM8K losses were −3.0 EM at 320 tokens and −0.8 under a second seed; only the 160-token result excluded zero.Source: §8 · p. 9
  • ≈16× was the warm-cache time-to-first-token benefit at 8K context.The measured serving benefit grows with context.Source: §8 · p. 9
  • Two-branch peak memory was only 12% lower at 8K, and storage inspection found no physical prefix sharing.The implementation reuses KV values but copies their storage; shared-cache memory savings are not achieved.Source: §8 · p. 9
  • Partial recomputation showed no demonstrated advantage, the ridge translator did not earn its cost, and specialist-dependence contrasts did not resolve.Source: §8 · p. 9

A Specialist-dependence contrasts

Individual math reuse penalty and QA seam intervals exclude zero, but the contrasts between specialists do not. The penalty-difference and seam difference-in-differences intervals include zero, with different sample sizes across separate runs.

A Specialist-dependence contrasts

  • The individual math reuse penalty and QA seam intervals exclude zero, while the penalty difference (n = 120) and seam difference-in-differences (n = 500) intervals include zero.The DiD point estimate is positive; the contrasts come from separate runs with differing n.Source: Figure 4 · p. 10

B Exploratory measurement history

Development measurements of the question-takeover penalty changed, but they are not independent replications because sample draw and prompt construction changed together. The final follow-up overlaps the boundary-development slice, so earlier measurements are treated as development history rather than evidence of a fixed bug.

B Exploratory measurement history

  • The question-takeover penalty changed across development measurements, where sample draw and prompt construction changed together.Adding special tokens was ruled out as a cause because it was a no-op for this tokenizer; the shared instruction preamble and overlapping GSM8K slices remain confounded.Source: B Exploratory measurement history · p. 10
  • The final follow-up overlaps the boundary-development slice (test[80:200] ⊂ test[80:580]), so it is not an independent held-out sample.The final row is reported; earlier measurements are treated as development history, not evidence of a fixed bug.Source: Table 7 · p. 10, B Exploratory measurement history · p. 10

C Reproducibility

The appendix identifies the evaluated models, task setups, measurement procedures, and remaining reproducibility boundaries. Code, checkpoints, configurations, prompts, and per-example scores are available in the repository, while the translator comparison is reported there.

C Reproducibility

  • Code, adapter checkpoints, exact configs, prompts, and per-example score arrays are available in the repository, alongside a fully pinned artifact.Some items remain unpinned, and the appendix is an outline.Source: C Reproducibility · p. 10
  • The experiments use Qwen/Qwen3-1.7B with rank-16 LoRA specialists for HotpotQA QA and GSM8K math, with one aLoRA QA adapter also included.Training seeds were not varied: each task used a single seed, leaving replication across seeds for future work.Source: C Reproducibility · p. 11
  • GSM8K uses four fixed few-shot demonstrations and greedy decoding capped at 160 tokens, while QA uses token-level F1 on the first output line.GSM8K answer extraction takes the last signed-integer match; cap-induced truncation is unlogged, and QA context truncation can drop answer text and was not audited.Source: C Reproducibility · p. 11
  • Systems measurements use one NVIDIA RTX PRO 4500 Blackwell GPU, with timing protocol details pinned in the repository and reported timings from single runs.Single-prefix cost, warm-cache latency, and two-branch peak memory come from separate runs; the physical-sharing check tests prefix-tensor data-pointer identity.Source: C Reproducibility · p. 11
  • The ridge KV translator was not adopted because its comparison with direct reuse showed no quality gain and added per-token matrix-multiply cost.Its per-head ridge maps were calibrated on 40 held-out HotpotQA distractor contexts; comparison values are in the repository.Source: C Reproducibility · p. 11
Loading 2609.17109v1…