PaperlayerView exact revision on arXiv

Source-linked AI summary

CacheSmith: Accurate RAG Serving under Latency SLOs with Adaptive KV Reuse

Xinyue Ma, Damin Lee, Aashaka Shah, Myeongjae Jeon

arXiv:2610.18925v1cs.ARcs.DC

TL;DR

RAG context caching reduces prefill cost but omits cross-chunk attention, and static selective-recomputation methods do not adapt well to heterogeneous requests or dynamic workloads. CacheSmith combines configurable WarmUp, Probe, and Patch phases with budget-aware planning and SLO-aware scheduling. It improves quality-gated SLO attainment by up to 27.5% over four state-of-the-art methods.

  • Problem

    Context caching misses cross-chunk attention, while prior selective-recomputation methods use static designs that do not adapt to heterogeneous requests and fluctuating load.

    Source: §1 · p. 1, §2.2 · p. 3, §3 · p. 3

  • Method

    CacheSmith uses configurable WarmUp, Probe, and Patch recomputation with a budget-aware planner and SLO-aware scheduler to coordinate execution configurations, batch formation, and budget allocation.

    Source: §1 · p. 2

  • Results

    CacheSmith improves quality-gated SLO attainment by up to 27.5% over EPIC, ProphetKV, CacheBlend, and FusionRAG across four tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CacheSmith coordinates batching and dynamic execution planning across workflow stages to maximize quality-gated SLO attainment.

    Source: §8 · p. 12

Abstract

from arXiv · show

This paper presents CacheSmith, a KV cache reuse system for RAG prefill that maximizes quality-gated SLO attainment. While context caching lowers prefill latency by reusing precomputed KV states for retrieved chunks, it misses cross-chunk attention that a full prefill would compute over concatenated inputs. Prior selective recomputation schemes tackle this problem but enforce static, one-size-fits-all configurations that fail to effectively handle request heterogeneity or dynamic latency budgets. CacheSmith decomposes selective recomputation into a three-phase workflow of WarmUp, Probe, and Patch that can be dynamically configured to exploit a wide range of cost-accuracy trade-offs. At serving time, an SLO-aware scheduler in CacheSmith forms batches to mitigate inter-request interference during layer-by-layer prefill, while a budget-aware planner configures recomputation phases across the batch to maximize the number of requests satisfying both generation quality and latency targets. Both components make fast, solver-free decisions over profiled cost-accuracy bands. On four benchmarks across varying arrival rates and burstiness levels, CacheSmith improves quality-gated SLO attainment by up to 27.5%, over four state-of-the-art baselines.

1 Introduction

RAG context caching lowers prefill cost but misses cross-chunk attention, while fixed selective-recomputation plans struggle with heterogeneous requests and changing latency budgets. CacheSmith combines configurable recomputation with budget-aware planning and SLO-aware batching; it improves quality-gated SLO attainment by up to 27.5% over four state-of-the-art methods.

1 Introduction

  • Context caching misses cross-chunk attention, and fixed recomputation plans can under-recompute some requests or exceed stringent TTFT SLOs.The missing attention matters for queries that synthesize facts or contextual dependencies across chunks.Source: §1 · pp. 1–2
  • CacheSmith decomposes selective recomputation into WarmUp, Probe, and Patch, with four tunable knobs exposing cost-accuracy trade-offs and enabling request-level adaptation.The knobs are selection signal, warm-up depth, probe length, and patch coverage.Source: §1 · p. 2
  • A budget-aware planner selects accuracy-maximizing configurations, while an SLO-aware scheduler batches requests and allocates budgets using their remaining latency slacks.Runtime scheduling over profiled budget bands requires simple arithmetic, imposing minimal serving-critical-path overhead.Source: §1 · p. 2
  • WarmUp computation can mask initial host-to-GPU KV transfers, while deeper Probe layers may increase transfer volume and risk stalls that threaten SLOs.CacheSmith accounts for potential transfer stalls when evaluating plans and can steer toward alternatives within the latency budget without sacrificing accuracy.Source: §1 · p. 2
  • Up to 27.5% higher quality-gated SLO attainment and up to 86% lower P99 TTFT than EPIC, ProphetKV, CacheBlend, and FusionRAG.Evaluated across four tasks under varying arrival patterns, SLO constraints, and data-transfer bandwidth.Source: §1 · p. 2

2 Background

RAG prefill processes retrieved chunks alongside the query, increasing input length and making TTFT SLOs challenging. Context caching avoids repeated chunk computation but loses cross-chunk context; selective recomputation restores some of it at runtime.

2 Background

  • RAG prefill handles retrieved text chunks in addition to the query, inflating input length and prefill cost and making TTFT SLOs difficult to meet.Prefill builds the per-token KV cache and produces the first output token; decode then generates subsequent tokens.Source: §2 · p. 2

2.1 RAG Context Caching

  • Context caching stores precomputed KV caches for popular chunks and transfers a cached chunk to GPU memory when a request hits it, avoiding repeated prefilling.The caches are computed per chunk in isolation, so they lack preceding-token attention; Full Reuse stitches them together, whereas Full Prefill computes the exact full-prompt KV at higher latency.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Selective Recomputation

  • Selective recomputation updates a small fraction of tokens over concatenated chunks, using KV deviation or query-context interaction to select tokens for recomputation.CacheBlend directly estimates KV deviation, while EPIC uses token position as a static proxy; ProphetKV measures query-context interaction across layers, while FusionRAG approximates it using a query-only pass and final-layer context attention.Source: §2.2 · p. 3
  • Prior approaches use static, isolated designs and do not assess their performance under dynamic serving workloads or production-system goals.The paper frames request heterogeneity and fluctuating load as motivations for workload adaptation.Source: §2.2 · p. 3

3 Static Reuse Under Dynamic Workloads

Static selective-recomputation configurations face request heterogeneity, workload-dependent selection signals, and changing latency slack. The section examines how these factors affect coverage needs, accuracy-cost trade-offs, and batching under load.

3.1 Request-Dependent Recomputation Demand

  • At 40 passages, CacheBlend@15% accuracy falls from 65% at 10 passages to 39%; five warm-up layers cut median sufficient coverage from 20% to 10% and p90 from 65% to 30%, raising accuracy to 63%.Even within a passage-count group, minimum sufficient coverage ranges from near zero to over 80%, motivating per-request coverage.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Task-Dependent Selection Signal

  • At 10% coverage, CacheBlend has 20% higher accuracy than ProphetKV on PassageRetrieval, while ProphetKV has 17% higher accuracy than CacheBlend on RULER-VT.At 80% coverage on PassageRetrieval, the methods reach comparable accuracy (97% vs. 95%) under equivalent TTFT budgets; probing overhead can also exceed full-prefill latency beyond 40% coverage.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Load-Dependent Latency Budget

  • A queue-aware configuration under FCFS batching satisfies two of three illustrative requests by serving them separately, unlike policies that co-batch requests with mismatched recomputation demands.A request counts as served only if it meets both its TTFT SLO and quality target; under fluctuating load, queueing can consume part of its latency budget.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

4 Workload-Adaptive Recomputation

CacheSmith makes selective recomputation configurable so it can adapt token selection and recomputation coverage to workload demands and TTFT budgets. Its WarmUp, Probe, and Patch phases expose the trade-offs among signal quality, overhead, and coverage.

4 Workload-Adaptive Recomputation

  • The design calls for dynamically choosing KV deviation or QCI, exposing warm-up depth and probe length as knobs, and bounding recomputation by the remaining TTFT budget.Warm-up depth can defer signal extraction to deeper layers, while probe length bounds QCI overhead.Source: §4 · p. 5
  • CacheSmith represents recomputation with (S, w, p, r): WarmUp computes initial layers, Probe gathers a selection signal, and Patch recomputes top-ranked tokens up to coverage r.The first two phases determine token ranking; coverage sets the fraction of tokens patched.Source: §4 · p. 5
  • Warm-up and probing complement each other: exact early-layer KV states restore chunk-boundary ground truth, and probing accumulates query-context relevance over those representations.KV deviation alone cannot capture the relevance that probing accumulates.Source: §4 · p. 5
  • The abstraction gives CacheSmith a configurable design space for operating across different TTFT budgets, while keeping online decisions lightweight because they count against TTFT.The figure is identified as the overall CacheSmith architecture.Source: §4 · pp. 5–6, Figure 6 · p. 6

5 CacheSmith Design

CacheSmith coordinates a budget-aware recomputation planner with an SLO-aware scheduler to maximize requests meeting both quality and latency targets. It profiles costs and demand offline, then makes runtime choices about configurations, batches, budgets, and KV movement.

5 CacheSmith Design

  • A request counts toward quality-gated SLO attainment (QSA) only if it meets both its TTFT SLO and its task-specific quality gate.The design separates configuration selection by CacheSmith-Planner from batch formation and budget allocation by CacheSmith-Scheduler.Source: §5 · p. 6

5.1 Recomputation Planner

  • The planner chooses shared selection signal, warm-up depth, and probe length alongside per-request coverage to maximize aggregate quality within the batch budget.It uses normalized QCI-score entropy to choose between QCI and KV-dev, with the batch adopting the majority signal among its requests.Source: §5.1 · p. 6
  • The planner profiles demand by rank depth and selects the depth with the greatest expected number of requests passing the quality gate, without requiring precise demand prediction.The profile uses three anchor depths from synthetic PassageRetrieval requests and interpolates between them.Source: §5.1 · pp. 6–7
  • WarmUp and Probe are split using a transfer-aware cost model that accounts for layer compute times, effective bandwidth, and overlap between computation and KV loading.When transfer lags, the GPU may idle for KV; the planner can use those stalls to recompute lightweight early tokens.Source: §5.1 · p. 7
  • Coverage is shifted between requests to maximize expected quality-gate passes, and a budget band lets the scheduler bypass planning when budgets are too low or high for optimization to help.The default band runs from 5% to 50% coverage; the planner re-evaluates feasible coverage if bandwidth drops and risks a budget violation.Source: §5.1 · p. 7

5.2 SLO-Aware Scheduler

  • The scheduler plans one batch at a time and evaluates budgets against a conservative projection in which waiting requests run FCFS at their budget floors.It groups similar-length, similar-slack requests to avoid pairing an urgent short request with a long request whose computation sets the batch time.Source: §5.2 · pp. 7–8
  • For transient bursts that exceed KV capacity, the scheduler can raise concurrency through partial KV allocation, while per-batch search considers compositions by deadline and context-length buckets.It searches again with the effective capacity from partial allocation and uses the composition that places more requests; once the bucket fits, the next batch runs fully resident.Source: §5.2 · p. 8

5.3 Serving Runtime

  • The runtime plans transfers for each chosen configuration and loads cached KV asynchronously; partial allocation turns the cache into a sliding window over layers.The window must be at least as deep as the deepest profiled Probe, which may rewind execution to the last WarmUp layer.Source: §5.3 · pp. 8–9

6 Evaluation

Across the evaluated settings, CacheSmith improves quality-gated SLO attainment over BestStatic and combines quality with latency performance. The evaluation also tests component contributions, signal selection, budget-band sensitivity, and transfer-aware planning across workloads and models.

6.1 Experimental Setup

  • The evaluation uses four tasks, multiple arrival patterns and SLO constraints, and bandwidth conditions, with Full Reuse and Full Prefill as design-space extremes alongside four selective-recomputation baselines.The workloads include a mixed-task setting; BestStatic selects one configuration per SLO and arrival pattern using measured results across tasks.Source: §6.1 · pp. 9–10

6.2 End-to-End Performance

  • CacheSmith exceeds BestStatic in 93 of 96 settings and improves QSA by 2–27.5%.The settings span four tasks, six arrival patterns, and four SLO targets.Source: §6.2 · p. 10, §6.1 · p. 10
  • CacheSmith reaches 97% attainment at 89% accuracy on PR, a combination no prior method reaches at any coverage, while its P99 TTFT is shortest except for Full Reuse.The reported comparison attributes the gain to co-optimizing accuracy and SLO attainment rather than leading on either alone.Source: §6.2 · p. 10
  • As link bandwidth drops, baselines spend 12–72% of serving time stalled, while CacheSmith stalls 3–7% and its accuracy rises from 69% to 92% on PR and 66% to 78% on VT.At 3 GB/s, the planner uses 13 WarmUp layers and one Probe; at 6 GB/s, it reaches rank depth 14 almost entirely by probing.Source: §6.2 · p. 10, Figure 10 · p. 10

6.3 Ablation

  • CacheSmith-Scheduler raises QSA by 10% on PR and 20% on VT for CacheBlend, and by 29% for ProphetKV, when assigning their coverage to meet its budget.At CV 1, queue-derived SLO budgeting accounts for nearly the entire improvement; scheduling additions are worth up to 7.5% at CV 3.Source: §6.3 · p. 11, Figure 11 · p. 11
  • CacheSmith exceeds fixed-signal variants on mixed-preference MIX and PR+VT workloads, while on single-task VT KV deviation reaches 33% QSA at CV 3 versus 63% for CacheSmith.The gain on mixed-preference workloads shrinks with burstier arrivals, when larger batches more often co-batch opposing preferences.Source: §6.3 · p. 11, Figure 12 · p. 11
  • Raising the budget-band saturation point to 75% costs 4–8% QSA on PR at CV 3, while changing the floor moves QSA by at most 3%.The passage attributes the saturation-point loss to spending budget that no longer helps the current batch while requests behind it wait.Source: §6.3 · p. 11, Figure 13 · p. 11
  • On Qwen3-32B with TP=2, CacheSmith exceeds BestStatic on VT by 20% at CV 2.5 and 17% at CV 3, and stays within 4% of it on PR across arrival patterns.The rank-accuracy profile was recalibrated with 20% of the full set; Full Prefill reaches 4% on PR at CV 3.Source: §6.3 · p. 11, Figure 13 · p. 11

7 Related Work

CacheSmith occupies the prefill stage of RAG serving, making recomputation amount a runtime decision rather than assuming a fixed amount of work. Its approach complements systems that manage KV placement or broader pipeline scheduling.

7 Related Work

  • Prior KV-reuse systems manage where reusable KV resides and how it moves, whereas CacheSmith chooses recomputation under a budget to address cross-chunk attention.Fine-tuning approaches can avoid online recomputation but require modifying the served model.Source: §7 · p. 11
  • Unlike serving systems that schedule a fixed amount of work per request, CacheSmith sets a batch's recomputation budget alongside its composition, complementing systems that govern resource sharing across RAG pipeline stages.It varies recomputation within one model rather than switching models, and optimizes the prefill stage within broader RAG serving systems.Source: §7 · p. 12

8 Conclusion

CacheSmith addresses rigid selective-recomputation configurations that struggle with heterogeneous requests and dynamic latency budgets. It coordinates SLO-aware batching with budget-aware execution planning to maximize quality-gated SLO attainment.

8 Conclusion

  • CacheSmith pairs an SLO-aware scheduler with a budget-aware planner to maximize quality-gated SLO attainment.The system coordinates batching and dynamic execution planning across workflow stages.Source: §8 · p. 12
  • CacheSmith decomposes selective recomputation into a modular workflow.Source: §8 · p. 12
  • Existing selective-recomputation schemes use rigid configurations that fail under dynamic latency budgets and heterogeneous requests.Source: §8 · p. 12
Loading 2610.18925v1…