PaperlayerView exact revision on arXiv

Source-linked AI summary

KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

Zedong Liu, Xinyang Ma, Dejun Luo, Hairui Zhao, Bing Lu, Wenjing Huang, Yida Gu, Xingchen Liu, Zheng Wei, Jinyang Liu, Dingwen Tao, Guangming Tan

arXiv:2605.13734v1cs.DCcs.AIcs.NI

TL;DR

Disaggregated LLM serving makes KV cache a dominant network and storage payload, while static compression configurations cannot reliably match changing workloads and service conditions. KVServe builds a modular compression strategy space, searches it efficiently with Bayesian profiling, and selects profiles online under quality and SLO constraints. It achieves up to 9.13× JCT reduction in PD-separated serving and up to 32.8× TTFT reduction in KV-disaggregated serving.

  • Problem

    Static KV compression configurations do not adapt to changing workload, bandwidth, and SLO or quality conditions in disaggregated serving.

  • Method

    KVServe combines a modular, composable strategy space, Bayesian profiling with a 3D Pareto candidate set, and an online controller using analytical latency modeling and lightweight bandit correction.

  • Results

    Up to 9.13× JCT reduction is achieved in PD-separated serving, and up to 32.8× TTFT reduction in KV-disaggregated serving.

  • Takeaways & Limitations

    KV compression can be treated as a constrained, service-state-dependent control problem rather than a fixed algorithm choice.

  • Takeaways & Limitations

    The latency model assumes model execution cost is approximately invariant to compression strategy for a fixed model and serving configuration, while offline parameters can drift from online conditions.

Abstract

from arXiv · show

LLMs are widely adopted in production, pushing inference systems to their limits. Disaggregated LLM serving (e.g., PD separation and KV state disaggregation) improves scalability and cost efficiency, but it also turns KV into an explicit payload crossing network and storage boundaries, making KV a dominant end-to-end bottleneck. Existing KV compression are typically static runtime configurations, despite production service context varies over time in workload mix, bandwidth, and SLO/quality budgets. As a result, a fixed choice can be suboptimal or even increase latency. We present \emph{KVServe}, the first service-aware and adaptive KV communication compression framework for disaggregated LLM serving: KVServe (1) unifies KV compression into a modular strategy space with new components and cross-method recomposition; (2) introduces Bayesian Profiling Engine that efficiently searches this space and distills a 3D Pareto candidate set, reducing $50\times$ offline search overhead; and (3) deploys a Service-Aware Online Controller that combines an analytical latency model with a lightweight bandit to select profiles under constraints and correct offline-to-online mismatch. Integrated into vLLM and evaluated across datasets, models, GPUs and networks, KVServe achieves up to $9.13\times$ JCT speedup in PD-separated serving and up to $32.8\times$ TTFT reduction in KV-disaggregated serving.

1 INTRODUCTION

Disaggregated serving makes KV cache a networked payload and bottleneck, while changing workloads and service conditions make static compression choices unreliable. KVServe addresses this with a composable strategy space, efficient offline profiling, and service-aware online selection.

  • Motivation: Disaggregated serving separates prefill and decode or offloads KV state, turning KV cache into a large payload transferred across network and storage boundaries.This exposes KV movement as a bandwidth-dependent system cost, especially for long-context workloads.
  • Motivation: 2.1 Tbps of KV egress bandwidth is required for 32K-token Qwen3-235B requests on a 64-node prefill cluster.Common cloud cross-cluster bandwidth can be below 100 Gbps.
  • Problem: Static choices of transforms, quantization granularities, and codecs can become suboptimal because workload, bandwidth, and SLO conditions change dynamically.The latency-optimal compression choice can switch across workloads and bandwidth regimes.
  • KVServe: KVServe combines a service-aware online controller with an analytical latency model and lightweight bandit to select profiles under quality and SLO constraints.The controller also corrects mismatches between offline profiling and online execution.
  • KVServe: KVServe unifies compression methods as pluggable modular components and expands the strategy space through a new quantization component plus cross-method composition and reuse.This creates an enumerable and extensible space for strategy selection.
  • KVServe: Bayesian Profiling Engine reduces expensive offline search by using Bayesian optimization over the combinatorial strategy space.The reported offline search overhead falls from 1000 hours to the 20-hour scale.

2 BACKGROUND AND MOTIVATION

Disaggregated LLM serving makes KV movement a latency-critical bottleneck, and compression effectiveness varies across workloads and bandwidths. These observations motivate adaptive selection, but the resulting strategy space makes exhaustive profiling impractical.

  • Bottlenecks: At 10–50 Gbps, communication accounts for 16%–60% of JCT in PD-separated serving.Separating prefill and decode can break the shared high-speed interconnect domain, amplifying KV migration cost.
  • Bottlenecks: Up to 66% of end-to-end time is spent on KV communication under 5–15Gbps links for state-disaggregated serving.Remote KV reads therefore become latency-critical for RAG, multi-turn, and templated requests.
  • Workload dependence: KIVI has the best accuracy on Qasper but ranks near the bottom on GSM8K and HumanEval, while DuoAttention shows the opposite pattern.The passage reports workload-dependent accuracy differences across compression strategies.
  • Workload dependence: 6.20× is CacheGen’s best compression ratio on Multi-News, versus 3.98× on HumanEval and 5.36× for MixHQ on HumanEval.The same strategy therefore does not provide stable compression benefits across workloads.
  • Bandwidth dependence: 50/55/110 Gbps are the bandwidth thresholds for CacheGen, MixHQ, and KIVI, respectively, beyond which compression can worsen latency.At higher bandwidth, communication savings may no longer offset compression and decompression overhead.
  • Search challenge: Nearly 10^4 candidates can arise from fine-grained strategy tuning, with each end-to-end profiling run taking about 15 minutes.This combinatorial growth makes exhaustive profiling impractical.
  • Selection challenge: Selecting by compression ratio or quality alone can produce infeasible or suboptimal profiles under latency and SLO constraints.A constrained model must jointly capture compression overhead, post-compression volume, and quality degradation.

3 PROBLEM FORMULATION

KVServe models request-level KV movement as constrained profile selection under workload, bandwidth, latency-budget, and quality requirements. Its JCT model separates strategy-independent execution from compression and movement costs, exposing the factors that determine whether compression helps.

  • Serving System Model: KV movement is decided per request segment for both PD migration and KV offloading or reuse, with one profile kept throughout each request.The decision is made when KV movement begins.
  • Serving System Model: The service context comprises workload class, effective bandwidth, latency SLO, and minimum quality requirement.Workload labels come from an upper-layer router or classifier, while bandwidth abstracts network or I/O goodput.
  • Serving System Model: A profile is represented by compression ratio, effective (de)compression throughput, and workload-dependent quality.Compression ratio compares uncompressed KV volume with compressed volume, while throughput uses encoding and decoding performance.
  • Constrained Optimization: KVServe selects a profile for each dynamic context to satisfy latency and quality requirements while optimizing end-to-end performance.The resulting optimization problem is defined over the selectable strategy set.
  • Constrained Optimization: JCT is decomposed into strategy-independent model execution and additional KV compression and movement costs.Model execution and unrelated operator or scheduling overheads are treated as approximately invariant to the compression strategy for fixed serving conditions.
  • Constrained Optimization: The compressed KV volume equals uncompressed volume divided by the profile’s compression ratio.This relationship feeds the latency model for strategy comparison.
  • Constrained Optimization: Bandwidth bounds possible transfer savings, throughput determines compression overhead, compression ratio determines transmitted volume, and quality captures cost.The model is used to derive benefit conditions and policy responses to changing conditions.

4 DESIGN OVERVIEW

KVServe combines a modular compression strategy pool, efficient offline profiling, and service-aware online selection. It profiles candidate strategies offline, selects under current constraints online, and executes the chosen profile at KV movement boundaries.

  • System Overview: KVServe unifies compression methods into a composable strategy space and optimizes selection under SLO and quality constraints.The framework is designed for adaptive KV communication compression in disaggregated LLM serving.
  • Modular Strategy Pool: The Modular Strategy Pool represents compression as pluggable pipeline components, including new components and cross-method combinations.Representative existing methods are mapped into an enumerable and extensible abstraction.
  • Bayesian Profiling Engine: The Bayesian Profiling Engine uses Gaussian-process Bayesian Optimization to reduce expensive profiling runs and construct a 3D Pareto candidate set.The candidate set supports fast online profile selection.
  • Service-Aware Online Controller: The Service-Aware Online Controller senses runtime context and selects profiles using an analytical latency model plus a lightweight bandit.The bandit corrects system drift and improves robustness using runtime observations.
  • System Pipeline: KVServe operates through Offline Profiling, Online Selection, and Runtime Serving stages.The system searches the strategy pool, chooses a profile under current state and constraints, then executes it at KV movement boundaries.

5 OFFLINE PROFILING ENGINE

KVServe turns KV compression into a composable strategy-space search and uses Bayesian optimization to efficiently identify feasible, non-dominated quality–compression–latency profiles. The profiling engine reduces exhaustive offline search to a manageable budget while preserving trade-off information for runtime selection.

  • Strategy-space construction: KVServe decomposes KV compression into atomic, pluggable stages and explores their Cartesian-product combinations, including the new MixHQ quantization component.MixHQ allocates precision across Retrieval Heads and Streaming Heads rather than making binary pruning decisions, while supporting layer- and token-level importance schemes.
  • Search challenge: The strategy space grows exponentially as configuration granularity advances from pipeline choices to fine-grained parameter tuning, making brute-force profiling impractical.The search must jointly balance compression ratio and accuracy across a massive configuration space.
  • Bayesian profiling: Bayesian optimization with Gaussian-process modeling guides profiling through sample-efficient exploration of mixed categorical and continuous configurations.The engine uses sampled data as an accuracy proxy and exploits stable relative compression-ratio rankings to reduce evaluation cost.
  • Profiling efficiency: 50×: profiling overhead falls from approximately 1,000 hours for exhaustive search to about 20 hours, with convergence in fewer than 80 iterations.The optimized prediction and pruning process transforms the exponential search into a manageable offline task.
  • Candidate-set distillation: The engine projects feasible configurations into accuracy–compression-ratio–latency space and retains the 3D Pareto frontier as a runtime candidate set.This frontier captures non-dominated trade-offs among quality, KV footprint, and delay for later context-aware selection.

6 SERVICE-AWARE ONLINE CONTROLLER

KVServe’s online controller selects compression profiles per request using bandwidth, workload, SLO, and quality context. An analytical policy narrows choices efficiently, while a lightweight residual-corrected bandit adapts to offline-to-online drift under safety constraints.

  • Runtime selection: The controller selects one compression profile at each KV-movement boundary for a request under workload, bandwidth, SLO, and quality constraints.Request-level selection applies to PD migration and remote KV-pool fetching.
  • Analytical filtering: Compression is beneficial only below a profile-specific bandwidth threshold determined by compression ratio and (de)compression throughput, independent of KV volume.Profiles exceeding their threshold can be filtered online because communication savings no longer offset compression overhead.
  • Piecewise policy: For each workload and quality bucket, the latency-minimizing profile is piecewise constant in x = 1/B and can be obtained from the lower envelope of profile latency lines.The resulting policy supports interval lookup rather than searching the full candidate set online.
  • Drift correction: Offline-to-online drift can arise from changing GPU load, queue contention, scheduling, and concurrency, causing analytical latency predictions to deviate from runtime observations.Relying only on offline parameters can move selection away from the true optimum and require re-profiling or manual retuning.
  • Residual-corrected bandit: The residual-corrected bandit learns runtime deviations from the analytical model within a tiny candidate set centered on the model-optimal profile and its neighboring envelope profiles.The candidate set typically contains 2–3 profiles, and each candidate maintains an EWMA residual estimate.
  • Safety and overhead: The controller evaluates at most 2–3 candidates per request, uses SLO feasibility filters and cooldowns, and falls back to a conservative configuration when no profile is feasible.These guardrails limit online exploration risk while keeping control-plane overhead negligible.

7 EVALUATION

KVServe is evaluated across diverse models, datasets, hardware tiers, bandwidths, and disaggregated serving scenarios. It consistently improves latency while maintaining accuracy and adapting compression choices to changing service conditions.

  • Evaluation Setup: KVServe is evaluated across PD Separation and Prefix Caching using multiple models, datasets, hardware tiers, and bandwidth conditions.The evaluation includes Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-32B-Instruct, profiling and unseen datasets, and prefill nodes spanning 10–100 Gbps.
  • End-to-End Performance: 3.15× speedup: KVServe achieves this maximum JCT improvement on bandwidth-constrained devices while maintaining the 97% relative accuracy threshold.Static CacheGen and KIVI configurations frequently violate the threshold on Qwen2.5-7B-Instruct, whereas KVServe maintains precision.
  • End-to-End Performance: 9.13× JCT reduction: KVServe delivers drastic improvements on long-context workloads such as HotpotQA across diverse datasets and models.On short-context tasks such as GSM8K and HumanEval, compression overhead can outweigh communication savings and cause negative optimization for baselines.
  • PD Separation: 9.2× speedup: KVServe accelerates PD-separated serving at 5 Gbps and dynamically selects lower-overhead strategies as bandwidth increases.The evaluation uses Llama-3.1-8B-Instruct and Qwen2.5-32B-Instruct on 2WikiMQA with a Pro 6000 prefill node.
  • Prefix Caching: 32.8× speedup: KVServe converts otherwise infeasible Prefix Caching fetches into valid cache hits while satisfying SLO constraints across 5–15 Gbps.CacheGen falls back to costly recomputation when it cannot find a valid configuration, whereas KVServe selects profiles from its Pareto frontier.
  • Latency Breakdown: < 1 ms: online profile decisions add negligible control overhead, while compression and decompression overhead remains small enough to shift the system from network-bound back to compute-bound.The latency breakdown reports online decisions below 1 ms and compression overhead of 6–9%.
  • Accuracy and Compression Ratio: 7.42× average CR: KVServe-Unified generalizes to unseen datasets with less than 2% relative accuracy loss, while KVServe-Aware reaches 8.28× average CR and 10.12× on Multi-News.KVServe-Aware also reports 100.35% average relative accuracy against the Default baseline; MixHQ selectively preserves significant features while filtering noise.

8 RELATED WORK

Prior KV compression work improves accuracy–compression trade-offs through distribution reshaping, fine-grained precision allocation, and optimized kernels. KVServe differs by adapting compression to dynamic service context and complements broader disaggregated-serving optimizations.

  • KV Cache Compression: Existing KV compression methods mainly use quantization, distribution reshaping, fine-grained precision allocation, and optimized implementations to reduce quality loss and runtime overhead.These techniques target better low-bit representations, sensitivity-aware allocation, and efficient compression and decompression.
  • KV Cache Compression: Most prior KV compression approaches are service-agnostic, using fixed configurations rather than adapting to dynamic runtime context.Aggressive reduction levels can also incur larger quality loss.
  • Disaggregated Serving Optimization: System-level disaggregated-serving work optimizes execution, scheduling, resource allocation, and KV movement, while KVServe studies service-aware KV compression as an orthogonal lever.The paper positions its compression approach as embeddable in both PD-separated and KV-disaggregated serving stacks.

9 CONCLUSION

KVServe treats KV compression as a constrained, service-dependent control problem for disaggregated LLM serving. The paper reports robust speedups across PD separation and KV state disaggregation under changing workloads and bandwidths.

  • Conclusion: KV movement becomes a dominant bottleneck because disaggregated LLM serving turns KV cache into a massive, latency-critical payload.KVServe addresses this by making compression decisions dependent on service state rather than choosing a fixed algorithm.
  • Conclusion: KVServe enables robust end-to-end speedups across PD separation and KV state disaggregation under dynamic workloads and bandwidth.The conclusion frames KV movement as a first-class, constraint-driven control problem.
  • Broader Implications: The service-aware control principle may extend beyond KV caching to other networked state-movement workloads, including parameter offloading and embedding retrieval.This broader applicability is presented as a belief rather than an evaluated result.
Loading 2605.13734v1…