PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving

Jiantong Jiang, Yue Yang, Peiyu Yang, Feng Liu

arXiv:2609.33536v1cs.LG

TL;DR

KV memory pressure can make target-only readiness trigger request stalls and preemptions. ElasticKV lets requests execute with a compact intermediate KV state and coordinates its storage, attention, and fidelity transitions. Under high concurrency, it reports lower TTFT than standard vLLM while largely preserving generation quality.

  • Problem

    KV cache growth under longer contexts and higher concurrency pressures GPU capacity, while target-only execution readiness can turn KV shortage into waiting or preemption.

    Source: §1 · p. 1

  • Method

    ElasticKV introduces executable BASE KV alongside TARGET, combining pair-structured storage, dual-mode attention, and pressure-aware fidelity control.

    Source: §1 · p. 2

  • Results

    3.8–4.0× lower mean TTFT and 9.1× lower P90 TTFT than standard vLLM under high concurrency, while largely preserving generation quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation reports latency reductions across workloads, models, and GPU platforms, supporting fidelity as a runtime-managed execution property that adapts to resource pressure.

    Source: §5 · p. 11

Abstract

from arXiv · show

Large language model (LLM) serving is increasingly constrained by the GPU memory consumed by key-value (KV) caches. Existing compression, eviction, and offloading techniques alleviate this pressure, but serving runtimes typically treat only the configured target KV representation as execution-ready. Under memory pressure, this target-only contract can turn KV shortage into request stalls and preemptions. We present ElasticKV, a mixed-fidelity KV runtime built on the observation that target fidelity need not gate execution. ElasticKV introduces a compact intermediate KV state, making fidelity a runtime-managed execution property. To realize this state in a paged serving runtime, ElasticKV combines (i) a pair-structured layout that turns fidelity reduction into reusable GPU capacity, (ii) a dual-mode attention backend that directly consumes the compact state while preserving the native target-only path, and (iii) pressure-aware fidelity management that adapts KV fidelity to memory pressure. Our extensive evaluation across diverse workloads, model families and scales, and GPU platforms demonstrates the effectiveness and generality of ElasticKV. Under high concurrency, ElasticKV achieves 3.8-4.0$\times$ lower time-to-first-token (TTFT) and 9.1$\times$ lower P90 TTFT than vLLM while preserving generation quality.

1. Introduction

The introduction identifies target-fidelity-gated readiness as a runtime constraint that can turn KV pressure into waiting or preemption. ElasticKV makes a compact intermediate KV state executable under pressure, combining layout, attention, and fidelity-management changes to reduce latency while largely preserving generation quality.

1. Introduction

  • Target-only readiness can turn KV shortage into waiting or preemption, potentially requiring later recomputation and producing a preemption-driven P90 TTFT knee as concurrency rises.The runtime may require TARGET on GPU before a request can execute.Source: §1 · p. 1
  • ElasticKV makes compact BASE directly executable and exactly refinable to TARGET, allowing smaller-footprint execution under pressure while keeping TARGET preferred when resources permit.BASE avoids the preemption-driven knee in the representative pressure case and keeps P90 TTFT more stable as concurrency rises.Source: §1 · p. 2
  • Paged runtimes must make reduced-fidelity capacity reusable, let attention execute intermediate KV without penalizing the TARGET-only path, and manage asymmetric transitions without disrupting requests.TARGET-to-BASE releases capacity; BASE-to-TARGET consumes it and therefore requires sufficient headroom.Source: §1 · p. 2
  • 3.8–4.0× lower mean TTFT and 9.1× lower P90 TTFT than standard vLLM under high concurrency, while largely preserving generation quality.ElasticKV also eliminates request preemptions in the reported high-concurrency comparison.Source: §1 · p. 2
  • ElasticKV addresses these challenges with pair-structured storage, dual-mode attention, and pressure-aware fidelity control in a vLLM implementation evaluated across workloads, models, and GPU platforms.The paper frames its contribution as decoupling execution from target-fidelity residency while preserving the native TARGET path.Source: §1 · p. 2

2. Background and Related Work

The background describes paged KV allocation as a common serving substrate and surveys approaches to KV capacity and execution. ElasticKV retains paged serving but makes fidelity itself a runtime-managed execution state.

2.1. Paged KV Cache Management

  • Paged KV management maps request-level logical blocks to fixed-size physical GPU blocks, letting attention follow non-contiguous storage while preserving logical token order.The passage describes PagedAttention as a common substrate extended to offloading, scheduling, block-structured attention, and KV reduction.Source: §2.1 · p. 3

2.2. KV Cache Optimization and Execution Contracts

  • Offloading and reuse systems mainly change where KV resides and how it is transferred, with execution typically resuming once target KV is available on GPU.Han et al. (2026) relax this requirement to overlap transfer with execution; ElasticKV instead manages live GPU KV fidelity under pressure.Source: §2.2 · p. 3
  • Quantization, sparsification, and eviction reduce KV size or retained entries, while some systems co-design reduction with runtime or kernels; ElasticKV uses fidelity transition as runtime control, not a new compression algorithm.The passage distinguishes the execution contract from the precision level of the configured TARGET representation.Source: §2.2 · p. 3
  • ElasticKV makes both BASE and TARGET executable and coordinates fidelity changes across physical capacity, readiness, and attention execution.This mixed-fidelity contract complements KV migration and reduction by treating fidelity as a runtime-managed execution state.Source: §2.2 · p. 3

3. ElasticKV Design and Implementation

ElasticKV extends paged serving with coordinated fidelity-aware memory management and execution. Its design makes compact KV capacity reclaimable, supports direct BASE attention, and manages demotion and promotion differently under pressure.

3. ElasticKV Design and Implementation

  • ElasticKV coordinates fidelity state and transitions in the control plane, pair-structured storage in the KV memory plane, and native or elastic dispatch in the GPU execution plane.Its implementation retains vLLM scheduling and uses KV-state metadata to guide dispatch.Source: §3 · pp. 3–4

3.1. Pair-Structured Elastic KV Layout

  • Two BASE logical blocks fit in one target-sized physical block, so demoting a TARGET pair frees one whole GPU block while preserving page-level regularity.Pairing changes physical organization while leaving each member a standard logical block; TARGET pairs retain the native paged layout.Source: §3.1 · pp. 4–5

3.2. Dual-mode Elastic Attention Backend

  • TARGET-only batches use the native attention path, while batches containing BASE use the elastic path, which consumes packed BASE values directly without materializing missing refinement bits.Common TARGET-BASE-TARGET sequences use compact boundary metadata; irregular sequences use an explicit pair-mode table and the general elastic path.Source: §3.2 · p. 5

3.3. Pressure-Aware Fidelity Management

  • Demotion is triggered proactively by shortage and batched to amortize small transitions; promotion is opportunistic, asynchronous, and delayed by a decaying reserve that protects recently freed capacity.BASE remains execution-ready until TARGET has been fully materialized.Source: §3.3 · pp. 5–6

3.4. Implementation

  • The vLLM implementation tracks fidelity, physical mappings, CPU backing, and transition status while preserving native scheduling; BASE remains executable during asynchronous refinement.Demotion is synchronous, and tensor-parallel ranks coordinate pair transitions to commit a consistent state.Source: §3.4 · pp. 6–7

4. Evaluation

ElasticKV is evaluated on LongBench quality and serving workloads spanning synthetic traffic, ShareGPT, and Mooncake traces, including a 70B deployment on AMD GPUs. Results show quality close to FP16 in practical BASE-exposure ranges and more stable serving under pressure, with benefits across model scale, GPU budgets, and context lengths.

4.2. Evaluation on Generation Quality

  • Across all three practical BASE-exposure caps, the 21-task LongBench average essentially matches FP16 on both models.For both models, 13 tasks reach at least 18% BASE exposure while subset-average scores remain within 0.1 points of FP16; Table 1 covers average scores over 21 tasks.Source: §4.2 · pp. 7–8, Table 1 · p. 7

4.3. Overall End-to-End Serving Performance

  • At the highest tested concurrency, ElasticKV achieves 3.8–4.0× lower mean TTFT and 9.1× lower P90 TTFT than Full-GPU across two models.It also achieves 2.0–2.6× lower mean TTFT and 2.5–4.7× lower P90 TTFT than Exact-offload; Figure 4 reports serving metrics across increasing concurrency.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4. Realistic Workloads and Runtime Adaptation

  • At the highest ShareGPT request rate, ElasticKV achieves 4.3–6.5× lower P90 TTFT than Full-GPU and 3.1–4.0× lower P90 TTFT than Exact-offload.Both baselines exceed 1,000 preemptions on each model, while ElasticKV remains preemption-free with 2.6% and 3.6% BASE exposure.Source: §4.4 · p. 9, Figure 5 · p. 9
  • Across three Mooncake trace windows, ElasticKV improves Eff@300 while remaining preemption-free and reduces mean waiting by 36% in the highest-load window.Figure 6 also shows offered input load, waiting requests, and resident BASE ratio over time in Window C.Source: §4.4 · p. 9, Figure 6 · p. 9

4.5. Large-Scale and Cross-Platform Generalization

  • On Llama-3.1-70B-Instruct across eight AMD MI355X GPUs, ElasticKV improves average Eff@300 by 34.1%, eliminates native preemptions, and reduces average drain time by 70.5%.Figure 7 reports these metrics for the TP2×DP4 deployment.Source: Figure 7 · p. 9, §4.5 · p. 10

4.6. Sensitivity Analysis

  • At the tightest tested GPU KV budgets, ElasticKV remains more stable with 13% BASE exposure on both models; at 32K input length, it reduces mean TTFT over Exact-offload by 21.7% and 21.8%.At 32K, BASE exposure is 6.3% and 15.9% for Llama-3.1-8B-Instruct and Qwen3-8B, respectively; Figure 8 covers budgets and context lengths.Source: §4.6 · p. 10, Figure 8 · p. 10

4.7. System Analysis

  • At maximum concurrency 38, ElasticKV adds 4.37 ms/step of measured critical-path work; asynchronous promotion is excluded.Synchronous demotion occurs on 11.1% of steps and contributes 2.82 ms/step averaged over the run; Table 2 covers ElasticKV-specific cost.Source: §4.7 · p. 10, Table 2 · p. 10
  • Under high pressure, eager promotion raises P90 TTFT from 2.8s to 4.4s, while disabling promotion nearly doubles BASE exposure.Forcing mixed-fidelity batches through the general elastic path increases P90 TPOT without changing BASE exposure; Table 3 reports the ablation.Source: §4.7 · p. 10, Table 3 · p. 10

5. Conclusion

ElasticKV allows requests to progress with an executable intermediate KV state under memory pressure, using a pair-structured layout, dual-mode attention, and pressure-aware fidelity control. The appendix describes how KV states are created, grow during decoding, and are reclaimed as requests progress or finish.

5. Conclusion

  • ElasticKV treats KV fidelity as a runtime-managed execution property, allowing progress with an intermediate state under memory pressure while preserving generation quality across evaluated settings.Its layout makes compact states reclaimable, its attention backend executes them directly, and its fidelity control manages changes dynamically.Source: §5 · p. 11

A. LLM Inference and KV-Cache Lifecycle

  • During prefill, a request materializes KV states; decoding appends states for each generated token, increasing the request’s memory footprint.Paged management maps logical KV blocks to fixed-size physical blocks and reclaims physical capacity at page granularity.Source: A. LLM Inference and KV-Cache Lifecycle · p. 16

B. Runtime Implementation Details

ElasticKV integrates pair-state tracking and scheduler hooks without changing vLLM’s native request and token scheduling policy. Its transition engine and attention backend manage TARGET and BASE states while preserving execution on BASE during promotion.

B.1. State and Scheduler Integration

  • The LogicalBlockStateTable tracks physical pair mappings and transition eligibility while preserving the serving stack’s standard logical-block namespace.It exposes demotable and promotable pairs; demotion excludes pairs without recoverable CPU backing and protected regions, then prioritizes non-urgent pairs.Source: §B.1 · p. 16
  • Scheduler hooks leave native request and token scheduling unchanged while probing shortages, submitting demotions, and issuing promotions after scheduling and allocation.BASE-ready requests remain execution-ready during promotion.Source: §B.1 · pp. 16–17

B.2. Transition Protocols

  • TARGET-to-BASE transitions synchronously pack pairs and release capacity only after completion, while BASE-to-TARGET refinement is asynchronous and leaves BASE executable in flight.Tensor-parallel transitions commit consistently across ranks.Source: §B.2 · p. 17

B.3. Elastic Attention Backend

  • The attention backend dispatches TARGET-only batches to native Triton and mixed batches to structured or general elastic paths.BASE loads its stored high-order byte into the native 16-bit representation with the unavailable low-order byte zero-filled, without materializing TARGET first.Source: §B.3 · pp. 17–18

C.1. Full LongBench Quality Results

Across LongBench, practical BASE-exposure caps keep average scores close to FP16, while forcing all KV execution in BASE lowers scores. A near-cap subset shows the close aggregate results also hold among tasks with substantial BASE exposure.

C.1. Full LongBench Quality Results

  • BASE exposure is pressure-induced, and each cap is an upper bound rather than a target, so selected runs may remain below the cap or never use BASE.The evaluation reports official task scores and observed BASE exposure for selected runs.Source: §C.1 · pp. 19–20
  • Across 21 tasks, average-score differences versus FP16 at 5%, 10%, and 20% caps were +0.17, +0.17, and +0.15 points for Llama-3.1-8B-Instruct and 0.00, −0.07, and −0.17 points for Qwen3-8B.All-BASE lowered average scores by 1.44 points and 1.72 points, respectively; the small cap-regime differences are not quality improvements.Source: §C.1 · p. 20
  • For the 13/21 tasks per model reaching at least 18% BASE exposure under the 20% cap, average scores were within 0.1 points of FP16.Llama-3.1-8B-Instruct scored 48.01 versus 48.07 under FP16; Qwen3-8B scored 45.37 versus 45.46.Source: §C.1 · p. 20

C.2. Additional ShareGPT Results

On ShareGPT, ElasticKV maintains more stable latency and higher output throughput as load rises, and avoids preemptions at the highest tested rate with little BASE exposure. Additional production-trace replay reports lower mean TTFT and waiting under lighter load.

C.2. Additional ShareGPT Results

  • As ShareGPT load rises past baseline preemption boundaries, ElasticKV maintains more stable latency and higher output-token throughput.The reported metrics include mean TTFT, mean and P90 TPOT, and output-token throughput.Source: §C.2 · p. 21
  • At 22 requests/s, ElasticKV had no native request preemptions on either model with 2.6% BASE exposure on Llama-3.1-8B-Instruct and 3.6% on Qwen3-8B.Full-GPU and Exact-offload recorded 1297 and 1392 preemptions on Llama, and 1574 and 1594 on Qwen.Source: §C.2 · p. 21

C.3.1. Trace and Replay Protocol

  • The replay preserves request order, lengths, and relative arrivals across three temporally separated 300-second Mooncake trace windows.A larger dilation factor means a lighter offered load.Source: §C.3.1 · p. 21

C.3.2. Mooncake Metrics

  • Mooncake effective-request ratio counts requests with TTFT below 30 seconds and TBT below the threshold, using τ=300ms.TBT is the mean of the slowest 10% of consecutive inter-token intervals.Source: §C.3.2 · p. 22

C.3.3. Lighter Production Trace Replay

  • With the lighter d=20 production-trace replay, ElasticKV had no preemptions across windows A, B, and C, while Exact-offload had 211, 69, and 212.Relative to Exact-offload, mean TTFT fell by 29.1%, 10.1%, and 23.4%, and mean waiting by 34.5%, 15.1%, and 27.4%, respectively.Source: §C.3.3 · p. 22, Figure 10 · p. 22

C.4. Trace Replay on AMD MI355X GPUs

On 8× AMD MI355X GPUs, ElasticKV improves effective-request outcomes and shortens drain time across three trace windows, with no native preemptions in the primary topology. A second topology shows similar behavior; the section also distinguishes mixed-fidelity execution from static FP8.

C.4. Trace Replay on AMD MI355X GPUs

  • The MI355X replay is a scale and cross-platform generalization test, not a direct cross-hardware speedup comparison.It differs from the single-A100 study in model scale, hardware, window duration, and arrival-time dilation.Source: §C.4 · p. 23
  • Across three MI355X trace windows, ElasticKV remained free of native preemptions while improving effective-request ratio and shortening drain time.Low BASE exposure in windows B and C shows benefits even when BASE is rarely visible to attention execution.Source: §C.4 · pp. 23–24
  • Under TP1×DP8, ElasticKV had only seven native preemption events total across the three-window replay and maintained favorable effective-request ratio and drain time versus baselines.This topology serves as a generalization check, not a comparison of TP and DP efficiency.Source: §C.4 · p. 24, Figure 11 · p. 24

C.5. Static FP8 as a Fixed-Fidelity Reference

  • Static FP8 changes the target representation, whereas ElasticKV retains a 16-bit TARGET and invokes lower-fidelity execution only when needed.The FP8 and ElasticKV results are not a backend-matched kernel comparison.Source: §C.5 · pp. 24–25, Table 10 · p. 24
  • The mixed-fidelity contract is orthogonal to TARGET precision, though this implementation uses a 16-bit TARGET and 8-bit BASE.The text gives a lower-precision TARGET with an additional lower-fidelity state as a possible extension.Source: §C.5 · p. 25

C.6.1. Profiling Setup and Methodology

The overhead profile uses a controlled high-pressure serving run and measures ElasticKV-specific critical-path work, including online control, synchronous demotion, and ElasticKV metadata.

C.6.1. Profiling Setup and Methodology

  • The profile uses Llama-3.1-8B-Instruct on one NVIDIA A100 80GB GPU with a 20GB KV budget, 600 requests, 4096-token inputs, 1024-token outputs, and concurrency 38.ElasticKV remains preemption-free at this operating point, with 14.7% average BASE exposure.Source: §C.6.1 · p. 25
  • The profile measures only ElasticKV-specific critical-path work across online control, synchronous demotion, and Elastic metadata.The breakdown is identified as Table 11's detailed profile.Source: §C.6.1 · p. 25, Table 11 · p. 25
  • Activity frequency and accumulated latency are normalized by 16,586 scheduling steps, while asynchronous CPU-to-GPU promotion transfer is excluded from critical-path cost.Host-side preparation and submission are included in online control.Source: §C.6.1 · p. 25

C.6.2. Detailed Runtime Overhead Breakdown

The profile attributes 4.365ms of critical-path work per scheduling step to ElasticKV-specific components; transitions are exercised and profiling has negligible measured serving impact.

C.6.2. Detailed Runtime Overhead Breakdown

  • Online control adds 0.32ms per scheduling step; demotion planning is its largest cost, while each of four other activities contributes less than 0.1ms/step.Those activities are shortage probing, promotion planning, restore-source preparation, and asynchronous promotion issue.Source: §C.6.2 · p. 25
  • 4.365ms of critical-path work per scheduling step comes from ElasticKV-specific components, including 2.815ms/step amortized synchronous demotion and 1.23ms/step of Elastic metadata.Synchronous demotion is invoked on 11.06% of scheduling steps and costs 25.44ms per invocation.Source: §C.6.2 · p. 26
  • All 1,835 demotion batches completed; 29,427 demoted pairs averaged 16.04 pairs per batch, near the configured minimum batch size of 16.The transition statistics are reported in Table 12.Source: §C.6.2 · p. 26, Table 12 · p. 26
  • Enabling profiling changed benchmark duration by 0.01%, mean TPOT by −0.03%, and output-token throughput by −0.01% in the paired check.Table 13 reports serving performance with profiling disabled and enabled.Source: §C.6.2 · p. 26

C.7. Ablation Settings

The ablation variants hold the workload and core ElasticKV components constant, changing one specified mechanism at a time.

C.7. Ablation Settings

  • All ablation variants use the same workload, pair-structured layout, BASE representation, transition engine, and serving configuration, changing only the stated mechanism.The setting matches the high-pressure runtime-overhead profile.Source: §C.7 · p. 26
  • Watermark demotion replaces shortage-aware triggering with a fixed free-capacity watermark, while eager promotion removes the protected reserve but retains eligibility and transfer-slack checks.Eager promotion sets the protected reserve to zero when computing the promotion budget.Source: §C.7 · p. 26
  • No promotion leaves demoted pairs in BASE until normal lifetime end, while the general elastic path routes mixed-fidelity batches through the general pair-mode implementation.TARGET-only batches retain the native Triton path in the general elastic path variant.Source: §C.7 · p. 26
Loading 2609.33536v1…