PaperlayerView exact revision on arXiv

Source-linked AI summary

DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory

Xuan Truong Nguyen, Tien Son Pham, Tuan Duc Chu, Wookeun Jung, Thanh Tuan Dao

arXiv:2609.34380v1cs.DCcs.AIcs.ETcs.PFcs.SE

TL;DR

Existing serving systems keep weight and KV-cache allocations fixed despite variable cache demand, while nested multi-precision models offer a way to reuse part of the weight footprint. DPS combines dual-precision execution with SUM to reassign that memory under KV pressure, improving throughput and effective pass@1 over Static FP16.

  • Problem

    Existing systems statically allocate weight and KV-cache memory, although variable KV demand can limit batch size and trigger preemption.

    Source: §I · p. 2

  • Method

    DPS uses SUM to let shared physical memory back residual weights or KV-cache blocks, switching between full-precision and nested lower-precision execution as KV pressure changes.

    Source: §I · p. 2

  • Results

    2.1–3.3× sustained throughput and up to +41 pp effective pass@1 over Static FP16, while preserving FP16-class accuracy.

    Source: §I · p. 2

  • Takeaways & Limitations

    DPS can extend existing serving systems such as vLLM while maintaining dynamic, page-based KV-cache management.

    Source: §VII · p. 12

    DPS focuses on coarse-grained nested models; extending it to layer-wise weight tensors and investigating KV-cache fragmentation remain future work.

    Source: §VI · p. 12

Abstract

from arXiv · show

Existing LLM serving systems virtualize and optimize KV-cache memory, but treat model-weight memory as fixed throughout execution. Recent work on multi-precision model representations challenges this design by allowing a single stored model to support both full-accuracy and lower-precision execution, making the effective weight footprint runtime-dependent. This creates an opportunity under bursty workloads, where temporary spikes in KV-cache demand often determine throughput and SLO compliance. We present DPS, a dual-precision LLM serving system that turns weight memory into an elastic resource: under normal load, DPS serves the full-accuracy model; under KV pressure, it switches to a nested, lower-precision variant and repurposes unused weight memory for KV cache blocks. DPS is built on Semi-Unified Memory (SUM), which partitions the weight region into a persistent lower-precision sub-region and a shared region that alternates between residual weight tensors and KV-cache blocks, preserving compatibility with paged KV-cache management. We implement DPS on top of vLLM and evaluate it across both dense and MoE models and various production workload traces. Our results show that \sysname improves sustained throughput by $2.1$--$3.3\times$ and effective pass@1 by up to $+41$\,pp over Static FP16, while preserving FP16-class accuracy.

I. INTRODUCTION

LLM serving systems statically partition GPU memory even though KV-cache demand varies, so bursts can constrain batch size and trigger request preemption. DPS addresses this by making weight memory elastic through dual-precision execution and SUM, reporting higher throughput and effective pass@1 than Static FP16.

I. INTRODUCTION

  • Static weight and KV-cache allocations cannot adapt at runtime, and insufficient KV capacity limits batch size and can force request preemption.For a 30B model on an 80GB H100, weights use 71.2% of GPU memory and KV cache uses 15.5%.Source: §I · pp. 1–2
  • FP8 can reduce model memory, allowing the residual precision footprint to serve as shared capacity for weights or KV cache.For the cited 30B model on an 80GB GPU, FP8 uses 36.4% of memory, leaving 50.8% for KV cache.Source: §I · p. 2
  • DPS serves FP16 under normal load, switches to nested FP8 under KV pressure, and reuses freed weight memory for KV blocks.When pressure subsides, it restores weights to return gradually to full mode.Source: §I · p. 2
  • SUM partitions weight memory into a persistent lower-precision region and a shared region for residual weights or KV blocks, retaining compatibility with vLLM’s default KV allocation.The implementation uses CUDA Virtual Memory Management.Source: §I · p. 2
  • FP16-to-FP8 switching needs no data movement, while the reverse requires host-to-device communication that a background restore overlaps with FP8 computation.A SUM-aware scheduler monitors KV pressure and chooses when to switch to prevent cache thrashing and mitigate accuracy loss.Source: §I · p. 2
  • DPS improves sustained throughput by 2.1–3.3× and effective pass@1 by up to +41 pp over Static FP16 while preserving FP16-class accuracy.The evaluation covers dense and MoE models and production workload traces.Source: §I · p. 2

A. LLM architecture and LLM serving systems

LLM serving systems use batching and KV-cache management to improve efficiency, but their memory partitions remain fixed as workloads vary. FP8 offers a smaller footprint and hardware-supported arithmetic, while decode remains memory-bound.

A. LLM architecture and LLM serving systems

  • Serving systems use continuous batching and paged KV-cache management, but most statically partition weight and cache memory at initialization.Those fixed partitions limit adjustment to workload variation.Source: A. LLM architecture and LLM serving… · p. 2

B. FP16/FP8 for LLMs

  • Figure 2 identifies workload fluctuations, TTFT/TPOT and SLO, and KV-cache usage and preemptions as serving signals.The caption names these signals but does not provide their values.Source: Figure 2 · p. 3
  • FP16 and FP8 are floating-point formats used for LLMs; FP8 can maintain high accuracy with small degradation while reducing bandwidth requirements by up to 2×.FP16 is E5M10, while FP8 formats include E5M2 and E4M3.Source: B. FP16/FP8 for LLMs · pp. 2–3

C. GPU Memory Characteristics

  • Decode is dominated by memory-bound operations requiring frequent HBM access, making memory use and data movement central to system performance.Prefill instead benefits from compute-intensive operations that use GPU tensor cores efficiently.Source: C. GPU Memory Characteristics · p. 3

III. MOTIVATION

LLM workloads fluctuate rapidly, and transient bursts can raise latency and KV-cache pressure enough to cause preemptions. DPS uses nested dual precision to expand cache capacity during pressure while retaining full-mode behavior when demand permits.

A. Challenge: Dynamic Load in LLM Serving

  • Production request rates fluctuate nearly fivefold at second-level frequency, while high KV pressure is interleaved with lower-pressure periods.The cited traces include Microsoft Azure LLM services and BurstGPT.Source: A. Challenge: Dynamic Load in LLM… · p. 3
  • Bursts can sharply increase TTFT and violate SLOs when memory cannot schedule prefills or continue decoding; long-context requests exacerbate the pressure.When traffic exceeds capacity, systems may preempt requests as KV cache approaches full utilization.Source: A. Challenge: Dynamic Load in LLM… · p. 3
  • Preempting long-context requests requires recomputing or swapping their KV cache, with cost proportional to input context length.For a coding agent processing a 64K-token repository context, preemption can add seconds of latency that generally exceed SLO budgets.Source: A. Challenge: Dynamic Load in LLM… · pp. 3–4

B. KV Cache Pressure and Preemption Mitigation via Dual-Precision Serving (DPS)

  • For a 13B model on an A100-40G, nested FP8/FP16 can remap 13 GB of lower FP8 weights to extend a 12 GB KV cache by 108.33%.The design must preserve page-based KV management and make restoration from fast to full mode smooth.Source: B. KV Cache Pressure and Preemption… · p. 4

IV. METHOD

DPS treats model-weight memory as elastic rather than wholly persistent. SUM lets a shared physical region serve residual weights in full mode or KV blocks in fast mode, enabling nested FP16/FP8 switching.

A. Design Methodology

  • Figure 3 depicts FP16, FP8, and SUM memory layouts, while Figure 4 depicts SUM mapping and weight-to-KV and KV-to-weight transitions.The captions identify the layouts and transitions but do not specify their mechanics.Source: Figure 3 · p. 4, Figure 4 · p. 5
  • A nested FP16 model stores lower-precision weights persistently and residual weights in a shared region, which can be freed for KV blocks during FP8 execution.This arrangement supports quick switching between FP16 and FP8 inference.Source: A. Design Methodology · p. 4
  • SUM introduces a shared physical region that dynamically backs either residual weights or the KV-cache pool.The design is presented as an abstraction directly integrable into systems such as vLLM.Source: A. Design Methodology · p. 4

B. Mapper: Semi-Unified Memory Abstraction

SUM makes weight memory elastic by sharing GPU memory between residual weights and KV-cache blocks. DPS uses that memory to switch between full- and lower-precision execution while preserving paged KV-cache management.

B. Mapper: Semi-Unified Memory Abstraction

  • SUM partitions memory into a persistent lower-precision region and a shared region, with a CPU residual backup and a manager coordinating memory bindings.The shared region can hold residual weights or KV-cache blocks.Source: B. Mapper: Semi-Unified Memory… · p. 5
  • The SUM Manager reallocates shared units between residual weights and KV blocks, then restores evicted weights in the background while permanent KV mappings protect in-flight reads.The residual backup supplies data for restoration; background work keeps VMM operations off the allocation and deallocation hot path.Source: B. Mapper: Semi-Unified Memory… · pp. 5–6

C. Scheduler: Dual-Precision Execution

  • Full-mode passes combine persistent FP8 weights with shared residuals; fast-mode passes use only FP8 weights while shared memory may hold KV data.The manager changes precision mode only between forward passes.Source: C. Scheduler: Dual-Precision Execution · p. 6
  • DPS switches to FP8 when KV pressure requires residual memory for KV blocks, using nested reconstruction kernels and per-layer FP8 encodings selected from weight statistics.The encoding extension supports E5M2 for layers outside E4M3's range.Source: C. Scheduler: Dual-Precision Execution · p. 6
  • Restoration reloads residual weights from CPU to GPU on a nonblocking copy stream, chunking runtime transfers to let other H2D traffic interleave.The reported fixed chunk size is 4 MiB; precision changes take effect at scheduling-step boundaries, so requests continue without recomputation.Source: C. Scheduler: Dual-Precision Execution · pp. 6–7

D. KV-Cache-Pressure-aware DPS

DPS allocates and frees KV blocks through SUM, switching to lower precision when KV demand needs residual-weight memory. Its switching policy is analyzed as a discounted Markov decision process.

D. KV-Cache-Pressure-aware DPS

  • When no KV unit has free blocks, allocation uses a free unit first, then a residual-backed unit, activating fast mode and marking its residual mapping stale.SUM tracks FREE, BACKED, AVAIL, FULL, and STALE states to manage these transitions.Source: D. KV-Cache-Pressure-aware DPS · p. 7
  • Freeing KV blocks moves units between FULL, AVAIL, and FREE; when a unit becomes free and evicted weights await restoration, the algorithm wakes weight restoration.This makes freed shared-region capacity available for residual weights again.Source: D. KV-Cache-Pressure-aware DPS · p. 7
  • The analysis models switching with KV occupancy and precision mode as state, stay-or-switch actions, and per-step latency, accuracy, and switching costs.The stated monotonicity condition yields a submodular optimal value function and a policy-search bound; threshold miscalibration adds per-step cost.Source: D. KV-Cache-Pressure-aware DPS · p. 8

V. EVALUATION

The evaluation methodology uses vLLM on three MoE models and replays accelerated production traces. It defines serving capacity using a joint latency SLO and also measures effective pass@1.

A. Methodology

  • The evaluation runs three MoE models on H100 80GB GPUs, using tensor parallelism when a model exceeds one device.The models are Phi-3.5-MoE, Qwen3-30B-A3B, and GLM-4.7-Flash; Phi-3.5-MoE uses TP=2.Source: A. Methodology · p. 8
  • The workload replays contiguous 2,000-request windows from BurstGPT and Azure traces at 200× arrival rates, comparing static-precision and KV-pressure-relief baselines.Baselines include BF16, static FP8, NestedFP, KV-OFFLOAD, KV-QUANT, and WEIGHT-OFFLOAD.Source: A. Methodology · p. 8
  • A request meets the joint SLO when maximum intra-request TBT is at most 5 × Tbase and TTFT is at most 2,000 ms; capacity requires at least 90% of requests to comply.Effective pass@1 is SLO attainment multiplied by pass@1 among served requests; requests missing the SLO count as zero.Source: A. Methodology · p. 8

B. Experimental Results

DPS preserves BF16-level offline accuracy while adapting to KV pressure, and outperforms Static FP16 in sustained throughput and effective pass@1 across the tested regimes.

1) Accuracy:

  • Both DPS modes preserve BF16 accuracy, with mean benchmark deviations of +0.25, −0.27, and −1.15 pp in full mode and +0.25, +0.26, and −0.17 pp in fast mode for Phi, Qwen, and GLM.Static FP8 drops −2.88 pp on Phi-3.5-MoE on average, driven by IFEval; DPS avoids this collapse through partial FP16 retention.Source: 1) Accuracy · p. 8, Table I · p. 9

2) Performance and Effective Pass@1:

  • On underloaded Phi-3.5-MoE, all seven methods maintain at least 97% SLO attainment through 8.5 req/s, and DPS remains within the 19.1–21.0% effective pass@1 envelope.With no KV saturation or mode transitions, DPS matches Static FP16 on operational metrics.Source: 2) Performance and Effective Pass@1 · pp. 8–9
  • At 4.5 req/s on Qwen3-30B-A3B, DPS reaches 56.4% effective pass@1 versus Static FP16’s 15.2%, a +41.2 pp advantage.Static FP16’s unconditional pass@1 remains about 57.4%; its effective-score collapse reflects SLO violations under limited KV capacity.Source: 2) Performance and Effective Pass@1 · p. 9
  • On GLM-4.7-Flash, DPS sustains 35.2% effective pass@1 at 4.5 req/s with full SLO compliance, a +9.5 pp gain over Static FP16.DPS sustains 3.78 RPS at 6.0 req/s versus Static FP16’s 1.80 RPS at 2.2 req/s, a 2.1× operating-range extension.Source: 2) Performance and Effective Pass@1 · p. 9
  • Under KV pressure, DPS extends the operating range by 2.1–3.3× and wins on effective pass@1 across the tested range, while no static configuration wins across regimes.Static FP8 matches DPS on throughput and SLO compliance but has permanent accuracy loss; Static FP16 performs well at low load but collapses under bursts.Source: 2) Performance and Effective Pass@1 · p. 10, Figure 6 · p. 10

3) DPS Characterization:

During representative bursty windows, DPS reclaims residual-weight memory for KV blocks and switches precision as pressure rises and subsides. This keeps TTFT below the SLO during steady serving, unlike Static FP16.

3) DPS Characterization:

  • DPS occupancy oscillates with bursts—0 to 0.93 on Qwen and 0 to 0.54 on GLM—while Static FP16 remains at its KV budget and can reach 170 s and 60 s mean TTFT, respectively.DPS enters performance mode for 79.2% of the Qwen trace and 47.1% of the GLM trace; during steady serving, its TTFT stays below the 2 s SLO.Source: 3) DPS Characterization · p. 11
  • At 4.5 req/s on Qwen3-30B-A3B and 6.0 req/s on GLM-4.7-Flash, DPS evicts up to 40 of 40 residual units on Qwen and 14 of 40 on GLM, reflecting GLM’s smaller per-request KV footprint.Source: 3) DPS Characterization · p. 11
  • DPS turns transient KV spikes into brief precision excursions, whereas fixed Static FP16 allocation cannot enlarge the KV budget and suffers a prolonged throughput collapse.The reported operating-range extension is 2.7–4.5×, with up to +41 pp effective pass@1.Source: 3) DPS Characterization · p. 11

4) Mode Switching Overhead:

Under bursty traces, mode switching is infrequent on two models, while restoring FP16 takes longer when more residual units were evicted. Background restoration overlaps with decode, leaving bounded PCIe contention as its runtime cost.

4) Mode Switching Overhead:

  • Two full →perf and two perf →full transitions occur on Qwen3-30B-A3B and GLM-4.7-Flash across each trace, while Phi-3.5-MoE remains in full-quality mode.The threshold-with-hysteresis policy consolidates bursts into a few long mode-windows; Phi-3.5-MoE has no KV pressure or transitions.Source: 4) Mode Switching Overhead · p. 11
  • Table II summarizes mode-switching activity under BurstGPT window traces, with restore latency reported as the per-cycle mean weight H2D wall time.The background restore worker overlaps the transfer with ongoing decode; its stated runtime cost is bounded PCIe contention, with negligible user-visible impact.Source: Table II · p. 11, 4) Mode Switching Overhead · p. 11
  • 8.5 s on GLM and 24.7 s on Qwen are the average end-to-end restore latencies, with 14/40 and 40/40 residual units transferred, respectively.The per-unit costs are 607 and 616 ms; restoration scales with evicted units rather than total model size.Source: 4) Mode Switching Overhead · p. 11

VI. RELATED WORKS AND DISCUSSION

The discussion situates DPS among memory-management and offloading approaches, and identifies fragmentation and coarse-grained nested precision as boundaries for future work. The conclusion presents SUM and asymmetric dual-precision switching as the basis for elastic weight memory while retaining dynamic KV-cache management.

VI. RELATED WORKS AND DISCUSSION

  • DPS preserves paged KV-cache management, while related approaches target memory sharing across vLLM engines or serving quantized and FP16 models.Prior offloading work hides host-to-device transfers during GPU computation, including through batched layer execution and assumed overlap [54].Source: §VI · p. 11
  • Page-based KV management may fragment memory, and DPS currently uses coarse-grained nested FP8-FP16 models to expand page-based KV cache.The authors identify fragmentation, broader dynamic-network support, and layer-wise weight tensors as future-work directions.Source: §VI · p. 12

VII. CONCLUSION

  • DPS combines SUM to mitigate KV-cache pressure and possibly reduce preemptions with asymmetric switching based on nested models.It repurposes unused weight memory for KV blocks and can extend existing serving systems such as vLLM while maintaining dynamic KV-cache management.Source: §VII · p. 12
Loading 2609.34380v1…