PaperlayerView exact revision on arXiv

Source-linked AI summary

PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding

Qiuyang Zhang, Kai Zhou, Kai Lu, Haocheng Lu, Jian Zhou, Yuanpeng Su, Kun Bao, Jiguang Wan, Fei Wu

arXiv:2609.34555v1cs.LG

TL;DR

Long-context decoding is constrained by KV-cache memory, and offloading shifts the bottleneck to variable, fragmented CPU–GPU recall I/O. PulseInfer combines interruptible layer-wise scheduling, adaptive offloading admission, and coalesced recall transfers. It improves decode throughput over SGLang and existing offloading systems while reducing TPOT and preserving near-lossless accuracy.

  • Problem

    Long-context KV caches limit GPU batch size, and offloading them shifts the bottleneck to recall I/O whose volume varies and whose headwise transfers are fragmented.

    Source: §1 · pp. 1–2

  • Method

    PulseInfer hides variable recall latency with interruptible layer-wise scheduling, adapts batch admission at runtime, and coalesces recalls with SoloHead and a gather-scatter engine.

    Source: §1 · p. 2

  • Results

    Up to 4.7× higher decode throughput than SGLang and 2.6× than the best existing offloading system, with up to 76% lower TPOT and near-lossless accuracy.

    Source: §1 · p. 3

  • Takeaways & Limitations

    PulseInfer demonstrates that explicitly scheduling, controlling, and coalescing recall I/O can improve decode throughput and GPU utilization while preserving near-lossless accuracy.

    Source: §7 · p. 13

Abstract

from arXiv · show

Long-context LLM serving is increasingly bottlenecked by decode, where large KV caches limit batch size and underutilize GPUs. Sparse KV cache offloading expands effective capacity by storing most historical KV blocks in CPU DRAM and recalling only selected blocks on demand. However, we find that existing offloading systems shift the bottleneck to CPU-GPU recall I/O: recall volume varies widely across layers, decode steps and requests, while headwise sparse selection fragments recalls into many small PCIe transfers. This paper presents PulseInfer, an I/O-centric sparse KV cache offloading system. PulseInfer hides variable recall latency with interruptible layer-wise scheduling, adapts offloading decisions with IO-Adaptive Offloading Admission, and coalesces fragmented transfers using SoloHead sparse selection and a gather-scatter I/O engine. Implemented on SGLang, PulseInfer improves decode throughput by up to 4.7x over SGLang and 2.6x over the best existing offloading baseline, while reducing TPOT by up to 76% and preserving near-lossless accuracy.

1 Introduction

Long-context decode is constrained by the GPU memory required for KV caches, while offloading shifts the bottleneck to variable, fragmented CPU–GPU recall I/O. PulseInfer schedules, adapts, and coalesces recall I/O, improving throughput and latency while preserving near-lossless accuracy.

1 Introduction

  • Longer inputs and outputs shift inference cost toward decode, where KV-cache memory limits batch size and degrades serving throughput.Decode can account for over 90% of end-to-end inference time.Source: §1 · p. 1
  • Offloading moves historical KV blocks to CPU DRAM, but per-layer recalls over PCIe can stall GPUs and offset its capacity benefit.Recall volume varies across layers, decode steps, requests, and batches, while headwise selection fragments transfers and reduces effective PCIe bandwidth.Source: §1 · p. 2
  • Interruptible layer-wise scheduling switches to normal batches during recall and resumes offload batches when I/O completes, hiding variable latency behind computation.It checks for I/O-completion interrupts at layer boundaries while preserving transformer decoding semantics.Source: §1 · p. 2
  • IO-Adaptive Offloading Admission uses a throughput model and runtime pruned binary-tree search to choose normal/offload batch assignments as workloads drift.The model uses overlap depth—the normal-batch layers needed to cover one recall-I/O round—to balance latency hiding against GPU memory pressure.Source: §1 · p. 2
  • SoloHead selects shared top-k blocks using the strongest retrieval head, while a gather-scatter engine combines scattered blocks into one large DMA transfer.The engine improves effective PCIe bandwidth by over 2× compared with per-block asynchronous copies; SoloHead can improve accuracy over headwise selection.Source: §1 · p. 2
  • Up to 4.7× higher decode throughput than SGLang and 2.6× than the best existing offloading system, with up to 76% lower TPOT and near-lossless accuracy.PulseInfer also sustains ~95% GPU utilization.Source: §1 · p. 3

2 Background

Decode attention uses sparse selection to reduce memory access, but long-context KV caches can still exceed GPU capacity. Sparse KV cache offloading stores most historical blocks in CPU memory and recalls selected missing blocks during decoding.

2.1 LLM Inference and Sparse Attention

  • During decode, each generated token attends to the preceding context, while KV-cache growth with sequence length and batch size makes long-context decoding memory-capacity-bound.This paper focuses on the decode instance in prefill-decode-disaggregated serving.Source: §2.1 · p. 3
  • Sparse attention selects only a subset of historical tokens or blocks, but methods such as Quest still assume the full KV cache remains on the GPU.Quest maintains compact per-head digests and lets each KV head independently select its top-k blocks for the query.Source: §2.1 · p. 3

2.2 Sparse KV Cache Offloading

  • Sparse KV cache offloading stores the full cache in CPU DRAM, retaining compact digests and hot blocks on GPU, then recalls selected blocks missing from GPU memory.A 128K-token Qwen-3 14B sequence requires 20 GB of KV cache; the recall set is the difference between selected top-k blocks and GPU-resident hot blocks.Source: §2.2 · p. 3
  • Existing offloading systems use synchronous recall in ArkVale, approximate retrieval with product quantization in PQCache, and one-layer-ahead prefetching in InfiniGen.InfiniGen predicts the next layer’s important blocks from an approximation of its query derived using the current layer’s hidden state.Source: §2.2 · pp. 3–4

3 Motivation

Sparse KV cache offloading can shift GPU memory pressure to recall-I/O stalls. Those stalls are difficult to hide because recall volume varies with layers and workloads, while headwise selection creates small, scattered transfers.

3.1 Offloading Shifts the Bottleneck to Recall I/O

  • Offloading systems make batches wait for recall I/O; short requests under 8K tokens see a 5× TPOT increase when co-scheduled with long requests, despite needing little or no recalled data.The comparison is against isolated, non-offloaded full-attention execution.Source: §3.1 · p. 4

3.2 Dynamic Recall I/O Defeats Fixed Prefetching

  • On Qwen-3 14B, recalled-token distributions are long-tailed and differ across layers, with median recall varying by up to 5×, making fixed one-layer prefetching unreliable.Query changes across decode steps make the non-overlapping selected tokens difficult to predict.Source: §3.2 · p. 4

3.3 Workload Drift Breaks Static Offloading Policies

  • ServeGen input-length distributions shift over time, and existing offloading systems that beat full attention on uniform 32K inputs can fall below it on the real trace.The shifting median and spread change the short-to-long request mix during serving.Source: §3.3 · pp. 4–5, Figure 7 · p. 5

3.4 Headwise Selection Fragments Recall I/O

  • Headwise selection fragments recall into small scattered transfers: an 8 KB transfer reaches about 6 GB/s effective bandwidth versus PCIe 5.0×16’s 64 GB/s theoretical peak.Per-head digests also occupy GPU memory and reduce the memory savings from offloading.Source: §3.4 · p. 5, Figure 8 · p. 5

4 Design and Implementation

PulseInfer treats recall I/O as a first-class schedulable event, with mechanisms to hide variable latency, adapt offloading decisions, and coalesce transfers. Its design combines interruptible layer-wise scheduling, IOAA, SoloHead, and gather-scatter transfer.

4.1 Design Overview

  • PulseInfer organizes recall-I/O management around three goals: hiding latency, adapting offloading decisions, and coalescing transfers.The system treats recall I/O as a first-class schedulable event.Source: §4.1 · p. 5

4.2 Hide: Interruptible Layer-wise Scheduling

  • IRQ separates normal and offload requests, switches at layer boundaries, and adaptively runs normal-batch layers during recall waits.Normal requests keep their KV caches on the GPU; short requests are isolated from offload-batch I/O.Source: §4.2 · p. 6, §4.2.1 · p. 6

4.3 Control: I/O-Adaptive Offloading Admission

  • IOAA searches normal/offload assignments using an expected-throughput model, with overlap depth estimated from offline profiles.It prunes memory-infeasible and length-monotonicity-invalid branches; with 30 waiting requests, it visits 3.16 nodes on average.Source: §4.3 · p. 7, §4.3.2 · p. 7, §4.3.1 · p. 7

4.4 Coalesce: SoloHead and Gather-Scatter Transfer

  • SoloHead uses one strong retrieval KV head per layer to select top-k blocks shared across all KV heads, eliminating per-head digest storage.The shared selection converts head-wise recalls to token-wise transfers while preserving the retrieval behavior needed for sparse attention.Source: §4.4.1 · p. 8
  • Across Qwen-3 14B layers, SoloHead’s logit difference is close to headwise and average selection, while reversed SoloHead diverges most.MAM varies substantially across heads and remains stable across Qasper and QMSum, suggesting head roles are largely model-structural rather than input-specific.Source: §4.4.1 · pp. 8–9
  • For transfers above 512 blocks, gather-scatter achieves over 2× the effective bandwidth of threaded cudaMemcpyAsync with block size 16.It gathers scattered DRAM blocks into a contiguous pinned CPU buffer, transfers them in one large DMA, then scatters them into the GPU KV cache.Source: §4.4.2 · p. 9

4.5 Implementation

  • PulseInfer is implemented on SGLang with a two-level scheduler and zero-sync I/O submission.A CPU polling thread submits recall tasks asynchronously so the main inference thread can continue enqueuing normal-batch kernels.Source: §4.5 · p. 9

5 Evaluation

Evaluation across three LLMs and synthetic and realistic workloads finds that PulseInfer improves decode throughput and TPOT over existing offloading systems. Performance analysis examines scheduling, admission, transfer, capacity, and accuracy.

5.1 End-to-End Performance

  • 1.5x–4.7× over Vanilla and 1.1×–2.6× over the best existing offloading baseline across models and workloads; gains increase with input length.On Qwen-3 14B, throughput falls from 324 tok/s to 81 tok/s for Vanilla as mean input length grows from 16K to 64K, versus 488 tok/s to 380 tok/s for PulseInfer.Source: Figure 17 · p. 10, §5.1.1 · p. 10
  • 16.0%–76.4% lower mean TPOT than InfiniGen across models and workloads; PulseInfer's mean TPOT is 2.2× Vanilla's.Vanilla avoids recall I/O and normal/offload batch switching, while PulseInfer trades higher TPOT for greater throughput.Source: Figure 18 · p. 10, §5.1.2 · p. 11

5.2 Performance Analysis

  • IRQ overlaps recall I/O with normal GPU execution, largely eliminating recall-induced idle time and sustaining GPU utilization close to Vanilla.Synchronous baselines wait for recall, while InfiniGen's fixed prefetch window leaves idle periods under dynamic recall demand.Source: Figure 20 · p. 11, Figure 21 · p. 11, §5.2 · pp. 11–12
  • 4.7× larger total batch size than Vanilla, with both the normal and offload batches individually exceeding Vanilla's batch size.Offloading long requests frees GPU capacity for more short requests in the normal batch while expanding serving capacity with the offload batch.Source: Figure 22 · p. 11, §5.2 · p. 12
  • All ablated components contribute to throughput improvement, with their combination reaching over 3× Vanilla's throughput.The components address recall-transfer efficiency, adaptive I/O hiding, scheduling overhead, and sparse-selection coalescing.Source: Figure 23 · p. 11, §5.2 · p. 12
  • IOAA achieves the highest throughput on both ServeGen and Mooncake traces; the best fixed admission threshold differs between traces.IOAA searches for the offloading admission decision at runtime to adapt to workload-dependent memory pressure and recall-I/O demand.Source: Figure 24 · p. 11, §5.2 · p. 12
  • PulseInfer throughput continues to improve with GPU KV-cache capacity and remains faster than Vanilla at 0.4× PCIe bandwidth.At 0.4× bandwidth, InfiniGen TPOT increases by 2.2×, compared with PulseInfer's 1.7× increase.Source: Figure 25 · p. 11, Figure 26 · p. 11, §5.2 · p. 12

5.3 Accuracy

  • PulseInfer achieves near-lossless accuracy across benchmarks and slightly outperforms full attention on the two Qwen-3 models.Selecting the lowest-MAM head instead causes a substantial accuracy drop.Source: Table 1 · p. 12, §5.3 · p. 13

6 Related Work

Related work addresses KV-cache offloading, fine-grained serving schedules, and head-aware sparse attention. PulseInfer applies an I/O-centric approach to sparse recalls across these areas.

KV cache offloading

  • ArkVale, InfiniGen, and PQCache optimize KV selection, indexing, caching, or prefetching, while PulseInfer targets dynamic and fragmented recall I/O.PulseInfer jointly hides, controls, and coalesces sparse KV recalls.Source: §6 · p. 13

Fine-grained scheduling for LLM serving

  • Sarathi-Serve, NanoFlow, and NEO refine scheduling for batching, prefill–decode overlap, or CPU–GPU load balance; PulseInfer schedules at layer boundaries around sparse-recall completion.Attention computation remains on the GPU.Source: §6 · p. 13

Head-aware sparse attention

  • MInference, DuoAttention, and LServe use head specialization for attention efficiency, whereas PulseInfer uses one retrieval-oriented head per layer to guide block selection for all KV heads.This applies head specialization to coalescing KV-cache recall I/O.Source: §6 · p. 13

7 Conclusion

PulseInfer is an I/O-centric sparse KV-cache offloading system for long-context LLM decoding. Its evaluation reports higher decode throughput and GPU utilization while preserving near-lossless accuracy.

7 Conclusion

  • PulseInfer is an I/O-centric sparse KV-cache offloading system for long-context LLM decoding.Source: §7 · p. 13
  • It combines interruptible layer-wise scheduling, IO-Adaptive Offloading Admission, and SoloHead sparse selection to schedule, control, and coalesce recall I/O.Source: §7 · p. 13
  • Evaluation shows improved decode throughput and GPU utilization while preserving near-lossless accuracy.Source: §7 · p. 13

Related papers

Loading 2609.34555v1…