DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving

1 Introduction

• CPU-resident requests advance only when GPU-controlled pipelines expose suitable overlap opportunities, so available host memory and CPU compute do not ensure sustained decoding.NEO’s safeguards keep CPU attention from extending the GPU critical path but make CPU progress conditional on GPU-side scheduling.Source: §1 · p. 2
• DualPace assigns prefills and GPU-resident decodes to a GPU engine and gives CPU-resident decodes their own queues, scheduling clock, and batches.Large host-memory capacity determines how many requests can remain CPU-resident, while AMX operators and host PagedAttention turn that residency into decode throughput.Source: §1 · p. 2
• CPU-owned batches keep CPU scheduling control while selected dense stages run locally or through GPU Dense Assist, guided by hardware-specific batching and routing thresholds.Offline profiling selects CPU-local execution or GPU assistance according to the latency–throughput tradeoff.Source: §1 · p. 2
• In decode-oriented scenarios, DualPace improves throughput by 6.9–43.2% over NEO and 11.5–90.6% over vLLM while maintaining the same level of latency.It also preserves NEO’s P99 TTFT advantage over vLLM while sustaining 7.3–50.5% higher request rates than NEO at the same P99 ITL.Source: §1 · p. 2

2.1 LLM Inference and Execution Boundaries

• Prefill processes [B, L, H] inputs, while each decode step processes [B, 1, H] and attends to B KV histories; these stages define DualPace’s placement boundaries.The execution partition includes a pre-layer, transformer layers split into pre-attention, attention, and post-attention, and a post-layer.Source: §2.1 · pp. 2–3

2.2 Host Memory as an Active Decode Tier

• CPU attention keeps historical KV local and exchanges Θ(BH) activations rather than transferring Θ(BLH) KV data for GPU attention.Decode attention is memory-bandwidth-bound; CPU PagedAttention is comparable at B=1 and remains within roughly one order of magnitude for larger tested configurations.Source: §2.2 · p. 3

2.3 AMX Expands the CPU Compute Envelope

• Granite Rapids reaches 40 TFLOPS, up to 9× AVX512 throughput; for packed QKV and FFN gate/up projections across 1–1,024 tokens, AVX512 trails RTX 4090 by nearly two orders of magnitude, while AMX narrows the gap to roughly one order.Source: §2.3 · p. 3

3 Performance Bottlenecks of GPU-Gated CPU Offloading

• The section identifies GPU-gated CPU offloading as the bottleneck and uses NEO as a representative state-of-the-art open-source system for examining its admission gates.NEO builds on local-CPU partial offloading, asymmetric pipelining, and per-iteration load-aware scheduling.Source: §3 · p. 3, §3.1 · p. 3

3.1 NEO’s GPU-Gated Asymmetric Pipeline

• NEO admits CPU decode work only when GPU work supplies enough overlap and the predicted request rate exceeds GPU-only execution.The conditions require sub-batch linear work and GPU attention to hide CPU attention; requests are removed when predicted margins are negative.Source: §3.1 · pp. 3–4

3.2 Intermittent CPU-Resident Decode Progress

• After roughly 600 seconds, CPU-resident tokens plateau even as NEO’s total generated-token curve continues increasing.The CPU-resident curve advances in bursts separated by repeated stalls, including a representative multi-second pause.Source: §3.2 · p. 4
• As prefill availability and KV lengths vary, overlap conditions switch between feasible and infeasible; once prefills drain, decode-dominated iterations lead to a sustained plateau.Prefills contribute many prompt-token rows to GPU linear work, while each decode request contributes only one current-token row.Source: §3.2 · p. 4

4.1 Residency-Aware Dual-Engine Scheduling

• The GPU engine owns prefills and GPU-resident decodes, while the CPU engine independently batches CPU-resident decodes using separate queues and scheduling clocks without a global iteration barrier.Requests enter through the GPU engine; when HBM cannot retain a prefill’s KV cache, the GPU can reserve CPU blocks and transfer K/V states asynchronously.Source: §4 · p. 4, §4.1 · p. 5
• DualPace coordinates finite HBM and host-DRAM capacity by minimizing request migration and tracking CPU KV obligations, so independent scheduling does not treat host memory as an unbounded spill buffer.Swap-in admission accounts for a request’s finish-peak KV footprint; swap-out is triggered when HBM cannot accommodate next-token KV growth.Source: §4.1 · p. 5

4.2 Stage-Boundary Cooperation

• The CPU retains ownership and scheduling authority while GPU Dense Assist executes selected dense stages at explicit stage boundaries, without moving the request’s KV cache or entering the GPU scheduler’s queues.The CPU executes the pre-layer, pre-attention, and attention locally; the GPU worker services ready assist requests between non-preemptive GPU-main stages.Source: §4.2 · p. 6

4.3 Profile-Guided Hardware Configuration

• Offline profiling derives post-attention routing thresholds and GPU batched-token and CPU decode batch limits, accounting for both CPU-visible progress and GPU-main service capacity consumed by assistance.The profiler trades off CPU-local execution against assist costs and GPU disturbance when selecting routes.Source: §4.3 · pp. 6–7
• On the profiled RTX 4090 configuration, thresholds (32, 16) cost 0.070% more than the exact minimum; the selected GPU batched-token limit is 1,024 and the CPU decode batch limit is 256 requests.The thresholds assign Full-CPU to batches up to 16, Split-Final to batches 17–32, and Fused-Assist to larger batches; the CPU limit is advisory, and latency-oriented deployments may choose a smaller batch.Source: §4.3 · pp. 7–8

• An LLMEngine launches separate asynchronous GPU-engine, CPU-engine, and completion tasks; each engine schedules and applies its own batch results without a shared iteration barrier.GPU and CPU workers are Ray actors, while Ray RPCs are used for setup and teardown rather than per-stage serving.Source: §5 · p. 8
• Fixed-layout shared-memory blocks carry engine and Dense Assist control state, while shared pools hold CPU KV and assist activations for explicit transfers and a CPU-Ready → GPU-Running → GPU-Done handshake.Prefill-to-CPU KV copies use a dedicated CUDA stream and complete layer by layer.Source: §5 · p. 8
• GPU execution uses BF16 Llama kernels and FlashInfer attention, while custom oneDNN CPU dense operators and an AVX512-BF16/OpenMP PagedAttention kernel operate over the shared KV pool.Tensor parallelism uses NCCL for GPU ranks and Gloo for CPU ranks, with CPU KV and assist buffers rank-sharded and NUMA-local.Source: §5 · p. 8

6.1 Experimental Setup

• The evaluation uses three model–server configurations, the first 2,000 requests from Azure Conversation and Code traces, and vLLM and NEO baselines.vLLM is configured without host KV offloading and with chunked prefill; prefix caching is disabled. Each CPU allocation is confined to one NUMA node.Source: §6.1 · p. 8
• Sustainable request rate is measured by interpolating where a metric reaches its fixed target, without extrapolating beyond the measured request-rate sweep.Main-figure targets are 1 s/token normalized latency, 5 s P99 TTFT, and 1.5 s P99 ITL.Source: §6.1 · p. 9

6.2 Sustainable Request Rate under Load

• At the 1 s/token normalized-latency target, DualPace sustains 6.9–43.2% higher request rates than NEO and 11.5–90.6% higher rates than vLLM across three Azure-Conversation configurations.The gains are 43.2%, 6.9%, and 18.1% over NEO, and 90.6%, 11.5%, and 23.8% over vLLM for Llama-3.1-8B, Qwen3-14B, and Qwen3-32B, respectively.Source: §6.2 · p. 9
• DualPace is most effective when GPU KV pressure coincides with sustained decode work; on Azure-Code, long prompts and short outputs leave GPU prefill dominant and limit decode-tier benefits.All three systems execute prefills on the GPU, and DualPace remains ahead of NEO on Azure-Code while the GPU-only baseline is favored in the Qwen3 configurations.Source: §6.2 · p. 9

6.3 Tail TTFT and ITL

• At the 1.5 s P99 ITL target, DualPace sustains 50.5%, 15.2%, and 7.3% higher request rates than NEO across the three configurations while remaining close to NEO at the 5 s P99 TTFT target.NEO’s CPU-resident requests advance only when the GPU pipeline exposes a suitable overlap window; DualPace gives them an independent CPU scheduling clock and execution budget.Source: §6.3 · pp. 9–10

6.4 Sensitivity Studies

• CPU TP = 2 raises Llama-3.1-8B’s sustainable request rate by 15.0%, 14.3%, and 29.7% under normalized-latency, P99 TTFT, and P99 ITL targets, respectively.The configuration adds an independently bound rank, exposing more parallel AMX work and distributing host PagedAttention across NUMA memory domains.Source: §6.4 · p. 10
• At the normalized-latency target, 120 GiB host KV capacity sustains 28.7% and 18.5% higher request rates than 30 and 60 GiB, respectively; at the P99 TTFT target, gains are 28.3% and 14.6%.Results plateau from 120 to 240 GiB as CPU compute and memory bandwidth become primary bottlenecks; smaller capacities favor GPU-resident per-token latency, making capacity an admission-control rather than monotonic ITL knob.Source: §6.4 · pp. 10–11
• Dense Assist polling adds 2–5 μs per layer, while CUDA execution accounts for 60.4% and 70.0% of measured component time for Qwen3-32B and Llama-3.1-8B, respectively.Within CUDA time, post-attention accounts for 79.8% and 79.2%; host launch and finalization account for the remaining 39.6% and 30.0% of measured component time. The stacked component bars are not strictly serialized end-to-end latency.Source: §6.4 · p. 11

7 Related Work

• GPU-centric systems improve batching, KV-cache management, or GPU-side prefill–decode coordination, while Splitwise and DistServe separate phases across GPU replicas.The passage names Orca, vLLM, vAttention, Sarathi-Serve, and POD-Attention, with bibliography markers [18] [36] and [1] [16] [34] [49] for the cited systems.Source: §7 · p. 11
• DualPace differs by keeping CPU-resident requests owned and scheduled by an independent CPU engine, with profile-guided GPU Dense Assist selectively accelerating dense stages.This extends GPU-centric serving with host-resident KV capacity and independent CPU decode scheduling.Source: §7 · p. 11
• Heterogeneous systems place attention near host-resident KV and coordinate it with GPU dense computation, while LIA demonstrates AMX-capable CPU execution of dense LLM operators [8] [10] [15] [24].Examples include FlexGen [38] for GPU, CPU, and storage capacity and FastDecode, NEO, APEX, and OmniServe [8] [10] [15] [24] for host-KV attention.Source: §7 · p. 11

8 Conclusion

• DualPace turns host memory and AMX compute into an independently scheduled LLM decode tier, using profile-guided GPU Dense Assist for selective cooperation.Source: §8 · p. 12
• In decode-oriented scenarios, DualPace improves serving throughput by 6.9–43.2% over NEO and 11.5–90.6% over vLLM while maintaining the same latency.Source: §8 · p. 12
• DualPace preserves NEO’s P99 TTFT advantage over vLLM while sustaining 7.3–50.5% higher request rates than NEO at the same P99 ITL.Source: §8 · p. 12

A Per Engine Scheduling Procedures

• GPU and CPU engines run local event loops whose rounds need not start or finish together.Source: A Per Engine Scheduling Procedures · p. 14
• In each round, an engine constructs a logical schedule, commits KV-block and ownership changes, executes at most one worker batch, and applies its result before its next round.Source: A Per Engine Scheduling Procedures · p. 14
• The engines share request ownership, transfer readiness, future migration intent, and host KV obligations.Source: A Per Engine Scheduling Procedures · p. 14

A.1 GPU Engine Scheduling Round

• GPU rounds retire completed transfers, restore next-token HBM feasibility through feasible evictions, and continue runnable decodes when pressure prevents full admission.Evictions that free only part of the required capacity trigger a migration-only round and reevaluation after transfers complete.Source: §A.1 · p. 14
• GPU rounds reserve host capacity for future evictions, prioritize GPU decodes, admit prefills within resource budgets, then commit migrations before one GPU batch.Prefill-to-CPU requests receive host KV blocks and temporary GPU staging blocks, and remain unready until transfer completion.Source: §A.1 · pp. 14–15

A.2 CPU Engine Scheduling Round

• CPU rounds preserve the running FIFO frontier and admit waiting requests only when that frontier remains intact and resource limits permit.Requests with incomplete transfers or migrations remain unready; the CPU engine waits at an unready queue head rather than bypassing it.Source: §A.2 · p. 15
• The engines maintain one scheduling owner per request and localize backpressure, with migration reserves protecting future GPU eviction paths.These properties require neither a shared GPU–CPU batch nor a cross-engine iteration barrier.Source: §A.2 · p. 15

B Construction of Post-Attention Routing Scores

• These latency functions populate route-score rows for profiled batch sizes; threshold fitting adds no routing objective beyond the main text’s score.The score uses CPU-visible route costs and corresponding GPU disturbance.Source: B Construction of Post-Attention… · pp. 15–16
• Profiles use the serving CPU kernels, GPU paths, buffers, tensor-parallel topology, CPU affinity, and NUMA placement for each CPU decode batch size.CPU primitives use wall-clock latency; GPU primitives use the rank-critical CUDA interval, with the maximum across ranks capturing the tensor-parallel critical path.Source: B Construction of Post-Attention… · pp. 15–16
• GPU-worker microbenchmarks measure batch-dependent CUDA latencies, with each measurement covering the assist’s required data movement and kernels.The measured paths include non-final post-attention and Split-Final post-layer calls.Source: B Construction of Post-Attention… · p. 16
• End-to-end calibration under matching model, topology, affinity, and NUMA placement captures Dense Assist control-path costs beyond microbenchmark execution latency.Calls are grouped by observed batch-size row to measure CPU-visible wait and GPU-worker launch and finalization intervals.Source: B Construction of Post-Attention… · p. 16
• Calibration averages batch-size groups equally, preventing a workload’s request-frequency distribution from becoming an implicit routing prior.The profiler separates CPU-visible non-CUDA wait from GPU-worker orchestration on the serialized GPU-main path.Source: B Construction of Post-Attention… · p. 16
• The profiler combines row-independent calibration constants with batch-dependent GPU measurements to construct wait and displaced-service primitives for each assist path.CUDA execution both delays CPU-batch completion and consumes GPU service that could advance GPU-main work.Source: B Construction of Post-Attention… · p. 16

C Additional Tail-Latency Results

• DualPace provides the lowest P99 ITL over most evaluated Azure-Code request rates, where independent host decoding progresses during long GPU prefills.Short outputs limit the work assigned to the CPU tier in this workload.Source: Figure 13 · p. 17, C Additional Tail-Latency Results · p. 17
• DualPace extends the Azure-Code P99 TTFT-compliant load range most clearly for Llama-3.1-8B on one RTX 4090.On the two multi-H20 configurations, it closely tracks vLLM at the TTFT target while sustaining a higher request rate than NEO.Source: C Additional Tail-Latency Results · pp. 16–17, Figure 13 · p. 17
• 61%, 15%, and 3% higher request rates than NEO at the 1.5 s P99 TPOT target on Azure-Conversation.The values correspond to the three configurations from left to right in Figure 14 (top).Source: Figure 14 · p. 17, C Additional Tail-Latency Results · p. 17
• DualPace generally outperforms NEO and frequently matches or outperforms vLLM on Azure-Code P99 TPOT across measured request rates.The host decode tier continues token generation while the GPU processes prompt work.Source: Figure 14 · p. 17, C Additional Tail-Latency Results · pp. 17–18
• 31% higher request rate at the 1 s P99 TPOT target with CPU TP = 2 than CPU TP = 1, at the same aggregate host KV-cache capacity.Each rank uses 12 physical cores pinned to one NUMA node; the additional rank exposes more parallel AMX execution and aggregate memory bandwidth.Source: Figure 15 · p. 18, C Additional Tail-Latency Results · p. 18
• The 120 and 240 GiB host KV configurations attain nearly identical P99 TPOT, while 30 and 60 GiB yield lower P99 TPOT over the measured request-rate range.The smaller pools keep more requests GPU-resident; increasing capacity to 240 GiB beyond 120 GiB does not change the active CPU execution regime.Source: Figure 15 · p. 18, C Additional Tail-Latency Results · p. 18