Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving

1 Introduction

• P/D disaggregation improves efficiency through phase specialization and isolation, but both benefits rely on a static partition between pools.Prefill is typically compute intensive, while decode is often constrained by memory bandwidth and KV-cache capacity.Source: §1 · p. 1
• Demand shifts make static pools mismatched: chat demand ratios reach 4.7× peak-to-mean, while the public agentic trace has a median 24.5× within-day hourly range.Reassigning replicas takes tens of minutes, so pool sizing risks either stranded capacity or queueing.Source: §1 · p. 1
• Crossflow keeps node roles fixed while decode nodes temporarily lend bounded compute for selected prefill work, stopping when decode needs the capacity.This permits sizing prefill for expected long-term demand while retaining a conservative decode margin.Source: §1 · pp. 1–2
• A short-lived multi-resource lease connects decode-node capacity publication, atomic cluster reservation, and locally revalidated prefill execution alongside decode.The design uses a hierarchical controller and bounded execution units.Source: §1 · p. 2
• 16.2–17.4% geometric-mean token-throughput improvement over static P/D, with up to 43.4% at high load and lower mean TTFT at every evaluated point.Evaluations span GPT-OSS-120B and GLM-5.2 with public and internal traces.Source: §1 · p. 2

2.1 Prefill–Decode Serving

• Prefill computes input tokens and materializes KV cache; decode generates tokens autoregressively, with user-facing TTFT and ITL reflecting different delays.TTFT includes admission, prefill queueing and execution, and KV transfer; ITL measures delays between consecutive generated tokens.Source: §2.1 · p. 2
• Prefix caching makes uncached input work and reusable KV location more informative for scheduling than prompt length alone.A long cached prompt may require little incremental prefill, while a shorter cold prompt may require more.Source: §2.1 · p. 2

2.2 Why Disaggregate Prefill and Decode?

• Separate P/D pools can tune parallelism, batching, replica count, and hardware by phase, while dedicated decode replicas avoid prefill-kernel interference.This supports decode ITL predictability and tail latency.Source: §2.2 · p. 2

2.3 The Static Capacity Boundary

• Because conventional P/D replicas belong to one pool, changing the phase capacity split requires reassigning a whole replica and pools are provisioned conservatively.Figure 2 presents per-minute demand-ratio distributions for three agentic use cases and one chat use case.Source: §2.3 · p. 2, Figure 2 · p. 3
• Routing adaptation must account for request-specific KV state as well as compute, since redirecting work can require cache loading, KV transfer, or remote execution.Source: §2.3 · p. 2

3.1 Static Provisioning Strands Capacity

• Minute-scale demand shifts include a 4.7× peak-to-mean ratio, while even the least variable use cases span 1.5× to 1.75× between fifth and ninety-fifth percentiles.The demand ratio counts uncached input tokens relative to output tokens; it represents workload mix, not the P/D resource ratio.Source: §3.1 · p. 3
• Across 2,959 public-trace hours, the median within-day hourly demand-ratio max/min range is 24.5×, and the systematic time-of-day component is small by comparison.Across 134 complete days, the median within-day coefficient of variation is 0.89, and a typical day has an hour deviating 3.9× from its daily mean.Source: §3.1 · p. 3
• Sizing each phase at its 95th-percentile demand strands 10.8–17.5% of a balanced configuration across four use cases; peak-based sizing strands 40% in the most variable case.At a 1:3 prefill-to-decode split, the stranded range is 8% to 14%; fleet estimates are lower bounds because they exclude aggregate-load variation.Source: §3.1 · p. 3
• Decode-to-prefill lending can recover decode-side surplus without replica movement, which takes roughly 11–18 minutes for a full pool to become ready in the measured setups.The end-to-end times are harness measurements, not an optimized lower bound.Source: §3.1 · pp. 3–4
• Asymmetric sizing—prefill near median demand and decode at the 95th percentile or higher—reduces stranded capacity from 11–18% to 6.0–7.4%.The proposed question becomes how much prefill a decode node can absorb now without violating its ITL SLO.Source: §3.1 · p. 4

3.2 Lending Capacity Is Shape Dependent

• SLO-safe lending depends jointly on resident decode shape and candidate prefill work, including batch occupancy, context lengths, output work, KV footprint, and uncached prompt tokens.Source: §3.2 · p. 4
• A 2.5× increase in decode batch size raises compute activity 1.44×, memory traffic 1.30×, and decode throughput 1.74×; longer context changes those resources differently.At batch 8, increasing context from 1K to 82K nearly doubles memory traffic, raises compute activity 1.24×, and reduces decode throughput by approximately 10%.Source: §3.2 · p. 4
• At fixed aggregate decode load, placement changes SLO-safe prefill throughput by 11.3–34.0%, and preferred layouts differ for 65K- versus 98K-token contexts.The controller therefore needs shape-aware cluster placement and a short-lived, locally measured SLO envelope.Source: §3.2 · p. 4

4 Crossflow Design

• Decode nodes temporarily accept selected prefill work under congestion, but only when the spilled path is expected to finish sooner and leased capacity is SLO-safe.Requests that fail any condition remain on the default disaggregated path.Source: §4 · p. 4

4.1 Detecting Prefill Pressure

• The virtual prefill queue charges arrivals for uncached prompt work between polls, allowing a burst-forming request to trigger lending before the next congestion report.Pressure stays active until estimated wait remains below a lower threshold for bounded updates; invalid worker state disables new lending.Source: §4.1 · p. 5

4.2 Deriving an SLO-Safe Computational Lending Budget

• Each decode node issues a revocable, short-lived lease covering compute, KV, projected output, transfer, and spill-slot credits, with feedback shrinking or expanding limits.Missing or unhealthy state and hard resource exhaustion produce a zero lease; SLO violations also set the next lease to zero.Source: §4.2 · p. 5

4.3 Shape-Aware Cluster Admission

• Home placement balances projected prompt work for prefill and projected output work plus turn count for decode; shape affinity breaks ties within a configured load margin.Conversations retain their decode home across turns, and successful completions record the longest prefix processed so far.Source: §4.3 · p. 6
• A spill must target its assigned decode home under prefill pressure, beat the default path’s predicted time, and satisfy request-shape, lease-credit, and cluster-concurrency checks.Admissions reserve credits before evaluating another candidate, preventing concurrent requests from overcommitting capacity.Source: §4.3 · p. 6
• Crossflow ranks a bounded set of waiting candidates by predicted completion benefit and effects on other requests, but ranking cannot override admission constraints.The score accounts for benefit to the forming prefill batch and interference with active decode requests.Source: §4.3 · p. 6

4.4 Decode-Node Execution and Recovery

• Decode-node executors bound and revalidate leased work, pause unissued prefill when conditions change, and retry pre-first-token failures through the prefill tier.After the first token, the request stays on its decode node until generation ends; only successful completion advances prefix coverage.Source: §4.4 · pp. 6–7

5 Implementation

• A 6.7K-line Python shim implements routing, virtual prefill queuing, home assignment, path comparison, lease reservation, and failure recovery in SGLang.Worker-side support adds 2,807 lines and removes 36 across 15 SGLang files; no new model kernels are required.Source: §5 · p. 7
• The SGLang disaggregated path is extended with a local prefill queue so decode workers can accept full requests, reuse resident prefixes, and build mixed batches.Workers check physical KV capacity, and tensor-parallel ranks validate the same lease before dispatch.Source: §5 · p. 7
• The executor interface is implemented with native chunked mixed-batch execution and PDMUX; both backends reach similar end-to-end performance, while Section 6 uses the native path.A MuxWise-style executor would also satisfy the interface.Source: §5 · p. 7

6.1 Experimental Setup

• Experiments use GPT-OSS-120B and GLM-5.2 on GB300 servers, replaying timestamped public TraceLab and internal trace segments against baselines on a common serving substrate.Matched comparisons hold checkpoint, precision, server count, P/D split, KV-cache capacity, and distributed L3 cache constant unless varied explicitly.Source: §6.1 · p. 7, §6.2 · p. 7

6.2 End-to-End Benefit over Static

• Across 16 matched load points, Crossflow improves request, input-token, and output-token throughput by 16.6%, 17.4%, and 16.2% geometrically over Static.Mean TTFT falls at every point: by 10.0–34.4% and 10.7–34.4% for GPT-OSS-120B on TraceLab and the internal trace, and by 46.1–57.4% and 32.2–47.7% for GLM-5.2 on those traces.Source: §6.2 · p. 7, Figure 5 · p. 8

6.3 Robustness to the Static P/D Partition

• On TraceLab, Crossflow raises input-token throughput 24.1–42.7% and output-token throughput 29.9–51.1% over Static at 1P5D–3P3D, but input throughput falls 2.8% at 4P2D.On the internal trace it raises input-token throughput 5.9–16.3% and output-token throughput 2.1–11.8% at every split; at 2P4D it exceeds 3P3D Static on both traces’ token-throughput metrics and has lower mean TTFT and ITL.Source: §6.3 · pp. 8–9, Figure 6 · p. 9

6.4 Comparison with State-of-the-Art Baselines

• On TraceLab, Crossflow has the highest measured output-token throughput at 16×, 18×, and 20×, beating the best non-Crossflow policy by 43.4%, 32.6%, and 20.9%.At higher rates PPD achieves lower TTFT, but its mean ITL is 5.04–6.28 ms, or 2.2–3.1× Crossflow.Source: §6.4 · p. 9, Figure 7 · p. 9

6.5 Hardware Requirements of Static P/D Provisioning

• At six nodes, Crossflow reaches 763K input tokens/s and 3.43K output tokens/s; nine-node Static remains 2.7% and 8.3% lower, despite using 50% more hardware.Seven-node Static approaches Crossflow’s mean/p99 TTFT at 2.30/16.69 s versus 2.40/24.75 s, but reaches only 2.83K output tokens/s.Source: §6.5 · p. 10, Figure 8 · p. 10

6.6 Ablation Study

• Combining spill execution, shape-aware selection, and pressure-aware concurrency, Crossflow improves input-token throughput 16.3–42.7% and output-token throughput 15.3–43.4% over Static.At 16×, synchronous D-capacity polling reduces throughput from 6.79 to 5.68 requests/s and adds 202 ms mean polling latency; disabling the virtual queue yields 5.27 requests/s and 56 spills versus 6.79 requests/s and 346 spills for Crossflow.Source: §6.6 · pp. 10–11, Figure 9 · p. 10, Table 1 · p. 11

LLM serving systems

• Serving systems use batching, cache reuse, kernels, quantization, and adapters to reduce computation or increase effective model and KV capacity; Crossflow targets the boundary between specialized prefill and decode pools.These approaches are described as orthogonal to Crossflow’s target.Source: §7 · p. 11

Prefill–decode disaggregation

• P/D systems specialize prompt processing and generation, while other approaches adapt at static, replica, or engine-configuration granularity; Crossflow keeps pool assignment fixed and borrows bounded prefill execution from decode nodes.Colocated systems manage sharing after both phases are assigned to a node; Crossflow adds a cluster-level decision about when to borrow decode-local capacity.Source: §7 · p. 11

Routing, cache affinity, and agentic workloads

• Routing and cache-affinity work addresses sessions, locality, and bursty agentic workloads; Crossflow distinguishes itself with short-lived multi-resource leases, atomic reservation, and local revalidation before bounded prefill execution.PPD is identified as the closest prior system for deflecting prefill using decode-side state.Source: §7 · p. 11

• Crossflow makes the static P/D capacity boundary elastic through short-lived, revocable lending capacity while decode nodes remain decode-first.A cluster scheduler admits and places prefill requests against that capacity; decode schedulers enforce reservations through a bounded, locally revalidated backend.Source: §8 · p. 11
• Lending decisions account for request shape on both sides of the spill, cache locality, and changing decode conditions rather than treating idle compute as a scalar resource.Source: §8 · p. 11
• 16.2–17.4% geometric-mean token-throughput improvement over static P/D, up to 43.4% at high load, with lower mean TTFT at every evaluated point.Evaluated across GPT-OSS-120B and GLM-5.2 on public TraceLab and internal-trace replays.Source: §8 · p. 11