PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving

Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly

arXiv:2609.16206v1cs.AI

TL;DR

Disaggregated serving separates prefill and decode, but placement remains unresolved even though request lengths and instance state affect completion time. This paper prices candidate placements using admission-time signals and evaluates a calibrated router on hardware, reporting higher mean goodput and lower trace spread on mixed traffic.

  • Problem

    After prefill and decode are split across pools, placement still matters because requests with very different output lengths can occupy decode slots for very different durations.

    Source: §I · p. 1

  • Method

    A calibrated router estimates completion-time costs for candidate instances using backlog, request work, cache pressure, and SLO class.

    Source: §I · p. 1, A. Marginal-cost scoring · p. 2, B. Calibration is part of the method · p. 3

  • Results

    Across three mixed-bursty traces, the learned router has the highest mean goodput and smallest trace-to-trace spread.

    Source: A. Goodput under mixed bursty load · p. 4

  • Takeaways & Limitations

    With calibrated costs, the router matches blind routing’s service using about 14% fewer GPUs.

    Source: §VII · p. 7

    Even after refitting to measured hardware and adding continuous batching, the simulator fails to reproduce the measured winner on any workload.

    Source: §VI · pp. 5–6

Abstract

from arXiv · show

Disaggregated LLM serving places compute heavy prefill and memory heavy decode on separate GPU pools. Systems such as DistServe, Splitwise, and Mooncake make this separation fast, but routing still determines which instances handle each request. We study a router that estimates the additional completion time on each instance using exact prompt length, predicted output length, post admission KV cache pressure, and SLO class. We develop the policy in a discrete event simulator and validate it on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools. All workloads run at measured saturation. Across three mixed, bursty arrival traces, the calibrated router achieves the highest mean goodput at 0.864, compared with 0.835 to 0.847 for round robin, least loaded, and a length heuristic. It also shows the lowest variance across traces. It beats round robin and the length heuristic on all three traces and least loaded on two. On the third, it trails by 0.003, within run to run noise. Hardware calibration matters: simulator derived constants cost 4.5 goodput points and roughly 40 percent of the tail latency advantage, reducing the scorer to little more than queue counting. Benefits grow with decode pool size and traffic heterogeneity but disappear in pools with three instances, where queue counts are often enough. Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, and blind spreading performs better. With calibrated costs, the learned router matches the goodput of round robin using six GPUs instead of seven.

I. INTRODUCTION

Disaggregating prefill and decode removes their interference but leaves placement unresolved, even though request lengths and instance state affect completion time. The paper develops and measures a router that prices placements using admission-time signals.

I. INTRODUCTION

  • A one-line turn and a thousand-token trace count equally to queue-based policies despite decode durations differing by two orders of magnitude.Sending a long request to an apparently idle instance can miss its SLO and those of requests queued behind it.Source: §I · p. 1
  • The proposed router prices candidate instances using backlog, request work, cache pressure, and SLO tightness, and the paper measures it on hardware.The contribution also includes a real-cluster validation harness and measurements of calibration, feature importance, robustness, and failure regions.Source: §I · p. 1

II. BACKGROUND AND RELATED WORK

  • Disaggregated systems such as DistServe [1], Splitwise [2], and Mooncake [3] separate phases, but their per-request placement policies remain simple.The paper identifies admission-time placement across the prefill/decode fabric as the gap it measures.Source: §II · pp. 1–2
  • The paper’s admission-time assignment is distinct from execution-order scheduling, post-admission migration, and model multiplexing.Continuous batching, chunked prefill [5], Llumnix [6], and AlpaServe [8] hold this assignment fixed.Source: §II · p. 2
  • Output-length prediction is treated as swappable, with the paper measuring how little accuracy the router needs; evaluation also targets burst fragility at measured saturation.The related-work discussion cites length estimators [9], [10], schedulers TetriInfer [11] and LAPS [12], and adaptive rescheduling for bursts [13].Source: §II · p. 2

A. Marginal-cost scoring

The router estimates the completion time a request would experience on each candidate instance, then chooses the lowest-cost prefill and decode placements.

A. Marginal-cost scoring

  • Each decode candidate receives an estimated completion-time cost for the request, based on its predicted output length, prompt length, and SLO class.The router scores prefill candidates analogously using prompt length.Source: A. Marginal-cost scoring · p. 2
  • Decode cost combines committed backlog, the request’s priced work, post-admission cache pressure, and an SLO multiplier for tight requests.The request is assigned to the argmin in each pool.Source: A. Marginal-cost scoring · p. 2
  • The architecture transfers the KV cache from the chosen prefill instance to the chosen decode instance, while SLO attainment and live load feed back to the router.Baselines replace only the selection rule.Source: Figure 2 · p. 3

B. Calibration is part of the method

Hardware calibration is part of the scoring method because its backlog and request-work terms must be expressed on compatible scales. The predictor is deliberately simple, and simulation develops hypotheses rather than supplying evaluation results.

B. Calibration is part of the method

  • A five-minute benchmark of the target cluster sets per-token prefill and decode costs and SLO targets before routing.When hardware constants are wrong, size-aware terms are silenced and the scorer degenerates into queue counting; simulator-derived decode time was 7.4× too small on the testbed.Source: B. Calibration is part of the method · p. 3

C. Output-length predictor

  • The output-length predictor is deliberately simple, and the router is evaluated with injected prediction error up to 150%.The paper treats the predictor as swappable because its quality bounds nothing else in the system.Source: C. Output-length predictor · p. 3

D. Development in simulation, evaluation on hardware

  • A deterministic discrete-event simulator develops the policy and hypotheses, but all reported results are measured on hardware.The paper reports divergences between simulated and measured service models, constants, and feature importance.Source: D. Development in simulation,… · p. 3

IV. MEASUREMENT METHODOLOGY

The evaluation uses an eight-GPU vLLM/NIXL testbed and a per-request-journaled protocol with correctness gates, run isolation, and workload-specific saturation search.

A. Testbed

  • Eight NVIDIA A40 GPUs run unmodified vLLM engines, with NIXL transferring KV caches; the primary topology uses two prefill and four decode instances.A router proxy tracks per-instance backlog and cache occupancy and journals every request.Source: A. Testbed · p. 3

B. Protocol

  • Correctness gates include exact token counts, bitwise greedy-output equivalence across decode GPUs, and warming every prefill–decode pair.The protocol resets prefix caches between runs after identical configurations varied by more than 20 points with session ordering.Source: B. Protocol · pp. 3–4
  • 0.003 is least-loaded’s edge over the learned router on one trace; the learned router still has the best mean goodput and smallest spread across three mixed-bursty traces.It beats round-robin and the heuristic on every trace.Source: Table II · p. 4
  • Each workload family is driven at the arrival rate where round-robin reaches mid-collapse, so policies are compared under contention.The protocol probes for that saturation point and journals runs for resumption.Source: B. Protocol · p. 4

C. Workloads and baselines

Under mixed, bursty traffic, the learned router has the strongest and most consistent goodput across three measured arrival traces. The section also identifies the baselines and evaluates workloads at measured saturation.

C. Workloads and baselines

  • The section compares round-robin, least-loaded (JSQ), and a predicted-length threshold heuristic across four workload families.Table II rows average three arrival traces; the figures and tables use measured testbed results.Source: C. Workloads and baselines · p. 4, §V · p. 4

A. Goodput under mixed bursty load

  • The learned router posts 0.860/0.877/0.855 across three traces, with the highest mean and smallest trace-to-trace spread; its lone baseline loss is by 0.003.Variance matters operationally because SLAs are written against worst-case behavior.Source: A. Goodput under mixed bursty load · p. 4

B. What calibration is worth

Hardware calibration substantially improves the learned router's goodput and tail latency by preserving the influence of size-aware costs. Routing gains vary with workload heterogeneity: they are strongest when request sizes differ and can disappear on homogeneous chat.

B. What calibration is worth

  • Calibration raises mean goodput from 0.819 to 0.864 and reduces P99 TTFT from 0.42–0.52 s to 0.25–0.30 s.The calibration step accounts for 4.5 goodput points and roughly 40% of the tail advantage.Source: B. What calibration is worth · p. 4
  • With mispriced constants, request counts balance while token load skews worse than JSQ, reducing the scorer to queue counting.The paper reports this as a failure mode for learned schedulers tuned with simulation cost models.Source: B. What calibration is worth · p. 4

C. Effect of workload heterogeneity

  • On homogeneous chat, the heuristic reaches 0.855 versus 0.760 for the learned router, while the learned router leads on mixed bursts and long-output reasoning.On long-prompt documents, round-robin falls to 0.585 with 1.1 s tails, while every load-aware policy holds at least 0.695.Source: C. Effect of workload heterogeneity · p. 4

D. Effect of pool width

The learned router's advantage depends on available alternatives and provisioned capacity: it leads in the deployable middle of the GPU frontier, but not in very narrow or severely starved pools. Under rising load and bursts, its tail behavior compares favorably with the baselines.

D. Effect of pool width

  • At decode width 3, the learned router ties JSQ at 0.816 versus 0.822; it leads at width 4, then policies reconverge at width 6.At width 4, it posts 0.864 versus 0.835 for JSQ; at width 6, it posts 0.858 versus 0.860 for round-robin.Source: D. Effect of pool width · p. 4

E. Goodput versus provisioned GPUs

  • With 1 prefill and 2 decode GPUs, the learned router collapses to 0.292 while round-robin reaches 0.680; greedy scoring concentrates load, so blind spreading performs better.The inversion was reproduced four times, including with calibrated constants.Source: E. Goodput versus provisioned GPUs · pp. 4–5
  • At 6 GPUs, the learned router reaches 0.870 goodput, matching the goodput round-robin needs 7 GPUs for; it leads the deployable middle of the frontier.At 5 GPUs it is best at 0.823; at the overprovisioned end, policies interleave within noise.Source: E. Goodput versus provisioned GPUs · p. 4, Figure 4 · p. 5

F. Load and tails

  • During a sustained burst train, the learned router keeps rolling P95 TTFT roughly 20% lower, at 0.28 s versus 0.30–0.37 s for round-robin.At the operating point, its P99 TTFT is 0.321 s versus 0.337 s for round-robin and 0.346 s for JSQ.Source: F. Load and tails · p. 5

G. Which feature does the work

Ablations identify predicted output length as the dominant feature, while predictor-noise tests show the router remains robust to substantial estimation error.

G. Which feature does the work

  • Removing predicted output length costs 3.2 points under steady load and 4.7 under saturation, the largest loss in both regimes.The ablation makes output-length information more consequential than prediction precision in these tests.Source: G. Which feature does the work · p. 5
  • Cache pressure contributes nothing measurable at the 60ktoken KV budget, while the SLO term helps at moderate load (−1.7) but mildly backfires under deep saturation (+3.7).The authors caution that the cache result does not establish the feature is unhelpful because its regime was not reached.Source: G. Which feature does the work · p. 5

H. How good the predictor needs to be

  • Goodput shifts from 0.838 to 0.880 as predictor relative error rises from 0 to 150%, within trace noise.The router still beats the heuristic at the worst injected error; a length signal matters more than its precision.Source: H. How good the predictor needs to be · p. 5

VI. DISCUSSION AND LIMITATIONS

The simulator fails to reproduce measured policy rankings, even after batching improves its hardware-physics fit. The measured routing advantage is bounded by cluster conditions and the evaluated hardware and model scope.

VI. DISCUSSION AND LIMITATIONS

  • 7–13×-off simulator cost constants and a serial-service model fail to reproduce the measured winner on any workload.After re-fitting, simulator goodput has a mean absolute residual of 0.068 across the 4 × 4 grid.Source: §VI · p. 5
  • Batching brings P99 TTFT error to 0.5–1.4× on three workloads but does not improve policy ranking; the model still misses key hardware interactions.Those include intra-batch interference, chunked-prefill contention, and KV fragmentation; on bursty, simulated policies fall within 0.01 while hardware separates them by 0.03.Source: §VI · p. 6
  • Restoring the proxy backlog estimator recovers only +0.1 goodput points in simulation versus +4.5 on hardware, attributed to batch-level interference absent from the serial model.The simulator's cluster-clock backlog signal gives the policy true remaining work, unlike the deployed proxy's estimate based on assumed constants.Source: §VI · p. 6
  • The router’s advantage requires decode-pool width ≥4, heterogeneous traffic, and contention short of overload; under extreme scarcity, greedy placement concentrates load on the cheapest instance.The authors suggest power-of-two sampling or an admission guard; cache pressure was inert at their tested KV budget.Source: §VI · p. 6
  • Evaluation scope is one silicon per experiment set, single-node NIXL transfer, a 3B model, and hand-set scoring weights with fitted constants.Cross-node RDMA would add a transfer term the scorer already accommodates.Source: §VI · p. 6

VII. CONCLUSION

On a real disaggregated vLLM/NIXL cluster, the calibrated marginal-cost router achieves the highest mean goodput and lowest variance across traces. Its gains depend on hardware calibration and are bounded by pool width, traffic heterogeneity, and load concentration under scarcity.

VII. CONCLUSION

  • The calibrated router achieves the highest mean goodput on mixed traffic and the lowest variance across traces.These results are reported for a real disaggregated vLLM/NIXL cluster.Source: §VII · pp. 6–7
  • With calibrated costs, the learned router matches round-robin goodput using about 14% fewer GPUs.Source: §VII · p. 7
  • Calibrating the cost model to measured per-token times is worth 4.5 goodput points; without it, the scorer reduces to queue counting.Source: §VII · p. 7
  • The advantage is bounded by pool width and traffic heterogeneity, while greedy cost minimization can concentrate requests under scarcity.Source: §VII · p. 7

Related papers

Loading 2609.16206v1…