Source-linked AI summary
GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
Alessio Ricci Toniolo, Rome Thorstenson, Abinaya Dinesh
TL;DR
Cross-region LLM routing must balance cache locality, replica load, and network latency, but existing policies do not jointly model these costs. GORGO uses an online-tuned, multi-signal routing cost and improves held-out p95 TTFT by 6.9–15.5% and E2E latency by 14.3–30.9% over session affinity.
Problem
Existing cross-region routing policies do not jointly account for network latency, replica load, and KV-cache locality when optimizing TTFT.
Method
GORGO routes requests using a tunable additive cost for network latency, prefill cost, and queueing delay, with weights optimized online from production-like traces.
Results
6.9–15.5% lower p95 TTFT and 14.3–30.9% lower E2E latency than session affinity were achieved on held-out evaluation windows.
Takeaways & Limitations
GORGO provides a routing policy that balances TTFT with request concentration across replicas on long-context, high-prefix-reuse workloads.
Takeaways & Limitations
GORGO does not model heterogeneous hardware, prefill/decode disaggregation, or replica-side KV-cache eviction, and optimizing p95 TTFT can trade E2E and ITL tails for TTFT.
Abstract
from arXiv · showhide
Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at https://github.com/Arcadia-Research-Team/GORGO.
1 Introduction
Cross-region LLM serving must route requests while balancing TTFT’s prefill, network, and queueing costs, especially when prefix-cache reuse depends on replica selection. GORGO addresses this gap by jointly modeling these costs with tunable parameters and online tuning, then evaluates the policy on ART-Chat-2.5M workloads.
- Motivation: TTFT is dominated by prefill time, client-to-replica round-trip time, and queueing delay behind in-flight requests.Prefix-caching can eliminate prefill for previous turns in multi-turn conversations.
- Motivation: Prefix-cache savings depend on routing each session to a replica containing its prefix, motivating affinity-oriented policies such as consistent hashing and prefix reuse.These policies aim to distribute load while maximizing KV-cache reuse.
- GORGO: 10ms to 1s is the cross-region network-latency range that existing least-load, session-affinity, and prefix-reuse policies do not incorporate.GORGO accounts for replica load-related TTFT costs, network latency, and queueing through tunable parameters jointly optimized online.
- Evaluation: GORGO benchmarks ART-Chat-2.5M user workloads across variable time scales designed to saturate replicas without unrealistic head-of-line queueing delay.The evaluation examines how the policy balances TTFT with request concentration across replicas.
2 GORGO
GORGO extends LLM request routing beyond replica load and KV-cache locality by incorporating client-to-server network latency, prefill time, and queueing delay into a tunable cost model. It uses online evolutionary calibration to tune routing weights against p95 TTFT under policy-dependent cache and queue dynamics.
- Routing Signals: GORGO is the first LLM load-balancing policy described here to optimize network latency alongside replica load and KV-cache locality.Its design draws on content-delivery-network routing signals: server load, network proximity, and cache locality.
- Weight-Selection Methods: GORGO uses online calibration because a request log from one routing policy cannot reproduce the counterfactual cache and queue evolution of another.Accurate evaluation therefore requires replay infrastructure, shadow execution, or a sufficiently accurate simulator of cache and continuous-batching dynamics.
- Cost Model: GORGO estimates TTFT from network latency, queueing delay, and prefill time for tokens not already present in a replica’s KV-cache.Continuous batching and delayed queue metrics complicate measurement; network latency uses an EWMA of ping round-trip time, while queue input counts unfinished request tokens.
- Cost Model: GORGO routes each request to the replica with minimum weighted cost, using Wrtt, Wprefill, and Wqueue to normalize latency, prefill, and queueing inputs.The prefill weight is fixed to 1 because routing depends on relative replica costs.
- Weight-Selection Methods: GORGO tunes Wrtt and Wqueue with a (1+1) evolutionary strategy that optimizes p95 TTFT through multiplicative log-space perturbations.Accepted offspring replace the incumbent, and the step size follows Rechenberg’s 1/5 success rule.
3 Dataset
ART-Chat-2.5M addresses the lack of long-context, prefill-bound, high-prefix-reuse workloads in existing chatbot datasets. It is a replay-ready synthetic trace generated from production inference metadata, enabling controlled replica-load scaling.
- Dataset motivation: Existing chatbot datasets lack long-context, prefill-bound requests and multi-turn workloads with high prefix reuse.These limitations affect cache-aware policy benchmarking in LMSYS-Chat-1M and WildChat-4.8M.
- Dataset characterization: ART-Chat-2.5M has 89% intra-user prefix reuse and 0.3% cross-user reuse.Its reuse structure reflects long-context, multi-turn data.
- Dataset construction: ART-Chat-2.5M is synthetically generated from a week-long production inference metadata trace with the original workload’s prefix-reuse structure.The dataset provides long-context, multi-turn requests for benchmarking cache-aware serving policies.
- Dataset construction: The released Mooncake FAST’25 trace includes per-request timestamps, metadata, and synthetic chat completions, allowing linear scaling of inter-request times to control replica load.This makes the dataset replay-ready for controlled experiments.
4 Experimental Setup
The evaluation uses identical multi-region GORGO deployments and workloads, compares online and static policies against several baselines, and tunes weights on one ART-Chat-2.5M window before evaluation.
- Proxy and engine configuration: GORGO runs on a CPU proxy in us-ashburn controlling dedicated SGLang engines in us-ashburn, eu-frankfurt, and ap-seoul.Each engine uses two L40S GPUs and serves Qwen3.5-35B-A3B in FP8 format.
- Proxy and engine configuration: All policies run in parallel on the same workload to rule out variance in network conditions.SGLang metrics are scraped every 30 seconds from the engine’s Prometheus /metrics endpoint.
- Baseline policies: The study compares GORGO’s online and static modes with least-load, least-request, and prefix-cache baselines.least-load uses queued requests and occupied per-token KV slots; least-request selects the replica with fewest in-flight requests; prefix-cache uses prefix overlap.
- Tuning and evaluation windows: Three 30-minute ART-Chat-2.5M windows are selected, with Apr 5th used to tune weights against p95 TTFT on a rolling 128-request window.The rolling window has hop size 32; wrtt and wqueue start at 0.5 and 0.1 and are restricted to [0.05, 2.0] and [0.05, 0.5], respectively.
5 Results
GORGO’s tuned policy generalizes across held-out windows, improving tail TTFT and E2E latency over session affinity. Additional replay and load-sweep experiments show gains under high prefix reuse and sensitivity to saturation, while continuous batching exposes a TTFT–E2E tradeoff.
- Results across three windows: 6.9–15.5% p95 TTFT and 14.3–30.9% E2E latency improvements over session-affinity demonstrate generalization across held-out evaluation windows.GORGO’s weights are updated during tuning to minimize p95 TTFT, then fixed for evaluation.
- Results across three windows: GORGO learns wrtt=0.276 and wqueue=0.5 from initialization at wrtt=0.5 and wqueue=0.1, then outperforms every baseline on p95 TTFT.The learned weights are frozen on held-out evaluation windows.
- BFCL multi-turn replay: 8.9% p50 TTFT, 7.6% p95 TTFT, 10.8% p50 E2E latency, and 7.9% p95 E2E latency reductions occur under high load.The BFCL replay uses 200 teacher-forced episodes, two Qwen3-4B H200 replicas, 128 concurrent episodes, and at most 256 output tokens per request.
- Load Sweep: 3× improvements in p95 TTFT and p95 E2E latency occur after time_scale=3.0, as lower timescales cause saturation and HOL queueing delay.GORGO breaks the saturation point earlier than other policies because of its well-tuned load term.
- Exploiting Continuous Batching in SGLang: 17% better p95 TTFT than the next-best policy is achieved with wqueue=0 and wrtt near 0.23, but at the cost of high p95 E2E latency.For this window, GORGO sent 100% of requests to the closest replica.
6 Related Work
Related work spans single-replica prefix-cache systems, cross-replica and cross-region serving architectures, and online hyperparameter adaptation. GORGO is positioned among these approaches through its routing, serving, and tuning design choices.
- Prefix caches and reuse: Single-replica systems such as RadixAttention and PagedAttention support prefix sharing, while KVLink, ChunkKV, KVFlow, and Learned Prefix Caching improve cache management without selecting the serving replica.These methods store or manage KV state within a replica rather than deciding which replica handles each request.
- Cross-replica routing and cross-region serving: Preble performs longestprefix-match routing across replicas with load-balance fallback, and AIBrix packages a production-grade variant that supplies the evaluation baseline.Mooncake provides a KV-cache-centric architecture and the trace format used by the work.
- Online hyperparameter adaptation: The (1+1)-ES adapts parameters with fast, overhead-free convergence in GORGO’s 2-dimensional weight space, unlike Bayesian-optimization and bandit methods that maintain surrogate models.This distinguishes the online tuning method by its lack of a surrogate model and its low adaptation overhead.
7 Limitations
GORGO’s conclusions are bounded by unmodeled hardware heterogeneity and a tuning objective that can favor p95 TTFT at the expense of other tail metrics. A lower bound on the queue term mitigates but does not eliminate this trade-off.
- GORGO does not model heterogeneous replicas with different prefill rates, queueing behavior, or KV-cache capacities.
- Optimizing p95 TTFT can concentrate load on the nearest replica through continuous batching, trading E2E and ITL tails for TTFT.
- A lower bound on the queue term mitigates but does not eliminate the trade-off between TTFT and E2E and ITL tails.
8 Conclusion
GORGO treats cache locality, replica load, and wide-area latency as jointly important routing signals, then uses online TTFT feedback to optimize their additive cost weights. On a long-context, high-prefix-reuse production trace, the GORGO policy family is evaluated under this adaptive approach.
- 8 Conclusion: GORGO balances cache locality, replica load, and wide-area latency when routing requests across LLM replicas.The conclusion frames routing as a three-signal decision and notes that production heuristics often commit to one signal.
- 8 Conclusion: GORGO combines the three signals into an additive per-replica cost rather than selecting a single routing signal.This cost formulation is designed to address cases where the limiting factor changes.
- 8 Conclusion: Online TTFT feedback optimizes GORGO’s scaling weights instead of operator-tuned constants or offline profiling.The evaluation uses a long-context, high-prefix-reuse production trace.
A Evaluation Window Characteristics
Table 5 describes the workload characteristics of the three tuning and held-out evaluation windows reported in Table 2.
- Evaluation Window Characteristics: Table 5 characterizes three tuning and held-out workload windows used in the evaluation.The passage does not provide the individual workload characteristics.
B Apr 7 at time_scale=2.0
This section reports results for the Apr 7 19:45–20:15 held-out evaluation window at time_scale=2.0.
- The evaluation covers the Apr 7 19:45–20:15 held-out window at time_scale=2.0.
C WildChat replay · D Load Weight Adaptation
The WildChat replay evaluates fixed-weight GORGO under a held-out, multi-replica workload, while load-weight adaptation exposes reward hacking when queueing weight is zero. In that setting, gorgo-static improves TTFT percentiles by concentrating traffic on one replica but worsens E2E and ITL tails under saturation.
- C WildChat replay: Table 7 evaluates held-out WildChat-4.8M replay at time_scale=2.0 with 5.7 requests/s open-loop across three H100 replicas.The configuration uses c=64, and each arm contains 20,422 completed requests with zero failures.
- C WildChat replay: Fixed-weight GORGO uses weights learned on the first window: w_rtt=2.0 and w_queue=0.104.
- C WildChat replay: The replay places one H100 replica in each of us-west4, CANADA-2, and sines-2.
- D Load Weight Adaptation: With w_queue=0, tuned gorgo-static wins every TTFT percentile but is worst on the E2E and ITL tails.The reward-hacking window is Apr 2, 12:30–13:00; ITL is reported as the median.
- D Load Weight Adaptation: gorgo-static routes approximately 100% of traffic to a single replica, which saturates under load.
- D Load Weight Adaptation: On the midday diurnal trace with w_queue=0, gorgo-static wins TTFT p95 while its E2E p95 inflates to 12.6 s.