ExaServe: Large-Scale LLM Serving on Exascale HPC Systems

• Researchers seek in-facility serving for restricted data, campaign-scale throughput, and allocation-compatible accounting, while scientific workflows can issue thousands of inference calls per run.Open-source users also need deployments with thousands of concurrent replicas for workloads that outgrow lab clusters.Source: §I · p. 1
• Cloud serving frameworks provide mature engine and orchestration abstractions but assume a Kubernetes-style control plane and external load balancer, and offer no turnkey multi-node path for batch HPC.Leadership systems also vary in scheduler and accelerator hardware while sharing a batch-allocation model.Source: §I · p. 1
• ExaServe turns a declarative deployment specification into a managed endpoint inside a batch allocation, integrating the scheduler, MPI launch, accelerator selection, weight staging, and site-specific patches.The framework is evaluated on Aurora, a PBS-scheduled Intel XPU system.Source: §I · p. 1
• ExaServe’s scaling study finds near-linear request throughput, but centralized-proxy token streaming collapses under the interactive SLO while model servers remain within it.The paper also analyzes Ray/vLLM control-plane patterns that dominate deployment time at scale and identify a single-cluster growth limit.Source: §I · pp. 1–2

II. MOTIVATION AND BACKGROUND

• Five barriers span missing orchestration, unknown scaling limits, cloud-tuned defaults, non-CUDA setup, and site-specific topology.The first three apply broadly to batch-scheduled systems; accelerator and topology compatibility vary by site.Source: §II · p. 2

III. DESIGN AND IMPLEMENTATION OF EXASERVE

• ExaServe renders scheduler jobs, stages model weights, plans per-node replicas, launches serving applications, and optionally exposes them through a front-end proxy.Its serving layer wraps configurable vLLM or SGLang replicas behind an OpenAI-compatible API.Source: §III · p. 2
• Instrumentation hooks and site-compatibility patches support measurement and address Aurora-specific runtime behavior before Ray or the inference engine is imported.The components run partly on the login node and partly inside the batch allocation.Source: §III · p. 2

A. Lifecycle

• The lifecycle moves from login-node job rendering and submission to MPI-launched Ray workers, with model weights staged before Ray starts and the public proxy gated on health checks.A C helper reads weights once on the head node and broadcasts them to node-local tmpfs, avoiding more than 600 s of added startup time at 256 nodes for Llama-3-8B.Source: A. Lifecycle · pp. 2–3, Figure 1 · p. 3

B. Replica planning

• The planner uses model parallelism and context settings plus node and tile counts to determine replica count, tile assignments, and Ray placement groups.Aurora’s default single-tile placement yields 12 independent replicas per node; multi-tile and pipeline-parallel placements are also supported.Source: B. Replica planning · p. 3

C. Shard-aware pipeline parallelism

• For a 405B model at PP=2, shard-aware staging places each roughly 380 GiB pipeline stage only on the node that runs it, avoiding a full 760 GiB copy per node.ExaServe submits the node-pinned applications concurrently and uses a root-level ingress to route requests to replicas.Source: C. Shard-aware pipeline parallelism · p. 3

D. Front-end proxy

• Each proxy adapter renders configuration, starts its process, and reports a health endpoint; the package offers seven modes, including head-node proxies, Ray Serve, and direct per-node access.The at-scale sweep covers HAProxy, Envoy, Ray Serve, and LiteLLM; NGINX and Pingora pass single-node smoke tests.Source: D. Front-end proxy · p. 3

E. The declarative spec

• A single YAML specification includes cluster ports, model deployment, and front-end proxy settings, while users choose the model list, node count, and storage paths.Ports, per-node GPU count, and proxy settings use site-tested defaults that rarely change.Source: E. The declarative spec · pp. 3–4
• The minimal example sets 12 GPUs per node, tensor and pipeline parallelism to 1, and HAProxy with least-connections balancing.The example also includes node count and shared and local model-storage paths.Source: E. The declarative spec · p. 4

F. The two-tier patch system

• Stock Ray Serve and vLLM have Aurora-scale hazards: cloud-oriented health checks can kill healthy proxies, while vLLM’s XPU path mishandles pipeline-parallel lookup, device selection, and tokenizer thread-pool sizing.These settings are fixed at framework load, and vLLM engines start in fresh processes, so launcher-side changes cannot repair them afterward.Source: F. The two-tier patch system · p. 4
• Set A distributes corrected Ray Serve files to every node and raises proxy timeout and health thresholds; Set B applies vLLM/XPU runtime fixes in each process before affected code runs.Each patch has an independent environment-variable switch; Set A is engine-agnostic, while Set B is specific to vLLM and the XPU runtime.Source: F. The two-tier patch system · p. 4

IV. PERFORMANCE CHARACTERIZATION

• The lifecycle characterization covers cluster bring-up, steady-state serving, and a deployment recipe distilled from both.The recipe links each operational friction to its addressing mechanism and the measurement supporting it.Source: §IV · p. 4

A. Methodology

• Each Aurora node has two Intel Xeon Max sockets, six PVC accelerators presenting 12 logical XPU tiles, and an HPE Slingshot 11 link.Experiments use the frameworks/2025.3.1 module, including Python 3.12, Aurora Ray 2.53.0, vLLM 0.15.0, and oneAPI.Source: A. Methodology · p. 4
• The baseline workload uses ShareGPT prompts with at least 64 content tokens, serves an average of 74.7 input tokens, and caps output at 64 tokens.Shorter prompts are discarded, no padding is used, and every request reaches the output cap.Source: A. Methodology · p. 4
• 118 QPS is the profiled single-node ceiling, while multi-node sweeps offer 110 QPS per node to run under saturation stress.For Llama-3-8B-Instruct, latency begins climbing above approximately 109 QPS; the profile sweeps offered rate and two queuing and batching variables per model and workload.Source: A. Methodology · p. 4
• Requests meet the SLO when TTFT is at most 2 s and 99th-percentile TBT is at most 250 ms; reported attainment is the fraction meeting both bounds.The TBT bound follows sustained reading speed, while using its 99th percentile avoids hiding visible generation stalls through averaging.Source: A. Methodology · pp. 4–5
• The characterization matrix uses a baseline configuration and one-axis perturbations, with one-axis rows at 64 nodes and 405B and SGLang rows weak-scaled to 256 nodes.Proxy alternatives, workload and arrival variants, and the 405B deployment are among the characterized configurations.Source: A. Methodology · p. 5, Table I · p. 5

B. Cluster bring-up: time-to-serve at scale

• Replica creation, the deploy phase, grows mildly from 115 s at 64 nodes to 155 s at 256 nodes, while serve.run dominates bring-up at scale.The first two lifecycle phases stay roughly flat in node count: about 50 s for Llama-3-8B weight staging and 45 s for Ray cluster start; proxy start is subsecond, followed by an approximately 8 s first-request warm-up.Source: B. Cluster bring-up: time-to-serve at… · p. 5
• 1612 s of the 1767 s serve.run time at 256 nodes is proxy-side resolution, as discovery grows 42× from 64 to 256 nodes and reaches 1.38 million lookups.The wait phase rises from 38 s to 1612 s; every proxy resolves every replica through GCS as replica announcements arrive in waves.Source: Table II · p. 5, B. Cluster bring-up: time-to-serve at… · p. 5
• 512 nodes (6144 replicas) exceed the demonstrated single-head Ray limit: worker-to-GCS RPC load saturates GCS, workers lose connections, and deployment aborts.The paper identifies GCS sharding or multiple Ray clusters as required beyond the 256-node single-head ceiling.Source: B. Cluster bring-up: time-to-serve at… · p. 5
• The 405B PP=2 deployment takes 31–38 minutes from 2 to 128 replicas and is staging-dominated rather than controlled by replica discovery.Reading each node’s approximately 380 GiB stage from Lustre takes 1300–1600 s independent of node count; concurrent deployment adds 500–770 s.Source: B. Cluster bring-up: time-to-serve at… · p. 5

C. Steady-state serving: non-streaming throughput scales, streaming SLO does not

• Node HAProxy serving non-streaming completions reaches 27.1 k QPS (3.8 M tokens/s) at 0.0% error; a routing-free local-port baseline completes the same non-streaming throughput, indicating no measurable proxy routing cost.Source: C. Steady-state serving: non-streaming… · p. 6
• ∼4.7 k QPS: centralized HAProxy streaming plateaus from 128 nodes upward, with near-zero SLO attainment at 256 nodes.Across 3072 replicas, server-side TTFT and per-token TBT remain within the objective; the paper attributes the gap to the centralized token-delivery path.Source: C. Steady-state serving: non-streaming… · p. 6, Figure 2 · p. 6, Figure 4 · p. 7
• 18.1 k QPS at 100% success: direct-dispatch streaming’s lower-than-linear reported rate reflects the fixed measurement window, not lost capacity.Every cell completes its offered requests; the window-to-window-plus-drain ratio reproduces measured rates within 0.5%.Source: C. Steady-state serving: non-streaming… · p. 6
• 31.0 query/s at 128 replicas: direct dispatch scales Llama-3.1-405B throughput near-linearly, while HAProxy reaches 22.9 query/s with 0.3% errors.The remaining efficiency loss is attributed to the fixed measurement window and draining queued requests, not slower replica execution.Source: Figure 5 · p. 7, C. Steady-state serving: non-streaming… · p. 8
• At 64 nodes, SLO attainment falls from 0.94 to 0.65 for the short/high-rate baseline and from 0.92 to 0.60 for Poisson arrivals.Longer-context and lower-rate workloads stay within two points of their single-node values; the reported degradation occurs near the single-node rate knee.Source: C. Steady-state serving: non-streaming… · p. 7
• SGLang’s direct-dispatch baseline reaches 98% weak-scaling efficiency at 256 nodes, placing the observed ceiling at the shared proxy rather than the engine.Through the identical HAProxy path, both engines show near-linear weak scaling until reaching proxy limits; SGLang’s Torch-native attention backend on Aurora PVC is slower than vLLM’s.Source: C. Steady-state serving: non-streaming… · p. 8, Figure 6 · p. 8

D. Deployment recipe

• 27.1 k QPS of non-streaming completions at 256 nodes with zero errors makes head-node HAProxy the recommended single-endpoint choice.For streaming, centralized proxying plateaus near 4.7 k QPS; distributed dispatch sustains 18.1 k QPS at 100% success.Source: D. Deployment recipe · p. 8
• Stage weights to per-node tmpfs through MPI broadcast, apply the Set-A timeout overlay from 128 nodes, and retain thread-pool caps.This avoids more than 600 s of parallel file-system access overhead for Llama-3-8B at 256 nodes; at 256 nodes, the overlay avoids proxy kills seen with stock thresholds.Source: D. Deployment recipe · p. 8
• Use distributed benchmark clients for throughput characterization, and size Ray clusters for their efficient range rather than their maximum.Serving remains stable through 64–128 nodes; 256 nodes is the largest validated configuration, while 512-node bring-up fails from quadratic control-plane overhead.Source: D. Deployment recipe · p. 8

V. RELATED WORK

• Inference engines and serving optimizations improve execution, while ExaServe addresses deployment, integration, and scalability on native batch-scheduled exascale HPC platforms.The cited work spans inference engines –, distributed or memory-efficient inference,, and serving optimizations,,, –.Source: §V · p. 8
• Cloud serving layers provide orchestration, while ExaServe characterizes deployment and control-plane scaling on batch-scheduled exascale HPC.The cited serving layers include Ray Serve, NVIDIA Dynamo, and SkyServe.Source: §V · p. 9
• ALCF’s Inference Endpoints provides a managed gateway, whereas this work targets self-service deployment inside a user allocation.The ALCF Megatron / DeepSpeed stacks target training rather than serving.Source: §V · p. 9
• Kubernetes-based orchestration and native HPC management tools address distributed workloads, while ExaServe integrates Ray Serve with MPI launch, model staging, and site-specific adaptations in PBS-managed Aurora allocations.Related approaches include KubeRay, Kubeflow MPIJob, Volcano MPI, Flux, Mochi, and a Ray Slurm deployment guide.Source: §V · p. 9

A. Fault tolerance

• A single rank failure terminates the entire MPI job, trading fault sensitivity for dispatch throughput.Ray Serve restarts dead actors through its controller, with a default health check every 30 s.Source: A. Fault tolerance · p. 9
• Ray Serve’s actor-restart behavior is a benefit for long-running production endpoints, while MPI’s simplicity and performance dominate for batch evaluation runs..Source: A. Fault tolerance · p. 9
• A proposed hybrid keeps Ray Serve for control and lifecycle while using MPI-style dispatch on the hot path.The paper notes that this fits the existing ProxyBackend interface.Source: A. Fault tolerance · p. 9

B. Limitations

• The Ray overlay’s startup contribution has not been profiled beyond 256 nodes; the 512-node failure requires sharded GCS or multiple Ray clusters to go beyond that scale.The re-resolution cost remains until Ray’s long-poll payload carries resolved handles or diffs, an upstream change considered future work.Source: B. Limitations · p. 9
• A single endpoint concentrates every request and token stream on one head node, and distributed-client measurements may not be possible in production.The endpoint inherits head-node limits, including roughly 28 k HTTP/1.1 ephemeral ports and one proxy process’s CPU capacity.Source: B. Limitations · p. 9
• Beyond a few hundred nodes, serving requires several advertised endpoints or a proxy tier distributed across nodes..Source: B. Limitations · p. 9

VII. CONCLUSION

• At 256 nodes, HAProxy serves 27.1 k QPS of non-streaming requests, while centralized streaming plateaus near 4.7 k QPS and distributed dispatch reaches 18.1 k QPS at 100% success.The streaming bottleneck is attributed to concentrating token streams on the head node’s network path; the scaling pattern also appears with SGLang.Source: §VII · p. 9
• Ray Serve’s O(N^2) replica re-resolution grows serve.run to 1767 s at 256 nodes, and GCS saturation causes 512-node deployment failure.The paper identifies centralized token delivery and the centralized control plane as architectural bottlenecks.Source: §VII · p. 9