Source-linked AI summary
ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
Sangjin Choi, Sukmin Cho, Yifan Xiong, Ziyue Yang, Youngjin Kwon, Peng Cheng
TL;DR
Existing decode routers balance load but overlook expert locality, so equally loaded MoE workers can have different decode latency. ELDR routes requests using prefill-derived expert signatures and reduces median TPOT by 5.9–13.9% over the best load-balancing baseline across the evaluated workloads and models.
Problem
Load-only decode routing treats MoE workers as interchangeable, although latency depends on the union of distinct experts activated by each batch.
Method
ELDR derives expert signatures from prefill activations, partitions signature space offline, and uses locality-band routing to choose the least-loaded similar worker.
Results
7.0–13.9% lower median TPOT on the task workload and 5.9–10.0% lower median TPOT on the language workload versus the best load balancer.
Takeaways & Limitations
Expert-locality-aware routing adds a load-tolerant second axis that lowers median and tail TPOT across the evaluated models, workloads, and topologies.
Abstract
from arXiv · showhide
In prefill-decode (PD) disaggregated LLM serving, each request is assigned to a decode worker after prefill. Existing decode routers balance only load; for mixture-of-experts (MoE) models this is incomplete: equally loaded workers can differ in latency, since each decode step loads the weights of every distinct expert its batch activates. We present ELDR, an expert-locality-aware decode router for PD-disaggregated MoE serving. From a request's prefill expert activations, ELDR builds an expert signature predicting the experts it will activate during generation. Offline, balanced K-means partitions signature space across decode workers; online, locality-band routing sends each request to the least-loaded worker among those best matching its signature. A signature cache, co-indexed with the KV cache at KV-block granularity, keeps signatures exact under prefix caching. Implemented in vLLM and evaluated on deployments of up to 40 GPUs, ELDR reduces median TPOT by 5.9-13.9% over the strongest of four load-balancing baselines across three MoE models and two workloads, with model outputs unchanged.
2 Background
MoE serving combines sparse expert computation with PD disaggregation, but decode workers are not interchangeable: latency depends on the experts a batch activates, not merely its size or load.
- MoE architecture: MoE layers activate only a subset of expert FFNs per token, increasing model capacity while limiting per-token computation.This sparse design replaces a shared dense FFN with multiple expert FFNs selected by an input-dependent gate.
- PD disaggregation: PD disaggregation separates compute-bound prompt processing from sequential, latency-sensitive token generation onto independent worker pools.This reduces phase interference and allows prefill and decode capacity to scale separately.
- Routing gap: Existing routers assign decode workers using cache awareness or load balancing while ignoring model-internal expert activations.These policies decide placement without modeling the activated-expert cost imposed on the destination worker.
- Routing gap: MoE decode latency is governed by the union of distinct experts activated at each step, because each distinct expert requires a weight load.Thus, equally loaded workers can have different latency when their batches activate different expert sets.
3 Motivation
MoE decode cost depends on active-expert reuse, and request domains create exploitable locality. Prefill activations expose that locality before decode, enabling ELDR to route similar requests together while retaining load awareness.
- Active experts drive latency: 4.7× higher latency results when Qwen3-30B-A3B active experts grow from 16 to 128 at batch size 64.At fixed expert count, changing batch size barely moves latency, showing that active experts dominate batch size.
- Domain locality: Code, math, medical, and legal requests over-activate different expert subsets, while multilingual requests separate into distinct expert regions.This correlation across related requests makes expert locality a placement signal rather than merely token-level sparsity.
- Prefill as a signal: Prefill expert activations predict decode activation before output generation, exposing request-specific expert footprints at the routing handoff.The representation must preserve proximity that reflects decode-time expert overlap, independent of downstream clustering or routing.
- Domain locality: 17–21% fewer distinct experts are activated by same-domain task batches than mixed-domain batches across batch sizes 32–128.Language workloads show 3–10% fewer active experts for same-domain batches.
- System challenges: Prefix caching can make signatures incomplete unless per-block signatures remain coherent with KV blocks across partial hits, full hits, and evictions.A whole-prompt signature cannot describe a request that reuses leading KV blocks but diverges later.
4 Design
ELDR turns prefill-time expert activations into signatures that guide decode placement by locality while preserving instantaneous load balancing. Its offline, online, and caching mechanisms improve expert reuse without changing model computation or outputs.
- Expert signatures: ELDR uses each request’s prefill expert activation to predict decode-time expert overlap and place similar requests together.The signature is designed so nearby requests are likely to share experts during decode.
- Serving integration: A block-granular signature cache co-indexed with the KV cache reconstructs complete signatures under prefix-cache hits without recomputation.Cached and freshly computed blocks are summed coherently across partial hits, evictions, and reuse.
- Expert signatures: The signature combines per-layer expert counts with IDF reweighting and a learned layer mask to emphasize informative specialist experts.Greedy layer selection keeps the first N* layers where cumulative signature quality ρ peaks.
- Signature validation: Count-based signatures outperform continuous gate scores by >0.035 in mean ρ across the evaluated models and workloads.Continuous scores assign mass to experts that decode top-k gating never loads, whereas count-based variants preserve the discrete routing signal.
- Signature validation: IDF raises mean ρ over plain count by 1.5pt and worst-case min_cρ by 4.9pt, reaching 0.68 from 0.63 on GPT-OSS language.The largest gain is on GPT-OSS task, where IDF improves performance by +8.9pt.
- Signature validation: Every evaluated cell peaks at N* < L, exceeding all-layer ρ(L) by 0.005 to 0.032.Extra layers add dimensions that do not separate requests and dilute the signature.
- Clustering and routing: Balanced K-means creates equal-volume locality regions offline, then routes online to the less-loaded worker within the matching region.Hungarian-balanced K-means limits each centroid to at most ⌈N/K⌉ calibration points, preventing skewed clusters from causing decoder imbalance.
6 Evaluation
ELDR consistently lowers decode latency by routing requests according to expert locality while preserving live-load balancing. Across models, workloads, topologies, and design variants, its gains are largest where expert activations are most separable.
- Main results: ELDR reduces median TPOT by 7.0–13.9% and tail TPOT by 3.4–6.0% on the task workload versus the best load balancer.It remains below every baseline at every request rate and beats Domain by 1.4–6.9% on median TPOT.
- Main results: ELDR reduces median TPOT by 5.9–10.0% on the language workload versus the best load balancer.Mean tail TPOT improves by 6.2% on Qwen3-30B-A3B but regresses by 1.5% on GPT-OSS-120B and 0.2% on Gemma-4-26B-A4B.
- Mechanism: 22.0% fewer active experts per decode step explains ELDR’s TPOT improvement versus round-robin.This result is averaged across decode batch sizes on Qwen3-30B-A3B task serving at 8P16D.
- Design validation: ELDR’s count·idf signature reduces TPOT P50 by an additional 3 percentage points on average versus softmax gate-probability signatures.The improvement reaches 14 percentage points across the six evaluation cells.
- Design validation: Hungarian-balanced K-means reduces P50 by up to 12.6% while avoiding vanilla K-means tail regressions of as much as 17.4%.Vanilla K-means improves median TPOT by up to 9.8% but can worsen tail TPOT because of load imbalance.
- Design validation: A locality band of τ=0.1 removes pure top-1 routing’s tail regression on every workload while reducing median TPOT by 5.2–12.7% versus round-robin.Larger bands saturate tail reductions and erode median TPOT.
- Prefix caching: ELDR preserves its TPOT advantage with prefix caching, retaining approximately 13% P50 and 5% P99 improvement over round-robin.The locality and prefix-cache benefits are additive, while faster decoders can also reduce TTFT through back-pressure relief.
- Topology generalization: ELDR scales across decoder pools, with mean median-TPOT reductions versus round-robin of 8.0%, 9.8%, and 10.2% at 8P8D, 8P16D, and 8P24D.Tail TPOT stays within noise of round-robin throughout this sweep.
7 Related Work
ELDR extends prior MoE and PD-serving techniques with inter-worker expert-locality routing. It changes request placement rather than token expert selection, so it remains lossless and composes with intra-worker balancing.
- PD-disaggregated serving: ELDR adds expert-activation locality to PD decode routing, complementing cache-aware routers that otherwise treat decode workers as interchangeable for expert computation.Its prefill-derived signal targets which worker serves a whole request.
- MoE expert locality: Unlike intra-worker expert balancing, ELDR routes whole requests across decode workers to shrink each worker’s active-expert union.This makes ELDR orthogonal to methods such as expert replication and per-GPU token rerouting.
- Lossless routing: ELDR is lossless because it changes only the serving decode worker, never a token’s expert selection.Outputs therefore match standard top-k gating, while each worker may still run intra-worker balancing.
8 Conclusion
ELDR reframes MoE decode routing around expert locality rather than load alone. Its prefill-derived, cache-coherent routing lowers TPOT without changing model outputs.
- Conclusion: ELDR adds expert locality as a second decode-routing axis alongside instantaneous worker load.A prefill-derived signature enables locality-aware, load-tolerant assignment across workers.
- Conclusion: ELDR lowers median and tail TPOT across models, workloads, and topologies while preserving model outputs.The router changes only which decode worker serves each request.