PaperlayerView exact revision on arXiv

Source-linked AI summary

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang

arXiv:2610.12242v1cs.CL

TL;DR

Token-level routing can offer efficiency and quality gains beyond query-level routing, while serving must handle sparse, irregular routed-token arrivals. TokenRouter combines request-centric routing with asynchronous model-centric execution and delayed batching; experiments report 2.01–64.15× higher decoding throughput than existing implementations.

  • Problem

    Token-level routing offers efficiency and quality gains beyond query-level routing, but serving must handle sparse, irregular routed-token arrivals.

    Source: §1 · p. 2, §4.3 · p. 6

  • Method

    TokenRouter pairs a request-centric routing interface with asynchronous model-centric subservers and delayed batching.

    Source: §1 · p. 2

  • Results

    Across routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01–64.15× higher decoding throughput than existing implementations.

    Source: §6 · p. 9

  • Takeaways & Limitations

    TokenRouter provides a strong foundation for future token-routing research.

    Source: §6 · p. 9

    The mathematical model assumes geometrically distributed token counts between consecutive sends; handling corner cases that violate this assumption is left for future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.

1 Introduction

Token-level routing can improve efficiency and quality beyond query-level routing, but current single-LLM serving systems are not designed for its per-step model switches. TokenRouter separates request-centric routing logic from asynchronous model-centric serving and adds delayed batching.

1 Introduction

  • Token-level routing can exploit within-query variation in generation difficulty and combine complementary expertise, enabling efficiency and quality gains beyond coarse-grained routing.R2R matches a 32B model’s quality while routing 5% of tokens to it, compared with routing around 40% of queries to it.Source: §1 · p. 2
  • Single-LLM serving systems face step desynchronization, batch admission delays, and implementation complexity when models can change at each decoding step.Synchronous batches wait for the slowest model, while frequent switches can leave routed requests waiting for the next batch.Source: §1 · p. 2
  • TokenRouter lets developers express routing through route, send, and receive while its runtime uses asynchronous subservers and delayed batching.The request-centric interface separates routing logic from cross-model execution and scheduling.Source: §1 · p. 2
  • Across routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01–64.15× higher decoding throughput than existing implementations.The paper presents this as turning token-routing benefits into real-world speedups.Source: §1 · p. 2

2 Related Work

Prior work spans request-level model choice and token-level collaboration, while serving systems have largely been designed around single-model decoding. TokenRouter targets the irregular model switches of token-level routing.

2 Related Work

  • Coarse-grained routers choose a model per request or conversation turn, including efficiency-oriented routing under quality constraints [3] [16] [17] [19] [20] [21] and expert selection or combination [23] [24] [25].Production stacks [1] can reuse standard single-model batching and scheduling because each request stays with one model.Source: §2 · p. 3
  • Token-level methods route by token or short span using trained routers [5] [6] [12] [26], heuristic signals [27] [28], expert-token acceptance [7], or output-distribution fusion [9] [11].Their shared lifecycle of selecting a model, transferring request state, and receiving outputs motivates TokenRouter’s interface.Source: §2 · p. 3
  • Single-model serving techniques include continuous batching, paged KV-cache management, and structured-generation runtimes, whereas token-level routing causes irregular model-local arrivals and fragmented batches.Speculative decoding uses multiple models in a predictable pattern, unlike token-level routing’s data-dependent switches.Source: §2 · p. 3

3 Programming Interface

TokenRouter’s programming interface lets developers describe token-level routing from the perspective of one request, while the runtime executes it across cooperating models. Its route, send, and receive components specify model choice, transferred state, and how returned requests resume.

3.1 Principle: Sequential Request-centric Programming

  • TokenRouter’s request-centric programming, model-centric execution principle lets developers trace one request while the runtime handles execution through model-centric subservers.The request lifecycle repeatedly routes, transfers state when delegated, and resumes generation at the receiving model.Source: §3.1 · p. 4

3.2 Core Interface

  • The interface consists of route, which chooses whether each request continues locally or moves to a peer, send, which constructs the transferred request state, and receive, which converts it for local decoding.By default, receive commits the transferred tokens; algorithm-specific handoff semantics can override this behavior.Source: §3.2 · p. 4

3.3 Running Examples

  • In the R-Stitch example, the small model delegates high-entropy tokens to the large model, while the large model sends low-entropy tokens back.The small-model side excludes the uncertain token from the transfer so the large model can regenerate from that position.Source: §3.3 · p. 4
  • The evaluated algorithms differ in routing signals: CITER, R2R, and Co-LLM switch on uncertain tokens, R-Stitch switches symmetrically by entropy, and ME selects among more than two models using ensemble weights.Each is expressible through the route function.Source: §3.3 · p. 4

4 System Design

TokenRouter hosts cooperating LLMs in autonomous subservers behind one client interface. Asynchronous execution lets subservers progress independently, while delayed batching balances admission delay against batch formation and throughput.

4.1 System Overview

  • Each model runs in an autonomous subserver with its own scheduler, LLM runner, KV-cache pool, and routing functions, behind a single external server interface.The client protocol remains the same as for standard single-LLM servers.Source: §4.1 · p. 5

4.2 Efficient Serving with Asynchronous Execution

  • The decoupled inter-model loop sends and receives peer requests so each subserver progresses at its own pace instead of synchronizing every decoding step.While a routed request runs on a peer model, the local model continues processing its remaining requests.Source: §4.2 · pp. 5–6, Figure 4 · p. 6
  • Handoff-resume preserves a routed request’s serving state as pending, then resumes it without repeating prefix matching or KV-cache allocation.When the peer returns, the request is admitted locally as if it had never left, requiring little more than appending tokens.Source: §4.2 · p. 6

4.3 Efficient Scheduling with Delayed Batching

  • Under eager asynchronous scheduling, a request arriving during an active decoding step must wait for that step to finish, producing repeated admission stalls and fragmented batches.The delay is especially relevant because token-level routing causes frequent model switches and high per-step latency variance.Source: §4.3 · pp. 6–7
  • Delayed batching buffers requests until a same-model batch reaches threshold B, trading extra waiting for early arrivals against lower average batch admission delay.The throughput-optimal B* is derived using a DTMC model with concurrency N, routing probability P, and each LLM’s per-step latency Li.Source: §4.3 · p. 7

5 Experiment

TokenRouter is evaluated across routing algorithms, workloads, and model configurations, with results showing higher throughput and lower latency than baselines. Ablations examine throughput scaling, model pairs, and parallelism, while a throughput–accuracy comparison tests its Pareto frontier against query-level routing.

5.1 Setup

  • The evaluation covers five routing algorithms, three workload types, an 8×A100-80G server, official-code and SGLang-based baselines, and token throughput plus request completion latency.Workloads span short-input/short-output reasoning, short-input/long-output reasoning, and multi-turn agentic tasks.Source: §5.1 · p. 7

5.2 Serving Efficiency

  • 2.01–64.15× higher throughput than the stronger baseline and 2.03–63.64× lower end-to-end latency than Std. Serving across 15 algorithm–workload combinations.Figures 5 and 6 compare throughput and latency across serving systems at concurrency 4.Source: §5.2 · pp. 7–8, Figure 5 · p. 8, Figure 6 · p. 8
  • TokenRouter’s throughput holds steady as maximum output length rises from 2,048 to 8,192 tokens, while Std. Serving loses 58.1–85.2% of throughput.Source: §5.2 · p. 8
  • At concurrency 16, TokenRouter reaches 18.58× the throughput of official R2R at concurrency 1 while retaining 1.13× higher per-user speed.As concurrency rises from 1 to 16, TokenRouter throughput increases 8.61× and retains 51.7% of single-user speed; official R2R reaches 5.14× and retains 31.3%.Source: §5.2 · p. 8

5.3 Ablation Study

  • 132.78 to 230.79 token/s from engineering optimizations, then 296.86 and 372.48 token/s with asynchronous execution and delayed batching, reaching 2.76× overall.The 230.79 token/s result is 1.71× the official R2R implementation’s 134.9 token/s at concurrency 8.Source: §5.3 · p. 8
  • 1.99–3.21× higher throughput across model pairs; LLM tensor parallelism improves throughput, while SLM tensor parallelism has little effect.Unless otherwise specified, the evaluation uses LLM tensor parallelism size 2 and SLM tensor parallelism size 1.Source: §5.3 · pp. 8–9, Table 3 · p. 9

5.2 Serving Efficiency

  • 2.73–21.97× higher throughput and 2.78–21.61× lower latency than official implementations under the algorithms’ original settings.Table 2 reports TTFT, throughput, and end-to-end latency at concurrency 4.Source: §5.2 · p. 8, Table 2 · p. 9

5.4 Advancing the Pareto Frontier

  • TokenRouter shifts token-level routing to a new throughput–accuracy Pareto frontier, making it more competitive than query-level routing on SGLang.The comparison uses AMC23 and the Qwen3-0.6B/Qwen3-32B pair.Source: Figure 7 · p. 9, §5.4 · p. 9

6 Conclusion

TokenRouter separates routing algorithms from serving optimization through a request-centric interface and asynchronous model-centric execution. Its reported throughput gains span algorithms, workloads, and model pairs, while its mathematical model has a stated distributional assumption.

6 Conclusion

  • TokenRouter separates routing algorithms from serving optimization through a request-centric route-send-receive interface and asynchronous model-centric execution with delayed batching.Source: §6 · p. 9
  • 2.01–64.15× higher decoding throughput than existing implementations across routing algorithms, workloads, and model pairs.Source: §6 · p. 9
  • The mathematical model assumes geometrically distributed token counts between consecutive sends; handling corner cases that violate this assumption is future work.Source: §6 · p. 9

A Additional Experiment Results

Additional concurrency-1 experiments report higher throughput and lower latency than standard serving and available stronger baselines. Results under original algorithm settings also report efficiency gains for R2R, CITER, and Co-LLM, with comparable TTFT.

A.1 Additional Concurrency

  • 2.12–19.74× higher throughput and 2.12–18.83× latency speedup than standard serving across five algorithms and three workloads at concurrency 1.TokenRouter also delivers 1.60–19.74× higher throughput than the stronger available baseline between official code and standard serving.Source: §A.1 · p. 13
  • 1.93–10.20× higher throughput for R2R, CITER, and Co-LLM than their official implementations at concurrency 1 under original settings.Source: §A.1 · p. 13
  • 2.30–10.09× lower end-to-end latency for R2R, CITER, and Co-LLM, while maintaining comparable TTFT.Source: §A.1 · p. 13

A.2 Additional Ablation Study

The additional studies quantify TokenRouter’s optimization gains and examine deployment choices, cross-node performance, and the latency gap with Std. Serving.

A.2 Additional Ablation Study

  • 1.88–2.12× throughput improvement from CUDA graphs, with asynchronous execution and delayed batching adding 1.12× and 1.31×, respectively, at concurrency 4.The study attributes TokenRouter’s overall gain to the combined benefits of these optimizations.Source: §A.2 · p. 13, Table 6 · p. 14
  • 40.6% and 22.8% throughput gains from increasing LLM tensor parallelism from 1 to 2 at N = 1 and N = 4, versus 0.8% and 0.4% from increasing SLM tensor parallelism.TokenRouter therefore prioritizes LLM tensor parallelism and uses SLM TP size 1 by default.Source: §A.2 · p. 13, Table 7 · p. 15

A.3.1 Non-overlapping GPU Deployment

  • 6.43% average throughput improvement from deploying Qwen3-0.6B and Qwen3-32B on separate GPUs rather than using the original overlapping setting.Source: §A.3.1 · p. 14

A.3.2 Cross-node Deployment

  • 1.93–32.81× throughput improvements over the respective official implementations, with cross-node throughput comparable to single-node throughput using the same computational resources.Table 9 compares throughput across single-node and multi-node settings.Source: §A.3.2 · p. 14

A.4 Latency Breakdown of the Std. Serving Baseline

  • Prefix matching and RadixCache maintenance account for most profiled SLM-side latency in Std. Serving, while TokenRouter preserves request-level KV-cache state across model handoffs and resumes execution.Table 10 reports the SLM-side execution latency breakdown per decoding step.Source: §A.4 · p. 14

B Token-Routing-Specific Optimizations

TokenRouter adapts single-LLM serving assumptions that token-level routing breaks, addressing shared KV-cache allocation, extend-mode execution, router placement, and subserver state tracking.

B Token-Routing-Specific Optimizations

  • TokenRouter jointly allocates per-model KV-cache reservations within a shared GPU budget, computing the largest common KV-token capacity that fits and assigning each subserver its share before launch.This avoids manual reservation tuning and the over-reservation or idle-capacity problems described for separate per-model budgets.Source: B Token-Routing-Specific Optimizations · p. 15
  • TokenRouter captures CUDA graphs for configurable extend-batch sizes and token lengths, falling back to non-graph extend execution for batches outside the captured range.Because peer returns arrive as extend batches, this targets the handoff path; the common case avoids per-handoff kernel-launch overhead while long extends remain served.Source: B Token-Routing-Specific Optimizations · p. 15
  • TokenRouter co-locates the router with the scheduler on the LLM runner’s GPU, making each token-level route decision an in-process tensor operation.The routing function runs after every forward pass on every active request, so a cross-process call would add an IPC round-trip per token and serialize routing against decoding.Source: B Token-Routing-Specific Optimizations · p. 15
  • Subservers keep their active-request counts consistent by piggybacking admission and completion updates on peer messages rather than using a separate central coordinator.The delayed-batching threshold depends on the system-wide active-request count.Source: B Token-Routing-Specific Optimizations · pp. 15–16

C Implementation Details

TokenRouter is implemented on SGLang 0.5.1, and the appendix derives a throughput model for optimizing delayed-batching thresholds. The model represents routed requests and batch execution as a finite-state process and uses its stationary distribution to characterize throughput.

C Implementation Details

  • TokenRouter builds on SGLang 0.5.1, using adapted multi-model server arguments, ZeroMQ IPC, dedicated subprocesses, and scheduler subclasses for evaluated routing algorithms.Each subserver runs its three-stage iteration in a single event loop.Source: C Implementation Details · p. 16

D.1 Problem Formulation

  • The model seeks steady-state throughput as a function of batching thresholds, assuming stationary routing probabilities and independent routing decisions across requests.When routing probabilities vary over time, the analysis applies over intervals where they can be approximated as constant.Source: §D.1 · p. 17

D.2.1 Problem Modeling: Discrete-Time Markov Chain

  • The finite-state DTMC records each model’s waiting queue, running-batch size, and residual execution time at batch-completion events.A common time quantum discretizes model latencies so the candidate state space is finite.Source: §D.2.1 · pp. 17–18

D.2.2 Transition Matrix and Stationary Distribution

  • The model derives state transitions from routed batch completions and delayed-batching launches, then uses the reachable-state chain’s stationary distribution to characterize throughput.The authors report that the reachable-state chain was irreducible for all configurations encountered during optimization.Source: §D.2.2 · pp. 18–19

D.2.3 Throughput Derivation

The throughput model closely captures measured trends under different concurrency levels and delayed-batching thresholds, and predicts the empirically optimal threshold at every evaluated concurrency level. It derives throughput from committed tokens and elapsed time, then uses the model to optimize the threshold vector.

D.2.3 Throughput Derivation

  • The model derives steady-state aggregate committed-token throughput from expected committed tokens per state transition and expected time between transitions.It takes the ratio of long-run committed tokens to long-run elapsed time.Source: §D.2.3 · pp. 19–20

D.3 Delayed-Batching Threshold Optimization

  • The threshold optimizer selects the feasible delayed-batching vector that maximizes throughput according to the model’s stationary distribution.Feasibility rules out deadlock states in which no model has enough queued requests to launch a batch.Source: §D.3 · p. 20

D.4 Empirical Validation of the Throughput Model

  • At every evaluated concurrency level, the throughput model’s predicted optimal threshold matches the empirical optimum, while closely capturing measured throughput trends across thresholds.Figure 10 compares predicted and measured throughput with B0 = 1 while varying B1.Source: Figure 10 · p. 20, §D.4 · p. 21
Loading 2610.12242v1…