Source-linked AI summary
Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference
Xinyuan Gui, Shaowen Wang, Sheng Sun, Zijian Wang, Zishu Yu, Zheming Yang
TL;DR
MLLM inference is costly, while existing routers leave token-level uncertainty during generation unused. Pro-Router combines prompt pre-screening, token-aware verification, and adaptive edge-cloud serving, achieving the highest routing accuracy, over 10× faster routing signals, and more than 75% higher throughput than the existing routing pipeline.
Problem
MLLM serving is expensive, while existing routing methods rely on coarse request features or extra language-model passes and leave token-level uncertainty unused.
Method
Pro-Router progressively routes requests using a lightweight prompt pre-scorer, a token-aware verifier based on per-token sampling distributions, and an adaptive edge-cloud collaboration pipeline.
Results
Across 15 benchmarks and three models, Pro-Router achieves the highest routing accuracy, produces routing signals more than 10× faster than prior methods, and delivers more than 75% higher throughput than the existing routing pipeline.
Takeaways & Limitations
Token-level verification and adaptive serving allow the edge and cloud tiers to remain utilized without manual tuning while improving routing and end-to-end throughput.
Abstract
from arXiv · showhide
The remarkable performance of multimodal large language models (MLLMs) comes at the cost of substantial computational overhead, posing significant challenges to real-time deployment and cost effectiveness. Existing model routing approaches either decide from coarse request-level features alone or spend one or several extra language model passes to inspect the generated response, leaving the token-level uncertainty signals that emerge during generation unused. To address these limitations, we propose Pro-Router, a token-aware progressive model routing method with adaptive edge-cloud collaboration for efficient multimodal LLM inference. Pro-Router employs a two-stage progressive decision mechanism. First, a lightweight prompt pre-scorer module performs rapid pre-screening before token generation begins, guiding apparently simple requests to small models. Second, a token-aware verifier reads the sampling probability distribution of each token the small model generates, estimating the model's confidence in its own output to determine, per request, whether the answer ships or escalates to the cloud-based high-precision model. Furthermore, we design an adaptive edge-cloud serving pipeline that sizes every dispatch to each device's measured service rate, so both the edge and the cloud tiers stay fully utilized without manual parameter tuning and are not impacted by the network latency. Extensive experiments on multiple multimodal benchmark datasets and models demonstrate the effectiveness of Pro-Router. Compared to other methods, it achieves the highest routing accuracy and improves routing speed by more than 10x. Its serving pipeline also reaches more than 75% higher end-to-end throughput than the existing model routing pipeline. Our code is available at https://github.com/xinyuangui2/pro-router.
Introduction
MLLMs deliver strong multimodal capabilities but are expensive to serve, while smaller models can answer many requests on cheaper edge devices. Pro-Router combines prompt pre-screening, token-level verification, and adaptive edge-cloud scheduling to improve routing and utilization.
- Motivation: MLLM serving is costly because large autoregressive models process every generated token, while images add hundreds or thousands of tokens to computation and KV-cache demand.These models therefore tend to run on expensive cloud GPUs.
- Motivation: Many requests can be handled equally well by multimodal small language models that fit on much cheaper edge devices.The paper characterizes these small models as roughly an order of magnitude smaller than state-of-the-art large models.
- Existing Routing: Request-only routers are inexpensive but inaccurate, whereas response-based routers are more informative but require one or more additional language-model passes.The two existing strategies are distinguished by whether they decide before generation or after the small model produces an answer.
- Serving Challenge: Existing edge-cloud collaboration pipelines can waste GPU cycles because frequent interactions make tiers dependent on network conditions.Network fluctuation can stall whichever tier is waiting.
- Pro-Router: Pro-Router uses a lightweight pre-scorer before generation, a token-aware verifier during small-model decoding, and adaptive dispatch based on device service rates.The verifier determines whether each answer ships or escalates, while the scheduler feeds easy requests to edge models and hard or escalated requests to cloud models.
- Results: More than 10× faster routing decisions and more than 75% higher end-to-end throughput are reported across 15 benchmarks, three models, and four baselines.The experiments compare against a real 72B cloud target.
Related Work
Prior model-routing work primarily separates request-only methods from response-based methods, with recent extensions to edge-cloud and multimodal settings. These approaches trade lower decision cost for limited access to the small model’s actual answer or incur extra response-scoring computation.
- Foundations: Model routing builds on selective prediction, the reject option, and learning-to-defer frameworks.These lines of work motivate serving each request with a model capable of handling it.
- Request-Only Routing: Request-only routers use the prompt alone to select models, including RouteLLM, Hybrid LLM, and RouterDC.Their mechanisms include preference-data training, quality-gap prediction, and contrastive query encoding.
- Multimodal Routing: Recent multimodal routing work selects edge or cloud VLMs according to scenario or semantic difficulty, including ECVL-ROUTER and AVR.The passage places these methods within the broader request-only category because they do not observe the small model’s actual answer.
- Response-Based Routing: Response-based methods decide after the small model answers and include answer-text scoring and semantically enhanced edge-cloud routing.Examples include FrugalGPT, MLLM-based adaptive routing, SAEC, and AIVD.
Method
Pro-Router combines prompt-level pre-screening with token-level verification, then adaptively dispatches work across heterogeneous edge and cloud devices. Its routing objective and scheduling pipeline jointly target accurate escalation decisions and sustained device utilization.
- Progressive routing: Pro-Router first scores incoming requests before generation, serves the easiest from edge MSLMs, and verifies edge decoding before shipping or escalating each answer.The pre-scorer orders requests by estimated difficulty; the verifier uses per-token sampling distributions produced during small-model decoding.
- Routing objective: The routing objective trades off ship rate, conditional accuracy, latency, and resource utilization while penalizing wrongly shipped answers.A request ships when confidence p ≥ τ; shipped answers have conditional accuracy a, and incorrect shipments incur penalty λ.
- Prompt pre-scorer: The pre-scorer uses TF-IDF, image count, and prompt length in a lightweight logistic regression that guides traffic without competing for accelerator capacity.It provides approximate difficulty guidance rather than making the final ship decision, preserving edge utilization.
- Token-aware verifier: The verifier reuses token-level decoding distributions, including top-K entropy, so its feature sequence requires no extra model pass.Per-token features are stacked across generated tokens and processed by a transformer encoder.
- Adaptive collaboration: Adaptive scheduling estimates device throughput with an EMA and dispatches work proportional to measured service rates, using lookahead margins to maintain continuous utilization.The scheduler operates in fixed windows and skips devices under backpressure while filling edge and cloud queues according to request difficulty and escalation status.
- Dispatch policy: Cloud devices prioritize escalated requests and then the hardest scored requests, while edge devices receive the easiest scored requests.This queue policy aligns request difficulty with model capacity and separates already rejected work from new scored traffic.
Experiments
Pro-Router is evaluated across diverse multimodal and text benchmarks, models, routing baselines, accuracy metrics, latency, throughput, robustness, scalability, and ablations. It achieves leading routing accuracy and substantial serving-efficiency gains while preserving throughput under network delay and device expansion.
- Evaluation Setup: The evaluation spans 15 benchmarks, including eleven vision-language and four text benchmarks.The vision-language suite contains nine single-image and two multi-image benchmarks.
- Evaluation Setup: Experiments use three small models—Qwen2.5-VL-7B, LLaVA-OneVision-7B, and Pixtral-12B—and Qwen2.5-VL-72B as the large model.The serving study sweeps edge and cloud device counts using A10G edge devices and A100 cloud GPUs.
- Routing Accuracy: Pro-Router achieves the highest mean AUROC, macro PGR, AUARC, and AUACC across every small model and benchmark average.The lead holds under both correctness and pairwise shipping labels.
- Routing Latency: 2 to 3 ms is the verifier’s routing-signal latency, making it 19 to 28× faster than the cheapest alternative.The scorer baselines require 47 to 119 ms, while self-verification methods require much longer generation passes.
- End-to-End Throughput: 1.16 to 1.28× the strongest baseline signal is Pro-Router’s end-to-end throughput with two edge devices and one cloud device.Its CPU verifier runs in parallel with decoding, while prompt-score ordering raises the served ship rate.
- End-to-End Throughput: 1.77 to 1.79× faster than Ray Serve is the proposed pipeline on every tested model.The adaptive pipeline balances edge and cloud service rates instead of making the small model a bottleneck.
- Robustness and Scalability: 90 to 96% of un-delayed throughput is retained even with 1000 ms one-way network latency across the three models.Throughput also scales linearly with edge-device count, reaching 5.8 to 6.9× a single cloud device at M = 3 and N = 4.
- Ablation Studies: Removing batched dispatch drops throughput from 124 to 28 req/s, while disabling the KV-cache admission gate raises latency from 754 to 1634 ms.The ablations identify both optimizations as necessary for efficient edge-cloud serving.
Conclusion
Pro-Router combines token-aware progressive routing with adaptive edge-cloud serving to improve routing speed and throughput for multimodal inference. Across 15 benchmarks and three models, it reports over 10× faster decisions and more than 75% higher throughput than the existing routing pipeline.
- Pro-Router combines token-aware progressive routing with an adaptive edge-cloud pipeline.The verifier selects whether small-model answers ship, while the pipeline keeps edge and cloud tiers utilized without manual tuning.
- 10× faster decisions are achieved by the verifier across 15 benchmarks and three models.The paper reports that the verifier is the most accurate signal and decides over 10× faster.
- More than 75% higher throughput is achieved than by the existing model routing pipeline.